From 6682c5e308bc9c3845ef8ce80e495d66b412a9c1 Mon Sep 17 00:00:00 2001 From: Luke Piette Date: Sat, 14 Nov 2026 14:14:35 -0500 Subject: [PATCH 1/3] docs: add guides for migrating to Runpod from 14 other platforms Add a "Migrate to Runpod" section: an overview page and a guide each for Vast.ai, Lambda, CoreWeave, Paperspace and DigitalOcean, Modal, Replicate, Baseten, fal, Together AI, Fireworks AI, Hugging Face, AWS, Google Cloud and Azure. The existing Cog and OpenAI migration pages move into the new nav group, and the agent skills page lists the runpod-cloud-migrate skill. Co-Authored-By: Claude Fable 5.2 --- docs.json | 47 +- get-started/agent-skills.mdx | 5 + migrate/aws.mdx | 1381 ++++++++++++++++++++++++++++++++ migrate/azure.mdx | 1400 ++++++++++++++++++++++++++++++++ migrate/baseten.mdx | 1375 ++++++++++++++++++++++++++++++++ migrate/coreweave.mdx | 1279 ++++++++++++++++++++++++++++++ migrate/fal.mdx | 1392 ++++++++++++++++++++++++++++++++ migrate/fireworks-ai.mdx | 929 ++++++++++++++++++++++ migrate/google-cloud.mdx | 1446 ++++++++++++++++++++++++++++++++++ migrate/hugging-face.mdx | 1112 ++++++++++++++++++++++++++ migrate/lambda.mdx | 1008 ++++++++++++++++++++++++ migrate/modal.mdx | 1219 ++++++++++++++++++++++++++++ migrate/overview.mdx | 194 +++++ migrate/paperspace.mdx | 1333 +++++++++++++++++++++++++++++++ migrate/replicate.mdx | 1286 ++++++++++++++++++++++++++++++ migrate/together-ai.mdx | 1183 +++++++++++++++++++++++++++ migrate/vast-ai.mdx | 1424 +++++++++++++++++++++++++++++++++ 17 files changed, 18007 insertions(+), 6 deletions(-) create mode 100644 migrate/aws.mdx create mode 100644 migrate/azure.mdx create mode 100644 migrate/baseten.mdx create mode 100644 migrate/coreweave.mdx create mode 100644 migrate/fal.mdx create mode 100644 migrate/fireworks-ai.mdx create mode 100644 migrate/google-cloud.mdx create mode 100644 migrate/hugging-face.mdx create mode 100644 migrate/lambda.mdx create mode 100644 migrate/modal.mdx create mode 100644 migrate/overview.mdx create mode 100644 migrate/paperspace.mdx create mode 100644 migrate/replicate.mdx create mode 100644 migrate/together-ai.mdx create mode 100644 migrate/vast-ai.mdx diff --git a/docs.json b/docs.json index 981e9ac26..bcf1f8d7f 100644 --- a/docs.json +++ b/docs.json @@ -374,18 +374,53 @@ ] }, { - "group": "Reference", + "group": "Migrate to Runpod", "pages": [ - "references/gpu-types", - "references/cpu-types", - "references/security-and-compliance", + "migrate/overview", + { + "group": "GPU clouds", + "pages": [ + "migrate/vast-ai", + "migrate/lambda", + "migrate/coreweave", + "migrate/paperspace" + ] + }, + { + "group": "Serverless and inference platforms", + "pages": [ + "migrate/modal", + "migrate/replicate", + "migrate/baseten", + "migrate/fal", + "migrate/together-ai", + "migrate/fireworks-ai", + "migrate/hugging-face" + ] + }, + { + "group": "Hyperscalers", + "pages": [ + "migrate/aws", + "migrate/google-cloud", + "migrate/azure" + ] + }, { - "group": "Migrations", + "group": "Frameworks", "pages": [ "tutorials/migrations/cog/overview", "tutorials/migrations/openai/overview" ] - }, + } + ] + }, + { + "group": "Reference", + "pages": [ + "references/gpu-types", + "references/cpu-types", + "references/security-and-compliance", { "group": "Intro to containers", "pages": [ diff --git a/get-started/agent-skills.mdx b/get-started/agent-skills.mdx index 8775be7e3..694a715fa 100644 --- a/get-started/agent-skills.mdx +++ b/get-started/agent-skills.mdx @@ -57,6 +57,7 @@ The plugin installs a router and a set of focused skills: | **runpod-templates** | Picks and configures Runpod's official Pod templates, such as ComfyUI and PyTorch, and fixes problems with a running template. | | **runpod-build-template** | Builds a complete template project (Dockerfile and app files) for Pods or Serverless from a request, local code, or a GitHub repo. | | **runpod-migrate** | Migrates a codebase from the GraphQL API or REST v1 to REST v2. It inventories which API each call site uses, rewrites the call sites, and verifies the result. | +| **runpod-cloud-migrate** | Moves a project to Runpod from another cloud or AI platform. It inventories what the project uses, maps it with a per-platform playbook, plans, ports the code, and verifies the result. | The plugin also includes a collection of worked, end-to-end reference tasks, such as deploying a Whisper endpoint or running a model on a Pod, that guide your agent through common workflows. @@ -95,6 +96,10 @@ The command takes an optional scope and path: Running the command requires the Runpod skills plugin installed in your coding agent (see [Quick start](#quick-start)). For a manual walkthrough of the API changes, see the [migration guide](/api-reference-v2/migrate-from-v1). +## Migrate from another platform + +If your workloads run on another cloud or AI platform, the `/runpod:migrate-cloud` command moves them to Runpod. It scans your repository for the platforms it depends on, proposes a plan with a cost estimate, and waits for your approval before it creates anything. It then migrates one workload at a time and verifies each with a real request. See [Migrate to Runpod](/migrate/overview) for the guide to each platform. + ## Native install options The `npx skills add` command above works everywhere. If you'd rather install the plugin through your agent's native marketplace, use the route for your agent below. Each route installs the same router and skills. diff --git a/migrate/aws.mdx b/migrate/aws.mdx new file mode 100644 index 000000000..1d55bd4dc --- /dev/null +++ b/migrate/aws.mdx @@ -0,0 +1,1381 @@ +--- +title: "Migrate from AWS to Runpod" +sidebarTitle: "AWS" +description: "Move GPU training, inference, and batch workloads from EC2, SageMaker AI, and Bedrock to Runpod while keeping data and auth on AWS." +--- + +This guide is for teams that run GPU workloads on Amazon Web Services and want to move the GPU-heavy part (training, inference, batch) to Runpod. Most teams keep S3, IAM, databases, and the rest of the application on AWS, so the guide treats that hybrid setup as the default and a full exit as the second case. + +[Amazon EC2](https://docs.aws.amazon.com/ec2/) GPU instances map to [Pods](/pods/overview). [Amazon SageMaker AI](https://docs.aws.amazon.com/sagemaker/) real-time, asynchronous, and batch inference map to [Serverless](/serverless/overview) endpoints. SageMaker training and processing jobs map to Pods or the Runpod SDK. [Amazon Bedrock](https://docs.aws.amazon.com/bedrock/) maps to [Public Endpoints](/public-endpoints/overview) or a Serverless vLLM endpoint. [AWS ParallelCluster](https://docs.aws.amazon.com/parallelcluster/) and SageMaker HyperPod map to [Instant Clusters](/instant-clusters). When you finish, the GPU workload runs on Runpod, your callers use a Runpod URL and API key, your Terraform manages both clouds, and the AWS resources you replaced are shut down. + + +You can hand this migration to a coding agent. See [Migrate with a coding agent](#migrate-with-a-coding-agent) for the setup commands and a prompt. + + +## How AWS maps to Runpod + +| AWS concept | Runpod equivalent | Notes | +| --- | --- | --- | +| EC2 GPU instance (G and P families) | [Pod](/pods/overview) | A Pod is a container, not a virtual machine. You pick a GPU type and count instead of an instance type. | +| AMI, Deep Learning AMI, user data | Container image, [template](/pods/templates/overview), start command | Bake setup into the image. Runpod publishes PyTorch images such as `runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04`. | +| Launch template with an Auto Scaling group | Serverless endpoint with min and max workers | Pods don't autoscale. Autoscaling is a Serverless feature. | +| SageMaker AI real-time endpoint | Serverless [load balancing endpoint](/serverless/load-balancing/overview) or queue endpoint | Load balancing endpoints run your own HTTP server with a `/ping` health check, the closest match to a SageMaker inference container. | +| SageMaker asynchronous inference, batch transform | Serverless queue endpoint (`/run` with a webhook) | Jobs queue, and workers scale to zero when idle. | +| SageMaker Serverless Inference | Serverless endpoint | SageMaker Serverless Inference doesn't support GPUs. Runpod Serverless runs on GPUs or CPUs. | +| SageMaker training and processing jobs | Pod, or `@app.task` in the [Runpod SDK](/sdk/overview) | `@app.task` creates one Pod per call that runs to completion and terminates. | +| SageMaker Studio, notebook instances | Pod with JupyterLab | Runpod official images start JupyterLab on port 8888. | +| SageMaker JumpStart, Amazon Bedrock | [Public Endpoints](/public-endpoints/overview) or a Serverless [vLLM endpoint](/serverless/vllm/overview) | Public Endpoints cover only the models in the [model reference](/public-endpoints/reference). For any other open-weight model, deploy vLLM. | +| AWS Batch, ECS on GPU, EKS GPU node groups | Serverless queue endpoint, Pods, or [SkyPilot](/integrations/skypilot) and [dstack](/integrations/dstack) | Run each container as an endpoint or a Pod. SkyPilot and dstack schedule jobs onto Runpod from their own control planes. | +| AWS ParallelCluster, SageMaker HyperPod | [Instant Clusters](/instant-clusters) with [Slurm](/instant-clusters/slurm-clusters) | Multi-node clusters with Slurm preinstalled. Capacity depends on current stock. HyperPod's EKS orchestration has no managed counterpart. | +| Amazon ECR | Any registry, or ECR with a [delegation](/tutorials/pods/use-private-ecr-images) | Runpod pulls private ECR images through cross-account access you grant on the repository. | +| EBS root volume | Container disk | Ephemeral. Erased when the Pod stops. | +| EBS data volume | Volume disk, or a [network volume](/storage/network-volumes) | A volume disk is deleted with its Pod. A network volume exists independently of compute. | +| Amazon EFS, FSx for Lustre | Network volume, [high-performance storage](/storage/high-performance-storage) | Pinned to one data center. | +| Amazon S3 | Keep S3, or a network volume with the [S3-compatible API](/storage/s3-api) | The Runpod S3-compatible API is a file interface to a network volume. It isn't a general object store. | +| Security groups, load balancers | [Exposed ports](/pods/configuration/expose-ports), Serverless endpoint URLs | HTTP ports get an HTTPS proxy URL. TCP ports get a public IP and mapped port. | +| VPC private networking | [Global networking](/pods/networking) between Pods | A private network across your own Pods. It doesn't peer with a VPC. | +| AWS Secrets Manager, SSM Parameter Store | [Secrets](/pods/templates/secrets) | Referenced from environment variables as `{{ RUNPOD_SECRET_name }}`. | +| Regions and Availability Zones | Data centers | List them with `runpodctl datacenter list`. | +| On-Demand, per-second billing | On-demand Pods and Serverless, per-second billing | No data ingress or egress fees on Runpod. | +| Spot Instances | On-demand Pods, or Serverless with scale to zero | runpodctl, REST API v2, and the Terraform provider create on-demand Pods. For interruptible batch work, use a Serverless queue endpoint that scales to zero between jobs. | +| Savings Plans, Capacity Reservations, Capacity Blocks for ML | Custom plans through Runpod sales | Runpod savings plans are deprecated and can't be purchased. See [Pod pricing](/pods/pricing). | +| Terraform AWS provider, CloudFormation | [Runpod Terraform provider](/integrations/terraform/overview) 2.x | Built on REST API v2. | +| AWS CLI, boto3, AWS APIs | [runpodctl](/runpodctl/overview), the [Runpod SDK](/sdk/overview), [REST API v2](/api-reference-v2/overview) | One API at `https://api.runpod.io/v2`. | +| IAM users, roles, SigV4 | [API keys](/get-started/credentials) sent as a Bearer token | Keys can be scoped as All, Restricted, or Read Only. | +| AWS Organizations, IAM Identity Center | Teams and [organizations](/organizations/orgs-overview) | Organizations have built-in roles (admin, billing, developer, contributor) and custom roles. | +| CloudWatch Logs and metrics | `runpodctl pod logs`, `runpodctl serverless logs`, endpoint metrics | Logs are read live from Pods and workers. To keep history, ship logs from your container to CloudWatch or another log service. | + +## What's different + +### What works differently + +**Pods are containers.** An EC2 instance boots an AMI with its own kernel, systemd, and Docker daemon. A Pod runs one container image. Docker Compose isn't available, UDP ports aren't exposed, and Windows images aren't supported. Images must be `linux/amd64`, so images built for Graviton (`g5g`) or other arm64 instances need a rebuild. + +**Setup moves from boot time to build time.** A user data script runs as root at first boot. On Runpod, put the same commands in a Dockerfile, and keep only the final long-running command as the container's start command. + +**Authentication is a Bearer token.** Calls to `sagemaker-runtime` and `bedrock-runtime` are signed with SigV4 from IAM credentials. Runpod calls are HTTPS requests with `Authorization: Bearer YOUR_API_KEY`. There is no request signing and no instance role. + +**Health checks are similar, ports aren't.** SageMaker sends `GET /ping` to port 8080 and expects a `200`. Runpod load balancing endpoints also poll `GET /ping` by default and treat `200` as healthy, `204` as initializing, and anything else as unhealthy. The default port is 80, set through the `PORT` and `PORT_HEALTH` environment variables. See [Step 4](#step-4-move-your-images-and-code). + +**Model artifacts aren't unpacked for you.** SageMaker copies `model.tar.gz` from S3 into `/opt/ml/model` before your container starts. On Runpod you bake the model into the image, attach a network volume, use [cached models](/serverless/endpoints/model-caching) for Hugging Face models, or download from S3 at startup. + +**Autoscaling is built into the endpoint.** SageMaker registers a production variant with Application Auto Scaling and attaches a policy. A Runpod endpoint has min workers, max workers, an idle timeout, and one scaler (queue delay or request count). Scale to zero is the default (min workers is 0). On SageMaker, scale to zero needs asynchronous inference, Serverless Inference, or inference components with a step scaling policy. + +**Timeouts differ.** A SageMaker real-time container must answer `/invocations` within 60 seconds. A Runpod load balancing endpoint allows 5.5 minutes of processing per request and 30 MB payloads. A queue endpoint's execution timeout defaults to 10 minutes in the console (5 minutes when you create the endpoint through REST API v2) and can be raised up to 7 days. + +**Storage is tied to a data center.** A network volume lives in one data center, and anything that mounts it runs there. Choose the data center by GPU stock first, then create the volume. + +**Billing is prepaid by default.** Runpod accounts hold a credit balance. When the balance reaches \$0, running Pods are stopped, and Pods without a network volume are terminated. Organizations, the enterprise account model, are invoiced post-paid instead. See [Pod pricing](/pods/pricing). + +### What has no direct equivalent + +On the Runpod side: + +- **VPC, security groups, PrivateLink.** Runpod has no VPC peering or private link into AWS. Exposed Pod ports are reachable from the internet, and the HTTP proxy URL is unauthenticated, so put your own auth in front of anything you expose. Workaround: call AWS over its public endpoints with scoped credentials, or call your own API gateway. +- **IAM roles and instance profiles.** A Pod or worker has no AWS identity. Workaround: create a narrowly scoped IAM user or role, store its keys as Runpod secrets, and inject them as `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. For single objects, pass presigned S3 URLs in the request instead of credentials. +- **KMS and customer-managed keys.** The Runpod S3-compatible API doesn't support bucket policies or ACLs, so there's nowhere to attach a KMS key policy. Workaround: keep regulated data in S3 under your KMS keys and read it at run time, or encrypt before writing to a volume. +- **Multi-model endpoints, production variants, shadow variants, Model Monitor.** One Runpod endpoint runs one image. Workaround: one endpoint per model, a router in your own code inside a load balancing worker, and [rolling releases](/serverless/endpoints/rolling-releases) for updates. +- **Bedrock features beyond inference** (Guardrails, Knowledge Bases, Agents) and models that are only offered through Bedrock. A vLLM endpoint serves open-weight models, and Public Endpoints serve the listed models. Keep the other calls on Bedrock. +- **Capacity Blocks, Capacity Reservations, and Savings Plans.** Runpod has no self-serve future-dated reservation, and its savings plans can no longer be purchased. A running on-demand Pod isn't displaced by other users, but a stopped Pod releases its GPU, so starting it again depends on stock at that moment. For committed capacity, talk to Runpod sales. +- **Spot Instances.** runpodctl, REST API v2, and the Terraform provider create on-demand Pods. Workaround: run Spot-style batch work on on-demand Pods that you delete when the job ends, or on a Serverless queue endpoint that scales to zero. Keep checkpointing to a network volume so a job can resume on a new Pod. +- **Compliance controls as IAM policy.** Runpod expresses placement as deploy-time filters instead: Secure Cloud runs in T3 and T4 data centers with partners that hold SOC 2, ISO 27001, and PCI DSS certifications, and `runpodctl pod create` accepts `--country-code` and `--compliance` to restrict where a Pod lands. See [Security and compliance](/references/security-and-compliance). + +On the AWS side: + +- **Per-second GPU Serverless with scale to zero as the default.** SageMaker Serverless Inference doesn't support GPUs. +- **No data transfer fees.** Runpod doesn't charge for ingress or egress. AWS bills data transferred out of S3 and EC2 to the internet. +- **Cached models.** Runpod can cache a Hugging Face model on the host so workers load it without baking it into the image. See [cached models](/serverless/endpoints/model-caching). + +## Requirements + +- An AWS identity that can list and describe EC2, SageMaker, ECR, S3, CloudWatch Logs, and Application Auto Scaling resources, plus permission to delete them at the end. +- The [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) configured for each account and Region you use. +- A [Runpod account](https://www.console.runpod.io) with credits and an [API key](/get-started/credentials). +- Docker, to rebuild or retag images for `linux/amd64`. +- Terraform 1.6 or later, if you manage infrastructure as code. + +## Step 1: Take inventory of what you run on AWS + +List every GPU resource before you change anything. Run the commands once per Region. + +List running GPU instances with their type, AMI, and Availability Zone: + +```bash +aws ec2 describe-instances \ + --filters "Name=instance-type,Values=g4dn.*,g5.*,g5g.*,g6.*,g6e.*,g7.*,g7e.*,p3dn.*,p4d.*,p4de.*,p5.*,p5e.*,p5en.*,p6*" \ + "Name=instance-state-name,Values=pending,running,stopping,stopped" \ + --query "Reservations[].Instances[].[InstanceId,InstanceType,ImageId,State.Name,Placement.AvailabilityZone,Tags[?Key=='Name']|[0].Value]" \ + --output table +``` + +Stopped instances appear too, because their EBS volumes still bill. For each instance, export the user data script, which is your setup recipe: + +```bash +aws ec2 describe-instance-attribute --instance-id i-0123456789abcdef0 \ + --attribute userData --query "UserData.Value" --output text | base64 --decode +``` + +List launch templates and Auto Scaling groups, which hold the same settings for fleets: + +```bash +aws ec2 describe-launch-templates --query "LaunchTemplates[].[LaunchTemplateName,LaunchTemplateId]" --output table +aws autoscaling describe-auto-scaling-groups \ + --query "AutoScalingGroups[].[AutoScalingGroupName,MinSize,MaxSize,DesiredCapacity]" --output table +``` + +List SageMaker endpoints, then walk each one from endpoint to endpoint configuration to model. The model holds the image URI, the model artifact location, and the environment variables: + +```bash +aws sagemaker list-endpoints --query "Endpoints[].[EndpointName,EndpointStatus]" --output table +aws sagemaker describe-endpoint --endpoint-name my-endpoint --query "EndpointConfigName" +aws sagemaker describe-endpoint-config --endpoint-config-name my-endpoint-config \ + --query "{variants:ProductionVariants,async:AsyncInferenceConfig}" +aws sagemaker describe-model --model-name my-model \ + --query "PrimaryContainer.{image:Image,model:ModelDataUrl,env:Environment}" +``` + +An `AsyncInferenceConfig` in the output means the endpoint is asynchronous and maps to a queue endpoint. Export the autoscaling settings for each variant: + +```bash +aws application-autoscaling describe-scalable-targets --service-namespace sagemaker +aws application-autoscaling describe-scaling-policies --service-namespace sagemaker +``` + +List the remaining SageMaker and Bedrock resources that bill while idle: + +```bash +aws sagemaker list-notebook-instances --query "NotebookInstances[].[NotebookInstanceName,InstanceType,NotebookInstanceStatus]" --output table +aws sagemaker list-apps --query "Apps[].[DomainId,AppType,AppName,Status]" --output table +aws sagemaker list-training-jobs --status-equals InProgress +aws bedrock list-provisioned-model-throughputs +aws bedrock list-imported-models +``` + +List images, storage, and cluster resources: + +```bash +aws ecr describe-repositories --query "repositories[].repositoryUri" --output text +aws ec2 describe-volumes --query "Volumes[].[VolumeId,Size,State,Attachments[0].InstanceId]" --output table +aws efs describe-file-systems --query "FileSystems[].[FileSystemId,Name,SizeInBytes.Value]" --output table +aws fsx describe-file-systems --query "FileSystems[].[FileSystemId,FileSystemType,StorageCapacity]" --output table +aws batch describe-compute-environments --query "computeEnvironments[].computeEnvironmentName" +aws ecs list-clusters +aws eks list-clusters +``` + +Then find the code that calls AWS. This search covers boto3 clients, the SageMaker Python SDK, the SageMaker container contract, CLI calls, and Terraform: + +```bash +grep -rnE "sagemaker-runtime|invoke_endpoint|bedrock-runtime|\.converse\(|invoke_model|from sagemaker|model_fn|/invocations|/opt/ml/|aws (ec2|sagemaker|ecr|s3) |dkr\.ecr\.|169\.254\.169\.254|aws_sagemaker_|aws_instance|aws_launch_template" \ + --include="*.py" --include="*.ts" --include="*.js" --include="*.go" --include="*.sh" \ + --include="*.tf" --include="*.yaml" --include="*.yml" --include="Dockerfile*" . +``` + +A match on `169.254.169.254` is a call to the EC2 instance metadata service, which doesn't exist on Runpod. Record, for each workload: image, start command, environment variables, ports, disk sizes, GPU model and count, Region, autoscaling limits, and where its data lives. + +## Step 2: Set up Runpod + +Create an account, add credits, and [create an API key](/get-started/credentials#create-an-api-key). Then install the tools. + +```bash +# runpodctl +curl -fsSL https://cli.runpod.net | sudo bash + +# Runpod SDK (Python 3.10 to 3.14), which includes the rp CLI +pip install runpod + +# Authenticate both +export RUNPOD_API_KEY="YOUR_API_KEY" +rp login +``` + +`rp login` stores credentials in `~/.runpod/config.toml`. On macOS you can install runpodctl with `brew install runpod/runpodctl/runpodctl`. To let an AI tool manage resources, connect the [Runpod MCP server](/get-started/mcp-servers). For infrastructure as code, see [Migrate your Terraform](#migrate-your-terraform). + +Check which GPUs and data centers have stock before you plan placements: + +```bash +runpodctl gpu list +runpodctl datacenter list +``` + +## Step 3: Choose where each workload lands + +| AWS workload | Runpod target | Why | +| --- | --- | --- | +| EC2 GPU instance you SSH into, long training runs, notebooks | Pod | Persistent container with SSH, JupyterLab, and a volume. | +| SageMaker real-time endpoint with a custom container (`/invocations` and `/ping`) | Serverless load balancing endpoint | Your HTTP server runs unchanged apart from ports. | +| SageMaker real-time endpoint built on `inference.py` (`model_fn`, `predict_fn`) | Serverless queue endpoint | A handler wraps the same functions. You get queueing and retries. | +| SageMaker asynchronous inference, batch transform, AWS Batch array jobs | Serverless queue endpoint | Submit with `/run`, receive results by webhook, scale to zero between batches. | +| SageMaker training or processing job launched from code | `@app.task` in the Runpod SDK, or a Pod | One ephemeral Pod per call. | +| Bedrock on-demand calls to an open-weight model | Public Endpoint, if the model is [listed](/public-endpoints/reference) | No deployment to manage. | +| Bedrock Provisioned Throughput, Custom Model Import, JumpStart LLM endpoints | Serverless vLLM endpoint | OpenAI-compatible API on your own workers, with your own weights. | +| ParallelCluster or HyperPod Slurm cluster | Instant Cluster with Slurm | Multi-node, high-speed networking between nodes. | +| ECS or EKS GPU services | Serverless endpoint, or Pods through SkyPilot or dstack | Pick Serverless when the service is request driven. | +| Lambda functions | Keep on AWS | Move a function only when it's glue next to a GPU workload and benefits from running beside it, as a CPU Serverless endpoint. | + +Choose queue endpoints when every request must be processed, jobs run long, or traffic arrives in bursts. Choose load balancing endpoints for low-latency request and response traffic with your own routes. Load balancing endpoints don't queue: requests are dropped when all workers are busy, and there's no built-in retry. + +### Map EC2 instance types to Runpod GPUs + +Runpod selects GPUs by type ID, and Serverless endpoints select by pool. The GPU details below come from the EC2 instance type specifications and the [Runpod GPU reference](/references/gpu-types). + +| EC2 instance family | GPU (per instance) | Runpod GPU type ID | Serverless pool | +| --- | --- | --- | --- | +| `g4dn` | NVIDIA T4 16 GB (1, 4, or 8) | `NVIDIA RTX A4000`, `NVIDIA RTX 2000 Ada Generation` (16 GB), or `NVIDIA L4` (24 GB) | `AMPERE_16`, `AMPERE_24` | +| `g5` | NVIDIA A10G 24 GB (1, 4, or 8) | `NVIDIA RTX A5000`, `NVIDIA L4`, `NVIDIA GeForce RTX 4090` | `AMPERE_24`, `ADA_24` | +| `g5g` | NVIDIA T4g 16 GB (1 or 2), Graviton | Same as `g4dn`, after an amd64 rebuild | `AMPERE_16` | +| `g6`, `gr6` | NVIDIA L4 24 GB (`g6`: 1, 4, or 8; `gr6`: 1) | `NVIDIA L4` | `AMPERE_24` | +| `g6e` | NVIDIA L40S 48 GB (1, 4, or 8) | `NVIDIA L40S` | `ADA_48_PRO` | +| `g7` | NVIDIA RTX PRO 4500 32 GB (1, 2, 4, or 8) | `NVIDIA RTX PRO 4500 Blackwell` | Check `runpodctl gpu list` | +| `g7e` | NVIDIA RTX PRO 6000 96 GB (1, 2, 4, or 8) | `NVIDIA RTX PRO 6000 Blackwell Server Edition` | Check `runpodctl gpu list` | +| `p3dn` | NVIDIA V100 32 GB (8) | `Tesla V100-SXM2-32GB`, or a 48 GB type such as `NVIDIA RTX A6000` | `AMPERE_48` (for the 48 GB types) | +| `p4d` | NVIDIA A100 40 GB (8) | `NVIDIA A100 80GB PCIe`, `NVIDIA A100-SXM4-80GB` | `AMPERE_80` | +| `p4de` | NVIDIA A100 80 GB (8) | `NVIDIA A100 80GB PCIe`, `NVIDIA A100-SXM4-80GB` | `AMPERE_80` | +| `p5` | NVIDIA H100 80 GB (1 or 8) | `NVIDIA H100 80GB HBM3`, `NVIDIA H100 PCIe` | `ADA_80_PRO` | +| `p5e`, `p5en` | NVIDIA H200 141 GB (8) | `NVIDIA H200` | `HOPPER_141` | +| `p6-b200` | NVIDIA B200 (8) | `NVIDIA B200` | Check `runpodctl gpu list` | +| `p6-b300` | NVIDIA B300 (8) | `NVIDIA B300 SXM6 AC` | Check `runpodctl gpu list` | +| `p6e-gb200` | NVIDIA B200 (4), Arm CPU | `NVIDIA B200`, after an amd64 rebuild | Check `runpodctl gpu list` | + +The EC2 specifications list the A10G and L4 at 22 GiB of accelerator memory and the L40S at 44 GiB. The fractional-GPU `g6f` sizes and the AMD Radeon `g4ad` family have no like-for-like type. Use a full 16 GB or 24 GB GPU. Choose by fit for the workload: the VRAM the model needs, then throughput. + +On EC2 the instance type fixes vCPUs, RAM, and GPU count together. On Runpod you set the GPU type and `--gpu-count`, and vCPU and RAM come with the GPU. Stock varies by data center and changes through the day, so run `runpodctl gpu list` before you commit to a type, and give Serverless endpoints more than one pool where your model allows it. For an 8-GPU node or several nodes working together, check stock for that GPU count or use an [Instant Cluster](/instant-clusters). + +## Step 4: Move your images and code + +### Make images portable + +Runpod runs `linux/amd64` images from any registry. Build with the platform flag, especially on Apple silicon or when the image was built for Graviton: + +```bash +docker build --platform linux/amd64 -t YOUR_REGISTRY/my-worker:v1.0.0 . +docker push YOUR_REGISTRY/my-worker:v1.0.0 +``` + +Use a version tag or digest instead of `:latest`. If your image is built on an AWS Deep Learning Container, it runs on Runpod as long as it doesn't depend on SageMaker starting it with the `serve` or `train` argument. To drop the AWS base, start from a Runpod PyTorch image or an `nvidia/cuda` image. The host provides the NVIDIA driver, as it does on SageMaker, so don't install drivers in the image. Set a CUDA floor at deploy time with `--min-cuda-version` so workers land on hosts that match your framework build. + +### Keep images in ECR + +You can leave images in a private ECR repository. Runpod pulls them through an ECR delegation: you grant two fixed Runpod IAM roles pull access on the repository, then register the repository with Runpod. There are no tokens to rotate. + + + + Apply this repository policy. The two role ARNs belong to Runpod and must be used exactly as shown. + + ```bash + aws ecr set-repository-policy \ + --repository-name my-worker \ + --policy-text '{ + "Version": "2012-10-17", + "Statement": [{ + "Sid": "AllowRunpodPull", + "Effect": "Allow", + "Principal": "*", + "Action": [ + "ecr:GetAuthorizationToken", + "ecr:BatchCheckLayerAvailability", + "ecr:GetDownloadUrlForLayer", + "ecr:BatchGetImage" + ], + "Condition": { + "StringEquals": { + "aws:PrincipalArn": [ + "arn:aws:iam::550005742258:role/prod-us-east-1-deployment-role", + "arn:aws:iam::550005742258:role/prod-us-west-2-deployment-role" + ] + } + } + }] + }' + ``` + + If your AWS organization uses service control policies, confirm they don't block `ecr:GetAuthorizationToken`. + + + Register the repository ARN with REST API v2. Runpod validates the policy at this point, so the call fails if step 1 is missing. + + ```bash + curl -s -X POST https://api.runpod.io/v2/registries/delegations \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"name": "my-worker-ecr", "resource": "arn:aws:ecr:us-east-1:123456789012:repository/my-worker"}' + ``` + + In the console, the same step is **Settings**, then **Container Registry Authentication**, **Add Credential**, and the **AWS ECR** tab, where you enter the full image URI including a tag. In Terraform, use `runpod_registry_delegation`. + + + Use `123456789012.dkr.ecr.us-east-1.amazonaws.com/my-worker:v1.0.0` as the image, and leave the registry credential unset. Runpod matches the image to the delegation. Any tag in a registered repository works. + + + +The [private ECR tutorial](/tutorials/pods/use-private-ecr-images) walks through this for Pods. The REST API v2 reference describes delegations as covering Pods and Serverless endpoints, so test a Serverless deploy from ECR before you depend on it. For other private registries, create a credential with `runpodctl registry create` and pass its ID as `--registry-auth-id`. + + +Don't store the output of `aws ecr get-login-password` as a Runpod registry credential. An ECR authorization token is valid for 12 hours, so image pulls for new workers start failing after it expires. + + +### Translate a SageMaker `inference.py` to a queue endpoint + +The SageMaker framework containers load a script that defines `model_fn`, `input_fn`, `predict_fn`, and `output_fn`. Here is a typical PyTorch script: + +```python inference.py +import json +import os + +import torch + + +def model_fn(model_dir): + model = torch.jit.load(os.path.join(model_dir, "model.pt"), map_location="cuda") + model.eval() + return model + + +def input_fn(request_body, request_content_type): + if request_content_type != "application/json": + raise ValueError(f"Unsupported content type: {request_content_type}") + return torch.tensor(json.loads(request_body)["inputs"], device="cuda") + + +def predict_fn(input_data, model): + with torch.no_grad(): + return model(input_data) + + +def output_fn(prediction, content_type): + return json.dumps({"predictions": prediction.cpu().tolist()}) +``` + +On Runpod, a handler calls the same four functions. Keep `inference.py` as it is and add this file: + +```python handler.py +import json +import os + +import runpod + +from inference import input_fn, model_fn, output_fn, predict_fn + +# Load once per worker, outside the handler, so warm requests skip it. +model = model_fn(os.environ.get("MODEL_DIR", "/opt/ml/model")) + + +def handler(job): + body = json.dumps(job["input"]) + data = input_fn(body, "application/json") + prediction = predict_fn(data, model) + return json.loads(output_fn(prediction, "application/json")) + + +runpod.serverless.start({"handler": handler}) +``` + +The request body that SageMaker passed as raw bytes arrives as `job["input"]`, already parsed from JSON. The handler's return value becomes the `output` field of the response. Package it with the model files that SageMaker used to unpack from `model.tar.gz`: + +```bash +aws s3 cp s3://my-bucket/models/model.tar.gz . +mkdir -p model && tar -xzf model.tar.gz -C model +``` + +```dockerfile Dockerfile +FROM runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04 + +WORKDIR /app +RUN pip install --no-cache-dir runpod + +# SageMaker unpacked model.tar.gz here at start. Bake it in instead. +COPY model/ /opt/ml/model/ +COPY inference.py handler.py /app/ + +CMD ["python", "-u", "/app/handler.py"] +``` + +Baking the model into the image keeps `/opt/ml/model` valid, so `inference.py` doesn't change. For large models, put the files on a network volume and set `MODEL_DIR=/runpod-volume/my-model` instead. If `model.tar.gz` contains `code/requirements.txt`, install it in the Dockerfile. + +### Run a SageMaker inference container on a load balancing endpoint + +A bring-your-own SageMaker container already serves `POST /invocations` and `GET /ping` on port 8080. That contract lines up with a Runpod load balancing endpoint: + +| | SageMaker real-time container | Runpod load balancing endpoint | +| --- | --- | --- | +| Health check | `GET /ping`, `200` means ready | `GET /ping` by default (`HEALTH_CHECK_PATH`), `200` healthy, `204` initializing, other unhealthy | +| Port | 8080, fixed | `PORT` and `PORT_HEALTH`, each default 80 | +| Inference route | `POST /invocations` | Any path your server defines, including `/invocations` | +| Start command | `docker run IMAGE serve` | The image's own command, or one you set | +| Model files | Copied to `/opt/ml/model` | Yours to provide | +| Startup | Must pass health checks within 8 minutes | Misconfigured ports leave workers up for 8 minutes returning `502` | + +To run the container unchanged, set three things when you create the endpoint: `PORT=8080` and `PORT_HEALTH=8080` as environment variables, `8080/http` as the exposed port, and `serve` as the container command if your entrypoint dispatches on that argument. If `/ping` returns `200` before the model is loaded, change it to return `204` until the model is ready, so Runpod doesn't route requests to a worker that can't serve them. This minimal server shows the contract: + +```python serve.py +import os +import threading + +from fastapi import FastAPI, Request, Response + +app = FastAPI() +model = None + + +def load_model(): + global model + import torch + + loaded = torch.jit.load(os.path.join(os.environ.get("MODEL_DIR", "/opt/ml/model"), "model.pt")) + loaded.eval() + model = loaded + + +@app.on_event("startup") +def start_loading(): + # Load in the background so the server can answer /ping while the model loads. + threading.Thread(target=load_model, daemon=True).start() + + +@app.get("/ping") +def ping(): + # 204 while loading, 200 when ready. + return Response(status_code=200 if model is not None else 204) + + +@app.post("/invocations") +async def invocations(request: Request): + import torch + + payload = await request.json() + with torch.no_grad(): + prediction = model(torch.tensor(payload["inputs"])) + return {"predictions": prediction.tolist()} + + +if __name__ == "__main__": + import uvicorn + + uvicorn.run(app, host="0.0.0.0", port=int(os.environ.get("PORT", "8080"))) +``` + +Callers reach it at `https://ENDPOINT_ID.api.runpod.ai/invocations` with a Bearer token. Runpod checks the API key before the request reaches your server. + +### Turn an EC2 user data script into a Pod image + +A GPU instance launched from a Deep Learning AMI often has a user data script like this one: + +```bash user-data.sh +#!/bin/bash +# Runs as root on first boot of a g6e.xlarge. +aws s3 sync s3://my-bucket/checkpoints /opt/checkpoints +pip install vllm +MODEL=/opt/checkpoints/my-model +vllm serve "$MODEL" --host 0.0.0.0 --port 8000 +``` + +On Runpod, the install steps move into the image, and the commands that need run-time data stay in a start script: + +```dockerfile Dockerfile +FROM runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04 + +RUN pip install --no-cache-dir vllm awscli +COPY start.sh /opt/app/start.sh +CMD ["bash", "/opt/app/start.sh"] +``` + +```bash start.sh +#!/bin/bash +set -euo pipefail +# AWS credentials arrive as environment variables from Runpod secrets. +aws s3 sync s3://my-bucket/checkpoints /workspace/checkpoints +exec vllm serve /workspace/checkpoints/my-model --host 0.0.0.0 --port 8000 +``` + +The script ends with `exec` so the server is the container's main process. Checkpoints go to `/workspace`, the volume mount, so they survive a Pod stop and the next start skips the download. The instance profile that gave the EC2 instance S3 access is replaced by scoped IAM keys stored as Runpod secrets. Step 6 creates the Pod. + +### Move training jobs + +A SageMaker training container reads data from `/opt/ml/input/data/CHANNEL`, reads hyperparameters from `/opt/ml/input/config/hyperparameters.json`, and writes the model to `/opt/ml/model`, which SageMaker uploads to S3. On Runpod, pass hyperparameters as environment variables or function arguments, read data from a network volume or sync it from S3 at start, and upload results yourself. With the Runpod SDK, a training job is a decorated function that runs on its own Pod and terminates when it returns: + +```python train.py +import runpod + +app = runpod.App("training") +checkpoints = runpod.NetworkVolume("checkpoints", size=100) + + +@app.task(gpu="H100", mounts={"/checkpoints": checkpoints}, dependencies=["torch"]) +def train(epochs: int = 3, learning_rate: float = 3e-4): + import torch + + # Your training loop. Write checkpoints under checkpoints.path. + run_dir = checkpoints.path / "run-1" + run_dir.mkdir(parents=True, exist_ok=True) + torch.save({"epochs": epochs, "lr": learning_rate}, run_dir / "model.pt") + return {"checkpoint": str(run_dir)} + + +@runpod.local_entrypoint +def main(): + print(train.remote(epochs=5)) +``` + +Run it with `rp flash dev train.py`. See [Build an app](/sdk/apps/build-app) for the full set of options. + +## Step 5: Move your data and models + +You don't have to move data. A Pod or worker can read and write S3 over the public internet with IAM keys you store as secrets. Move data to Runpod when load time or transfer cost matters, which is usually true for model weights and training sets. + + +AWS charges for data transferred out of S3 to the internet. Copying a dataset to Runpod is a one-time transfer, so estimate it from your bucket size and the [Amazon S3 pricing page](https://aws.amazon.com/s3/pricing/) before you start. Runpod doesn't charge for ingress or egress. In a hybrid setup, every read from S3 at run time is also egress, which is a reason to keep hot data on a network volume. + + +| AWS storage | Runpod target | How to move it | +| --- | --- | --- | +| EBS root volume | Container disk | Rebuild as an image. Don't copy the disk. | +| EBS data volume, instance store | Volume disk or network volume | Copy the files from the instance to S3, then sync into a Pod. | +| EFS, FSx for Lustre | Network volume, or a [high-performance](/storage/high-performance-storage) network volume | Sync through S3, or `rsync` over SSH from an instance that mounts the file system. | +| S3 model artifacts and datasets | Network volume, or leave in S3 | `aws s3 sync` from inside a Pod. | +| Hugging Face models you mirrored to S3 | [Cached models](/serverless/endpoints/model-caching) | Reference the Hugging Face repo with `--model-reference` instead of copying. | + +Create the network volume in the data center where you plan to run, since everything that mounts it must run there: + +```bash +runpodctl network-volume create --name training-data --size 500 --data-center-id US-GA-2 +``` + +The most direct transfer runs inside a Pod with the volume attached, because data flows from S3 straight to Runpod. Create the IAM keys as [secrets](/pods/templates/secrets) named `aws_access_key_id` and `aws_secret_access_key`, start a Pod that maps them to environment variables, then sync: + +```bash +runpodctl pod create --name s3-transfer \ + --image runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04 \ + --gpu-id "NVIDIA RTX A4000" \ + --network-volume-id VOLUME_ID \ + --env '{"AWS_ACCESS_KEY_ID":"{{ RUNPOD_SECRET_aws_access_key_id }}","AWS_SECRET_ACCESS_KEY":"{{ RUNPOD_SECRET_aws_secret_access_key }}","AWS_DEFAULT_REGION":"us-east-1"}' + +# Then, in the Pod's terminal: +pip install awscli +aws s3 sync s3://my-bucket/datasets/imagenet /workspace/datasets/imagenet +``` + +Delete the transfer Pod when the sync finishes. The volume keeps the data. There are three other routes: + +- **Cloud Sync.** On a Pod's page in the console, **Cloud Sync** copies between the Pod and an S3 bucket using an access key, secret key, Region, and bucket path. See [Sync data with cloud storage](/pods/storage/cloud-sync). +- **runpodctl.** `runpodctl send` on your machine and `runpodctl receive` on a Pod copy files with a one-time code. This suits small and medium transfers. See [Transfer files](/pods/storage/transfer-files), which also covers `scp` and `rsync` over SSH. +- **The S3-compatible API.** Point the AWS CLI at the volume to upload from your machine without a Pod, as shown below. It needs Runpod S3 API keys and a data center that supports the API. + +```bash +aws s3 cp --recursive ./models/ s3://VOLUME_ID/models/ \ + --region US-GA-2 \ + --endpoint-url https://s3api-us-ga-2.runpod.io/ +``` + + +The Runpod S3-compatible API is not Amazon S3. It uses a per-data-center endpoint, separate S3 API keys from the Runpod console, the data center ID as the Region, and the volume ID as the bucket. Presigned URLs, versioning, bucket policies, ACLs, and object tagging aren't supported, and `aws s3 sync` can run into issues on directories with more than 10,000 files. Code that needs those features should keep using S3. See the [compatibility reference](/storage/s3-api#s3-api-compatibility-reference). + + +Network volumes mount at `/workspace` on Pods and `/runpod-volume` on Serverless workers, so paths in your code may need a prefix change between the two. + +## Step 6: Deploy on Runpod + +### Deploy a Pod + +Create the Pod from the image you built in Step 4. This replaces `aws ec2 run-instances` with a user data file: + + + + ```bash + aws ec2 run-instances \ + --image-id ami-0123456789abcdef0 \ + --instance-type g6e.xlarge \ + --key-name my-key \ + --security-group-ids sg-0123456789abcdef0 \ + --iam-instance-profile Name=s3-read-profile \ + --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs":{"VolumeSize":200}}]' \ + --user-data file://user-data.sh + ``` + + + ```bash + runpodctl pod create --name vllm-server \ + --image YOUR_REGISTRY/vllm-server:v1.0.0 \ + --gpu-id "NVIDIA L40S" \ + --container-disk-in-gb 50 \ + --volume-in-gb 200 \ + --ports "8000/http" \ + --min-cuda-version 12.8 \ + --env '{"AWS_ACCESS_KEY_ID":"{{ RUNPOD_SECRET_aws_access_key_id }}","AWS_SECRET_ACCESS_KEY":"{{ RUNPOD_SECRET_aws_secret_access_key }}","AWS_DEFAULT_REGION":"us-east-1"}' + ``` + + + +The security group becomes the `--ports` list, the instance profile becomes two secret-backed environment variables, and the root volume size splits into a container disk and a persistent volume at `/workspace`. The server is reachable at `https://POD_ID-8000.proxy.runpod.net`. That URL is public and has a 100-second request timeout, so turn on your server's own API key check before you send it real traffic. Check status and logs with: + +```bash +runpodctl pod get POD_ID +runpodctl pod logs POD_ID --follow +``` + +To override the image's command at deploy time, use `--docker-args` in runpodctl or the `cmd` field in REST API v2. In the console, deploy from **Pods**, then **Deploy**. + +### Deploy a queue endpoint + +Push the handler image from Step 4, create a Serverless template for it, then create the endpoint: + +```bash +runpodctl template create --name sm-migrated-worker \ + --image YOUR_REGISTRY/sm-migrated-worker:v1.0.0 \ + --container-disk-in-gb 20 \ + --serverless + +runpodctl serverless create --name sm-migrated \ + --template-id TEMPLATE_ID \ + --gpu-id AMPERE_24 \ + --workers-min 0 --workers-max 3 \ + --idle-timeout 30 \ + --min-cuda-version 12.8 +``` + +`--workers-min` and `--workers-max` replace the `MinCapacity` and `MaxCapacity` of the Application Auto Scaling target. With `--workers-min 0` the endpoint bills nothing while idle. Set `--workers-min 1` to keep one worker warm, as an always-on SageMaker instance did. Send a test job: + +```bash +runpodctl serverless run ENDPOINT_ID --input '{"inputs": [[1.0, 2.0, 3.0]]}' +``` + +### Deploy a load balancing endpoint + +`runpodctl serverless create` creates queue endpoints, so create load balancing endpoints with REST API v2, Terraform, or the console (**Serverless**, **New Endpoint**, then **Load Balancer** under **Endpoint Type**). This request deploys the SageMaker container from Step 4 on port 8080: + +```bash +curl -s -X POST https://api.runpod.io/v2/serverless \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "sm-realtime", + "type": "LOAD_BALANCER", + "image": "123456789012.dkr.ecr.us-east-1.amazonaws.com/my-worker:v1.0.0", + "cmd": ["serve"], + "ports": ["8080/http"], + "env": {"PORT": "8080", "PORT_HEALTH": "8080"}, + "disk": 20, + "gpu": {"pools": ["AMPERE_24"], "count": 1, "minCudaVersion": "12.8"}, + "workers": {"min": 0, "max": 3, "idleTimeout": 30}, + "scaling": {"type": "REQUEST_COUNT", "requestCount": 4}, + "timeout": 600000 + }' +``` + +Load balancing endpoints accept only the `REQUEST_COUNT` scaler. The response includes `requestUrls.base`, which is the URL your callers use. Remove `"cmd": ["serve"]` if your image starts its server without that argument. The endpoint type can't be changed after creation. + +### Deploy an LLM endpoint + +For Bedrock and JumpStart replacements, check the [Public Endpoint model reference](/public-endpoints/reference) first. If your model is listed, there is nothing to deploy. Otherwise deploy the vLLM worker from the Hub with your model: + +```bash +runpodctl hub search vllm +runpodctl serverless create --hub-id HUB_ID --name my-llm \ + --env MODEL_NAME=meta-llama/Llama-3.1-8B-Instruct +``` + +The endpoint serves an OpenAI-compatible API at `https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1`. For gated models, add a Hugging Face token. See [vLLM configuration](/serverless/vllm/configuration) for the environment variables. + +### Deploy a cluster + +Create Slurm clusters from the [Instant Clusters page](https://console.runpod.io/cluster) by choosing **Slurm Cluster** as the cluster type, or with the `runpod_cluster` Terraform resource. Slurm and munge are preinstalled and controller and agent roles are assigned for you. Review your `sbatch` scripts for paths, modules, and partition names that came from the ParallelCluster configuration. Cluster capacity depends on stock at the time you create it. + +## Step 7: Update the code that calls AWS + +### Replace SageMaker runtime calls + +A real-time endpoint call moves from a SigV4-signed boto3 call to an HTTPS request with a Bearer token. + + + + ```python + import json + + import boto3 + + runtime = boto3.client("sagemaker-runtime", region_name="us-east-1") + + response = runtime.invoke_endpoint( + EndpointName="my-endpoint", + ContentType="application/json", + Body=json.dumps({"inputs": [[1.0, 2.0, 3.0]]}), + ) + result = json.loads(response["Body"].read().decode("utf-8")) + print(result["predictions"]) + ``` + + + ```python + import os + + import runpod + + runpod.api_key = os.environ["RUNPOD_API_KEY"] + endpoint = runpod.Endpoint(os.environ["ENDPOINT_ID"]) + + result = endpoint.run_sync({"inputs": [[1.0, 2.0, 3.0]]}, timeout=60) + print(result["predictions"]) + ``` + + + ```python + import os + + import requests + + response = requests.post( + f"https://{os.environ['ENDPOINT_ID']}.api.runpod.ai/invocations", + headers={"Authorization": f"Bearer {os.environ['RUNPOD_API_KEY']}"}, + json={"inputs": [[1.0, 2.0, 3.0]]}, + timeout=120, + ) + response.raise_for_status() + print(response.json()["predictions"]) + ``` + + + +On a queue endpoint the raw HTTP form is `POST https://api.runpod.ai/v2/ENDPOINT_ID/runsync` with the body `{"input": {...}}`, and the result is under `output` in the response. `/runsync` accepts payloads up to 20 MB. A load balancing endpoint that has scaled to zero can return an error while the first worker starts, so retry with a delay, or keep one worker warm. + +### Replace asynchronous inference + +SageMaker asynchronous inference reads the request from S3, writes the result to S3, and notifies through SNS. A queue endpoint takes the request in the call, and posts the result to a webhook. + + + + ```python + import boto3 + + runtime = boto3.client("sagemaker-runtime", region_name="us-east-1") + + response = runtime.invoke_endpoint_async( + EndpointName="my-async-endpoint", + InputLocation="s3://my-bucket/requests/job-123.json", + ContentType="application/json", + ) + print(response["OutputLocation"]) + ``` + + + ```python + import os + + import requests + + response = requests.post( + f"https://api.runpod.ai/v2/{os.environ['ENDPOINT_ID']}/run", + headers={"Authorization": f"Bearer {os.environ['RUNPOD_API_KEY']}"}, + json={ + "input": {"input_url": "https://my-bucket.s3.amazonaws.com/requests/job-123.json?X-Amz-Signature=..."}, + "webhook": "https://api.example.com/runpod/webhook", + "policy": {"executionTimeout": 3600000}, + }, + timeout=30, + ) + print(response.json()["id"]) + ``` + + + +`/run` accepts payloads up to 10 MB, so keep large inputs in S3 and pass a presigned URL, which your handler downloads. Have the handler upload large outputs to S3 with a presigned PUT URL or scoped keys, and return the object location. The webhook receives the job result when it finishes and should return `200`. Without a webhook, poll `GET /v2/ENDPOINT_ID/status/JOB_ID`. Results are kept for 30 minutes after completion. For batch transform, submit one `/run` job per record or per shard and let the endpoint scale out to its max workers. + +### Replace Bedrock calls + +The Converse API has its own request shape. Runpod's vLLM workers and LLM Public Endpoints use the OpenAI Chat Completions shape, so the request needs translating. + + + + ```python + import boto3 + + client = boto3.client("bedrock-runtime", region_name="us-east-1") + + response = client.converse( + modelId="BEDROCK_MODEL_ID", + system=[{"text": "You are a concise assistant."}], + messages=[{"role": "user", "content": [{"text": "Name three GPU vendors."}]}], + inferenceConfig={"maxTokens": 256, "temperature": 0.2, "topP": 0.9}, + ) + print(response["output"]["message"]["content"][0]["text"]) + print(response["usage"]["inputTokens"], response["usage"]["outputTokens"]) + ``` + + + ```python + import os + + from openai import OpenAI + + client = OpenAI( + base_url=f"https://api.runpod.ai/v2/{os.environ['ENDPOINT_ID']}/openai/v1", + api_key=os.environ["RUNPOD_API_KEY"], + ) + + response = client.chat.completions.create( + model="meta-llama/Llama-3.1-8B-Instruct", + messages=[ + {"role": "system", "content": "You are a concise assistant."}, + {"role": "user", "content": "Name three GPU vendors."}, + ], + max_tokens=256, + temperature=0.2, + top_p=0.9, + ) + print(response.choices[0].message.content) + print(response.usage.prompt_tokens, response.usage.completion_tokens) + ``` + + + +| Converse field | Chat Completions field | +| --- | --- | +| `modelId` | `model` (the Hugging Face model name the endpoint serves) | +| `system: [{"text": ...}]` | A message with `"role": "system"` | +| `messages[].content: [{"text": ...}]` | `messages[].content` as a string | +| `inferenceConfig.maxTokens`, `temperature`, `topP`, `stopSequences` | `max_tokens`, `temperature`, `top_p`, `stop` | +| `output.message.content[0].text` | `choices[0].message.content` | +| `stopReason` | `choices[0].finish_reason` | +| `usage.inputTokens`, `usage.outputTokens` | `usage.prompt_tokens`, `usage.completion_tokens` | +| `converse_stream` | `stream=True` | + +For a Public Endpoint, use the model's slug in place of `ENDPOINT_ID`, for example `https://api.runpod.ai/v2/qwen3-32b-awq/openai/v1`. If you already call Bedrock through its OpenAI-compatible Chat Completions endpoint, the change is the base URL, the API key, and the model name. Code that uses `invoke_model` with a model's native body needs the same translation as Converse, starting from that model's request format. Tool use and model-specific fields depend on the model and vLLM settings, so test them before cutover. + +### Replace control-plane automation + +Scripts and CI jobs that manage AWS resources move to runpodctl or REST API v2. The [CLI command map](#cli-command-map) and [API map](#api-map) list the equivalents. Two patterns change shape: + +- **Deploying a new model version.** `UpdateEndpoint` with a new endpoint configuration becomes an image change on the endpoint: send `PATCH /v2/serverless/{id}` with the new `image`. Runpod creates a new release and deploys it as a [rolling release](/serverless/endpoints/rolling-releases), replacing workers as they cycle. Track it with `GET /v2/serverless/{id}/releases`. If you change the image with `runpodctl template update TEMPLATE_ID --image ...` instead, confirm that the endpoint's workers picked up the new image before you rely on it. +- **Waiting for readiness.** Polling for `InService` becomes waiting for a worker (`runpodctl serverless create ... --wait`) or polling `runpodctl pod get POD_ID` until `runtimeStatus` is `running`. + +## Step 8: Verify, cut over, and shut down + + + + Call the Runpod endpoint from the same network your callers use, with production-shaped payloads. Compare outputs with the AWS endpoint on a fixed set of inputs. Measure latency for warm requests and for a cold start from zero workers, and decide whether you need a warm worker. + + + Keep the AWS endpoint serving. Send a copy of traffic, or a small share, to Runpod and watch error rates, queue delay, and worker counts with `runpodctl serverless health ENDPOINT_ID` and the endpoint's **Metrics** tab. + + + Move traffic in steps behind a flag or a weighted route in your own gateway. Keep the AWS endpoint until Runpod has carried full load through a peak. + + + Cap `--workers-max`, set an idle timeout and an execution timeout on each endpoint, and stop or delete Pods you aren't using. A stopped Pod still bills for its volume disk. Runpod accounts have a default spend limit of \$80 per hour across all resources. Turn on low balance notifications in [billing](/accounts-billing/billing), and review spend with `runpodctl billing pods` and `runpodctl billing serverless`. + + + Delete what you replaced, in this order. Deleting a SageMaker endpoint doesn't delete its endpoint configuration or model, and terminating an instance doesn't always delete its volumes. + + + +Remove SageMaker autoscaling, endpoints, and notebooks: + +```bash +aws application-autoscaling deregister-scalable-target --service-namespace sagemaker \ + --resource-id endpoint/my-endpoint/variant/AllTraffic \ + --scalable-dimension sagemaker:variant:DesiredInstanceCount +aws sagemaker delete-endpoint --endpoint-name my-endpoint +aws sagemaker delete-endpoint-config --endpoint-config-name my-endpoint-config +aws sagemaker delete-model --model-name my-model +aws sagemaker stop-notebook-instance --notebook-instance-name my-notebook +aws sagemaker delete-notebook-instance --notebook-instance-name my-notebook +aws bedrock delete-provisioned-model-throughput --provisioned-model-id PROVISIONED_MODEL_ARN +``` + +Terminate instances, then find what they left behind: + +```bash +aws ec2 terminate-instances --instance-ids i-0123456789abcdef0 +aws ec2 describe-volumes --filters Name=status,Values=available --query "Volumes[].[VolumeId,Size]" --output table +aws ec2 describe-snapshots --owner-ids self --query "Snapshots[].[SnapshotId,VolumeSize,StartTime]" --output table +aws ec2 describe-addresses --query "Addresses[?AssociationId==null].[AllocationId,PublicIp]" --output table +aws ec2 describe-nat-gateways --query "NatGateways[?State=='available'].[NatGatewayId,VpcId]" --output table +``` + +Delete each leftover resource after you confirm nothing else uses it: + +```bash +aws ec2 delete-volume --volume-id vol-0123456789abcdef0 +aws ec2 delete-snapshot --snapshot-id snap-0123456789abcdef0 +aws ec2 release-address --allocation-id eipalloc-0123456789abcdef0 +aws ec2 delete-nat-gateway --nat-gateway-id nat-0123456789abcdef0 +aws ecr delete-repository --repository-name my-worker --force +aws logs describe-log-groups --log-group-name-prefix /aws/sagemaker/ --query "logGroups[].logGroupName" +aws logs delete-log-group --log-group-name /aws/sagemaker/Endpoints/my-endpoint +``` + +The shutdown checklist: + +- SageMaker endpoints, endpoint configurations, models, notebook instances, and Studio apps. +- Bedrock Provisioned Throughput and imported models. +- EC2 instances, Auto Scaling groups (set desired capacity to 0 first, or they relaunch instances), and launch templates. +- EBS volumes in the `available` state, and EBS snapshots and AMIs you no longer need. +- Elastic IPs that aren't associated, and NAT gateways that only served the GPU subnets. +- ECR repositories, unless Runpod still pulls from them. +- EFS and FSx file systems, and S3 buckets you copied in full. +- CloudWatch log groups, alarms, and dashboards for the deleted resources. +- Capacity Reservations, which bill whether or not instances run in them. Savings Plans run to the end of their term, so plan the cutover date around them. + +## Migrate your Terraform + +The [Runpod Terraform provider](/integrations/terraform/overview) 2.x is built on REST API v2. It runs in the same configuration as the AWS provider, so you can add Runpod resources next to the AWS ones, shift traffic, then remove the AWS blocks. + +### Resource map + +| AWS resource | Runpod resource | Notes | +| --- | --- | --- | +| `aws_instance` (GPU types) | `runpod_pod` | `instance_type` becomes `gpu_type_id` and `gpu_count`. `ami` and `user_data` become `image`. | +| `aws_launch_template` with `aws_autoscaling_group` | `runpod_template` with `runpod_serverless_endpoint` | Min and max size become `workers_min` and `workers_max`. | +| `aws_sagemaker_model` | `runpod_template` with `serverless = true` | `primary_container.image` and `environment` become `image` and `env`. `model_data_url` has no equivalent field. | +| `aws_sagemaker_endpoint_configuration`, `aws_sagemaker_endpoint` | `runpod_serverless_endpoint` | `instance_type` becomes `gpu_pools`. One resource replaces both. | +| `aws_appautoscaling_target`, `aws_appautoscaling_policy` | `workers_min`, `workers_max`, `scaling_type`, `scaling_value`, `idle_timeout` on the endpoint | No separate resources. | +| `aws_ebs_volume` | `persistent_volume` on `runpod_pod`, or `runpod_network_volume` | Use a network volume for data you can't recreate. | +| `aws_efs_file_system` | `runpod_network_volume` | `type = "HIGH_PERFORMANCE"` selects the high-performance tier where available. | +| `aws_ecr_repository` | Keep it, and add `runpod_registry_delegation` | Other registries use `runpod_registry`. | +| `aws_secretsmanager_secret` | Runpod secret, referenced in `env` as `{{ RUNPOD_SECRET_name }}` | The provider's resource list doesn't include a secret resource. Create secrets in the console or with `POST /v2/account/secrets`. | +| `aws_security_group` | `ports` on the Pod or endpoint | Ports are opened, not filtered by source. | +| `aws_key_pair` | `runpod_account_ssh_keys` | Account-level keys. | + +### Run both providers side by side + +Declare both providers in one configuration. The Runpod provider reads `RUNPOD_API_KEY` from the environment. + +```hcl versions.tf +terraform { + required_version = ">= 1.6" + + required_providers { + aws = { + source = "hashicorp/aws" + version = "~> 6.0" + } + runpod = { + source = "runpod/runpod" + version = "~> 2.0" + } + } +} + +provider "aws" { + region = "us-east-1" +} + +provider "runpod" {} +``` + +Keep whatever AWS provider version constraint you already use. With both providers loaded, Runpod resources can reference AWS ones. This example keeps images in ECR and lets Runpod pull them: + +```hcl ecr.tf +resource "aws_ecr_repository" "worker" { + name = "my-worker" +} + +resource "aws_ecr_repository_policy" "runpod_pull" { + repository = aws_ecr_repository.worker.name + policy = jsonencode({ + Version = "2012-10-17" + Statement = [{ + Sid = "AllowRunpodPull" + Effect = "Allow" + Principal = "*" + Action = [ + "ecr:GetAuthorizationToken", + "ecr:BatchCheckLayerAvailability", + "ecr:GetDownloadUrlForLayer", + "ecr:BatchGetImage", + ] + Condition = { + StringEquals = { + "aws:PrincipalArn" = [ + "arn:aws:iam::550005742258:role/prod-us-east-1-deployment-role", + "arn:aws:iam::550005742258:role/prod-us-west-2-deployment-role", + ] + } + } + }] + }) +} + +resource "runpod_registry_delegation" "worker" { + name = "my-worker-ecr" + resource = aws_ecr_repository.worker.arn + + depends_on = [aws_ecr_repository_policy.runpod_pull] +} +``` + +The `depends_on` matters because Runpod validates the repository policy when the delegation is created. A delegation can't be imported or updated in place, so any change replaces it. + +### GPU instance to Pod + + + + ```hcl + resource "aws_instance" "vllm" { + ami = "ami-0123456789abcdef0" # Deep Learning AMI + instance_type = "g6e.xlarge" + key_name = "my-key" + subnet_id = aws_subnet.gpu.id + vpc_security_group_ids = [aws_security_group.vllm.id] + iam_instance_profile = aws_iam_instance_profile.s3_read.name + user_data = file("${path.module}/user-data.sh") + + root_block_device { + volume_size = 200 + } + + tags = { + Name = "vllm-server" + } + } + ``` + + + ```hcl + resource "runpod_network_volume" "checkpoints" { + name = "checkpoints" + size = 200 + data_center = "US-GA-2" + } + + resource "runpod_pod" "vllm" { + name = "vllm-server" + image = "YOUR_REGISTRY/vllm-server:v1.0.0" + gpu_type_id = "NVIDIA L40S" + gpu_count = 1 + cloud = "SECURE" + disk = 50 + min_cuda_version = "12.8" + data_center_ids = [runpod_network_volume.checkpoints.data_center] + ports = ["8000/http"] + + network_volumes = [{ + volume_id = runpod_network_volume.checkpoints.id + path = "/workspace" + }] + + env = { + AWS_ACCESS_KEY_ID = "{{ RUNPOD_SECRET_aws_access_key_id }}" + AWS_SECRET_ACCESS_KEY = "{{ RUNPOD_SECRET_aws_secret_access_key }}" + AWS_DEFAULT_REGION = "us-east-1" + } + } + + output "vllm_url" { + value = "https://${runpod_pod.vllm.id}-8000.proxy.runpod.net" + } + ``` + + + +The subnet, security group, and instance profile have no Runpod resource. Ports replace the security group, and secret-backed environment variables replace the instance profile. The key pair becomes account-level SSH keys (`runpod_account_ssh_keys`), which a Pod receives when you set `start_ssh = true`. Changing `gpu_type_id`, `gpu_count`, `data_center_ids`, or `min_cuda_version` replaces the Pod, so keep state you care about on the network volume. To stop a Pod without removing it from the configuration, set `desired_state = "EXITED"`. + +### SageMaker endpoint with autoscaling to a Serverless endpoint + + + + ```hcl + resource "aws_sagemaker_model" "classifier" { + name = "classifier" + execution_role_arn = aws_iam_role.sagemaker.arn + + primary_container { + image = "${aws_ecr_repository.worker.repository_url}:v1.0.0" + model_data_url = "s3://my-bucket/models/model.tar.gz" + environment = { + LOG_LEVEL = "info" + } + } + } + + resource "aws_sagemaker_endpoint_configuration" "classifier" { + name_prefix = "classifier-" + + production_variants { + variant_name = "AllTraffic" + model_name = aws_sagemaker_model.classifier.name + initial_instance_count = 1 + instance_type = "ml.g5.xlarge" + } + + lifecycle { + create_before_destroy = true + } + } + + resource "aws_sagemaker_endpoint" "classifier" { + name = "classifier" + endpoint_config_name = aws_sagemaker_endpoint_configuration.classifier.name + } + + resource "aws_appautoscaling_target" "classifier" { + service_namespace = "sagemaker" + resource_id = "endpoint/${aws_sagemaker_endpoint.classifier.name}/variant/AllTraffic" + scalable_dimension = "sagemaker:variant:DesiredInstanceCount" + min_capacity = 1 + max_capacity = 4 + } + + resource "aws_appautoscaling_policy" "classifier" { + name = "classifier-invocations" + policy_type = "TargetTrackingScaling" + service_namespace = aws_appautoscaling_target.classifier.service_namespace + resource_id = aws_appautoscaling_target.classifier.resource_id + scalable_dimension = aws_appautoscaling_target.classifier.scalable_dimension + + target_tracking_scaling_policy_configuration { + target_value = 70 + + predefined_metric_specification { + predefined_metric_type = "SageMakerVariantInvocationsPerInstance" + } + } + } + ``` + + + ```hcl + resource "runpod_template" "classifier" { + name = "classifier" + image = "${aws_ecr_repository.worker.repository_url}:v1.0.0" + serverless = true + disk = 20 + args = "serve" + ports = ["8080/http"] + + start_ssh = false + start_jupyter = false + + env = { + LOG_LEVEL = "info" + PORT = "8080" + PORT_HEALTH = "8080" + } + + depends_on = [runpod_registry_delegation.worker] + } + + resource "runpod_serverless_endpoint" "classifier" { + name = "classifier" + template_id = runpod_template.classifier.id + endpoint_type = "LOAD_BALANCER" + ports = ["8080/http"] + + gpu_pools = ["AMPERE_24", "ADA_24"] + gpu_count = 1 + min_cuda_version = "12.8" + + scaling_type = "REQUEST_COUNT" + scaling_value = 4 + workers_min = 1 + workers_max = 4 + idle_timeout = 30 + timeout = 600000 + } + + output "classifier_url" { + value = runpod_serverless_endpoint.classifier.request_urls.base + } + ``` + + + +Five AWS resources become two. The IAM execution role has no counterpart, and `model_data_url` has no field: the model must be in the image or on a volume (`network_volumes` on the endpoint). `scaling_value = 4` adds a worker for every four in-flight requests, which is a concurrency target and not the per-minute invocation count that `SageMakerVariantInvocationsPerInstance` tracks, so derive it from how many concurrent requests one worker handles. `workers_min = 1` matches the always-on instance. Set it to `0` to scale to zero. For a queue endpoint, set `endpoint_type = "QUEUE"`, drop `ports`, and choose `QUEUE_DELAY` (seconds a request may wait) or `REQUEST_COUNT`. A queue endpoint that scales on `REQUEST_COUNT` doesn't take `idle_timeout`, so drop that too. The endpoint copies the template's settings when it's created, so pass later image changes through the endpoint. + +### Cut over in Terraform + +1. Add the Runpod resources and run `terraform apply`. The AWS resources are untouched. +2. Expose the Runpod URL as an output or pass it to the resource that configures your callers, then shift traffic. +3. Remove the AWS resource blocks in a separate change and apply it. Terraform destroys them. +4. Check for leftovers that Terraform didn't manage, using the commands in [Step 8](#step-8-verify-cut-over-and-shut-down). + +For every argument, see the [provider reference](https://registry.terraform.io/providers/runpod/runpod/latest/docs) on the Terraform Registry. + +## Reference + +### CLI command map + +| AWS CLI | runpodctl | +| --- | --- | +| `aws configure` | `export RUNPOD_API_KEY=...`, or `rp login` for the SDK | +| `aws ec2 run-instances` | `runpodctl pod create` | +| `aws ec2 describe-instances` | `runpodctl pod list --all`, `runpodctl pod get POD_ID` | +| `aws ec2 stop-instances` | `runpodctl pod stop POD_ID` | +| `aws ec2 start-instances` | `runpodctl pod start POD_ID` | +| `aws ec2 reboot-instances` | `runpodctl pod restart POD_ID` | +| `aws ec2 terminate-instances` | `runpodctl pod delete POD_ID` | +| `aws ec2 describe-instance-types` | `runpodctl gpu list` | +| `aws ec2 describe-regions` | `runpodctl datacenter list` | +| `aws ec2 create-volume` | `runpodctl network-volume create` | +| `aws sagemaker create-model` | `runpodctl template create --serverless` | +| `aws sagemaker create-endpoint-config`, `create-endpoint` | `runpodctl serverless create` | +| `aws sagemaker describe-endpoint` | `runpodctl serverless get ENDPOINT_ID`, `runpodctl serverless health ENDPOINT_ID` | +| `aws sagemaker list-endpoints` | `runpodctl serverless list` | +| `aws sagemaker update-endpoint` | `runpodctl serverless update ENDPOINT_ID`, `runpodctl template update TEMPLATE_ID --image ...` | +| AWS Cost Explorer | `runpodctl billing pods`, `runpodctl billing serverless`, `runpodctl billing network-volume` | +| `aws sagemaker delete-endpoint` | `runpodctl serverless delete ENDPOINT_ID` | +| `aws sagemaker-runtime invoke-endpoint` | `runpodctl serverless run ENDPOINT_ID --input '{...}'` | +| `aws s3 sync` | `aws s3 sync` inside a Pod, `aws s3 sync` against the volume's S3-compatible endpoint, or `runpodctl send` and `receive` | +| `aws ecr get-login-password` | Not needed with an ECR delegation. For other registries, `runpodctl registry create` | +| `aws logs tail --follow` | `runpodctl pod logs POD_ID --follow`, `runpodctl serverless logs ENDPOINT_ID --follow` | +| `aws ssm start-session`, `ssh` | `runpodctl ssh info POD_ID` prints the SSH command (see the [reference](/runpodctl/reference/runpodctl-ssh)) | + +Run `runpodctl COMMAND --help` for current flags. + +### API map + +| AWS API | Runpod REST API v2 | +| --- | --- | +| EC2 `RunInstances` | `POST /v2/pods` | +| EC2 `DescribeInstances` | `GET /v2/pods`, `GET /v2/pods/{id}` | +| EC2 `StopInstances`, `StartInstances`, `RebootInstances` | `POST /v2/pods/{id}/action` with `stop`, `start`, or `restart` | +| EC2 `TerminateInstances` | `DELETE /v2/pods/{id}` | +| EC2 `CreateVolume` | `POST /v2/network-volumes` | +| SageMaker `CreateModel` | `POST /v2/templates` | +| SageMaker `CreateEndpointConfig`, `CreateEndpoint` | `POST /v2/serverless` | +| SageMaker `UpdateEndpoint`, `UpdateEndpointWeightsAndCapacities` | `PATCH /v2/serverless/{id}` | +| SageMaker `DeleteEndpoint` | `DELETE /v2/serverless/{id}` | +| SageMaker Runtime `InvokeEndpoint` | `POST https://api.runpod.ai/v2/ENDPOINT_ID/runsync`, or your own path on `https://ENDPOINT_ID.api.runpod.ai` | +| SageMaker Runtime `InvokeEndpointAsync` | `POST https://api.runpod.ai/v2/ENDPOINT_ID/run` | +| SageMaker Runtime `InvokeEndpointWithResponseStream` | `GET https://api.runpod.ai/v2/ENDPOINT_ID/stream/JOB_ID`, or a streaming response from a load balancing worker | +| Bedrock Runtime `Converse`, `InvokeModel` | `POST https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1/chat/completions` | +| ECR `GetAuthorizationToken` | `POST /v2/registries/delegations` (once) | +| Secrets Manager `CreateSecret` | `POST /v2/account/secrets` | +| CloudWatch Logs `GetLogEvents` | `GET /v2/pods/{id}/logs`, `GET /v2/serverless/{id}/workers/{workerId}/logs` | + +Management routes use `https://api.runpod.io/v2`. Inference routes use `https://api.runpod.ai`. + +### Configuration field map + +| AWS field | REST API v2 field | runpodctl flag | +| --- | --- | --- | +| `InstanceType` | `gpu.id` and `gpu.count` (Pods), `gpu.pools` (Serverless) | `--gpu-id`, `--gpu-count` | +| `ImageId` (AMI), `PrimaryContainer.Image` | `image` | `--image` | +| `UserData` | `cmd`, `entrypoint`, or the image's own command | `--docker-args` | +| `PrimaryContainer.Environment` | `env` | `--env` | +| `SecurityGroupIds` (ingress ports) | `ports` | `--ports` | +| `BlockDeviceMappings` root volume size | `disk` | `--container-disk-in-gb` | +| EBS data volume | `mounts.persistent`, `mounts.network` | `--volume-in-gb`, `--network-volume-id` | +| `Placement.AvailabilityZone`, Region | `dataCenterIds` | `--data-center-ids` | +| `InitialInstanceCount`, `MinCapacity` | `workers.min` | `--workers-min` | +| `MaxCapacity` | `workers.max` | `--workers-max` | +| Target tracking `TargetValue` | `scaling.requestCount` or `scaling.queueDelay` | `--scale-by`, `--scale-threshold` | +| Scale-in cooldown | `workers.idleTimeout` | `--idle-timeout` | +| `ModelDataUrl` | No field. Image, network volume, or cached model | `--network-volume-id`, `--model-reference` | +| `ExecutionRoleArn`, `IamInstanceProfile` | No field. Secrets in `env` | `--env` | +| `KeyName` | `startSsh` with account SSH keys | `--ssh` | + +### Environment variable map + +| On AWS | On Runpod | Notes | +| --- | --- | --- | +| Instance ID from the metadata service | `RUNPOD_POD_ID` | Runpod has no metadata service at `169.254.169.254`. | +| Region or Availability Zone from the metadata service, `AWS_REGION` | `RUNPOD_DC_ID` | Keep `AWS_DEFAULT_REGION` set for boto3 calls back to AWS. | +| Public IPv4 from the metadata service | `RUNPOD_PUBLIC_IP`, `RUNPOD_TCP_PORT_22` | Present when the Pod has a public IP. | +| Credentials from the instance profile or execution role | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` from secrets | Set them yourself. | +| Port 8080 (SageMaker contract) | `PORT`, `PORT_HEALTH`, `HEALTH_CHECK_PATH` | Load balancing endpoints only. | +| `/opt/ml/model` | Your own path, such as `/runpod-volume/my-model` | Or bake the same path into the image. | +| `SAGEMAKER_BATCH`, `SAGEMAKER_MAX_CONCURRENT_TRANSFORMS` | None | Batch behavior is handled by your handler. | +| GPU count from the instance type | `RUNPOD_GPU_COUNT` | Pods also set `RUNPOD_CPU_COUNT`. | + +### Status map + +| AWS status | Runpod status | +| --- | --- | +| EC2 `pending` | Pod `PROVISIONING`, `STARTING` | +| EC2 `running` | Pod `RUNNING` | +| EC2 `stopping`, `stopped` | Pod `EXITED` | +| EC2 `shutting-down`, `terminated` | Pod `TERMINATED` | +| SageMaker endpoint `Creating`, `Updating` | Workers `INITIALIZING` | +| SageMaker endpoint `InService` | Workers `IDLE` or `RUNNING` | +| SageMaker endpoint `Failed` | Workers `UNHEALTHY`, or `THROTTLED` when waiting on capacity | +| Asynchronous request queued or processing | Job `IN_QUEUE`, `IN_PROGRESS` | +| Asynchronous success or error notification | Job `COMPLETED`, `FAILED`, `TIMED_OUT`, `CANCELLED` | + +A Runpod endpoint has no single endpoint-level status like `InService`. Read worker states from `runpodctl serverless health ENDPOINT_ID`. + +## Migrate with a coding agent + +A coding agent with the Runpod skills and MCP server can run this migration with you: it reads your AWS inventory, proposes a plan, and creates Runpod resources after you approve them. + + + + ```bash + # The plugin also connects the hosted MCP server + claude plugin marketplace add runpod/runpod-plugins-official + claude plugin install runpod@runpod + ``` + + + ```bash + # Skills, then the hosted MCP server (Codex, Cursor, and others) + npx skills add runpod/runpod-plugins-official --skill '*' --agent AGENT_ID --global + npx @runpod/mcp-server@latest add --client CLIENT_ID + ``` + + + +The MCP server signs in with OAuth ("Sign in with Runpod"), so you don't need an API key to start. See [Agent skills](/get-started/agent-skills) and [MCP servers](/get-started/mcp-servers) for details. Then give the agent this prompt from the root of your repository: + +```text +Use the runpod-cloud-migrate skill to migrate our AWS GPU workloads to Runpod. + +1. Inventory first, read-only. Use the AWS CLI with my current profile to list GPU EC2 + instances (and their user data), SageMaker endpoints, endpoint configs, models, + autoscaling targets, notebook instances, Bedrock provisioned throughput, ECR + repositories, and the S3 buckets they read. Search this repo for boto3 + sagemaker-runtime and bedrock-runtime calls, SageMaker inference scripts + (model_fn, /invocations, /opt/ml), and aws_instance or aws_sagemaker_* Terraform. +2. Show me a plan: each workload, its Runpod target (Pod, queue endpoint, load + balancing endpoint, Public Endpoint, vLLM endpoint, or Instant Cluster), the GPU + type or pool, where its data will live, what stays on AWS, and a cost estimate + from current Runpod pricing. List every gap that needs my decision (VPC-only + resources, IAM roles, KMS, multi-model endpoints). +3. Wait for my approval before creating anything billable on Runpod. +4. Migrate one workload end to end. Keep S3 as the data store unless I say otherwise, + and use scoped IAM credentials stored as Runpod secrets, never my admin keys. + If we use Terraform, add the Runpod provider next to the AWS provider. +5. Verify with a real request and compare the output with the AWS endpoint. +6. Never stop, delete, or modify AWS resources without my explicit approval. When + I approve shutdown, list the leftovers first: EBS volumes and snapshots, Elastic + IPs, NAT gateways, endpoint configs, models, log groups. +``` + +## Troubleshooting + +**Workers start, then requests return `502` and workers recycle after about 8 minutes.** The load balancer can't reach your health check. SageMaker containers listen on 8080, and Runpod checks port 80 unless told otherwise. Set both `PORT` and `PORT_HEALTH` to `8080` and expose `8080/http`. + +**The container exits immediately on Runpod but ran on SageMaker.** SageMaker starts inference containers with the `serve` argument. Without it, an entrypoint that dispatches on `serve` or `train` exits. Set the container command to `serve`, or change the image's `CMD`. + +**`FileNotFoundError` under `/opt/ml/model`.** Nothing unpacks `model.tar.gz` on Runpod. Bake the files into the image, or mount a network volume and point your code at it. + +**`Unable to locate credentials` or `NoCredentialsError` from boto3.** There's no instance profile or execution role. Create a scoped IAM user, store its keys as Runpod secrets, and map them to `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` in the template's environment variables. Set `AWS_DEFAULT_REGION` too. + +**Requests to `169.254.169.254` hang.** Code is calling the EC2 instance metadata service. Replace those lookups with the `RUNPOD_` environment variables and explicit configuration. + +**`exec format error` when the container starts.** The image was built for arm64 (Graviton or Apple silicon). Rebuild with `--platform linux/amd64`. + +**Image pull fails for an ECR image.** Confirm the repository policy lists both Runpod role ARNs exactly, that a delegation exists for that repository (`GET /v2/registries/delegations`), and that the tag exists. If you stored a `get-login-password` token as a registry credential, it expired after 12 hours. Use a delegation instead. + +**`{"error":"no workers available"}` from a load balancing endpoint.** The endpoint scaled to zero and the first worker hasn't passed its health check. Retry with a delay of 5 to 10 seconds, or set min workers to 1. + +**CUDA or driver errors on some workers only.** Hosts differ in driver version. Set `--min-cuda-version` (REST `gpu.minCudaVersion`) to the version your framework build needs. + +**A Pod can't see a network volume, or deploy fails for lack of capacity.** The volume pins the Pod to its data center, and the GPU you asked for isn't in stock there. Pick a GPU type that data center has, or create the volume in a data center with stock for your GPU. + +## Next steps + + + + Learn how endpoints, workers, and scaling fit together. + + + Review the health check, port, and timeout contract. + + + Manage Pods, endpoints, and volumes as code. + + + Set up cross-account pulls from Amazon ECR. + + + +Found a gap or an error in this guide? Open an issue in the [Runpod docs repository](https://github.com/runpod/docs/issues). diff --git a/migrate/azure.mdx b/migrate/azure.mdx new file mode 100644 index 000000000..2d179bfb1 --- /dev/null +++ b/migrate/azure.mdx @@ -0,0 +1,1400 @@ +--- +title: "Migrate from Azure to Runpod" +sidebarTitle: "Azure" +description: "Move GPU VMs, Azure Machine Learning endpoints, Container Apps GPU apps, and open-model inference from Azure to Runpod, with Terraform and code examples." +--- + +This guide is for teams that run GPU workloads on Microsoft Azure and want to run some or all of them on Runpod. Most teams move the GPU-heavy part (training, inference, batch) and keep data, identity, and the rest of the application on Azure, so the guide covers that hybrid setup first and a full exit second. + +[GPU virtual machines](https://learn.microsoft.com/azure/virtual-machines/sizes/overview) map to [Pods](/pods/overview). [Azure Machine Learning](https://learn.microsoft.com/azure/machine-learning/) online and batch endpoints and [Azure Container Apps serverless GPUs](https://learn.microsoft.com/azure/container-apps/gpu-serverless-overview) map to [Serverless](/serverless/overview) endpoints. Open-weight models you call through [Microsoft Foundry](https://learn.microsoft.com/azure/foundry/what-is-foundry) map to [Public Endpoints](/public-endpoints/overview) or a Serverless vLLM endpoint. By the end, you'll have one workload running on Runpod, verified with a real request, and a checklist for stopping the matching Azure charges. + + +You can hand most of this migration to a coding agent. See [Migrate with a coding agent](#migrate-with-a-coding-agent). + + +## How Azure maps to Runpod + +| Azure concept | Runpod equivalent | Notes | +| --- | --- | --- | +| GPU virtual machines (NC, ND families) | [Pods](/pods/overview) | A Pod is a Linux container with dedicated GPUs, not a VM. You pick a GPU type and count instead of a VM size. | +| NV and NG families (virtual desktop, visualization, cloud gaming), Windows GPU VMs | No equivalent | Pods run Linux containers only. Keep VDI and Windows workloads on Azure. | +| Virtual Machine Scale Sets | Serverless endpoints, or Pods you create through the API | Serverless scales workers for request-driven work. Pods don't autoscale, so you create and delete them through the API or Terraform. | +| Azure Machine Learning managed online endpoint (endpoint plus deployments) | [Serverless queue-based endpoint](/serverless/overview) or [load balancing endpoint](/serverless/load-balancing/overview) | A scoring script becomes a handler function. A custom container with its own routes becomes a load balancing endpoint. | +| Azure Machine Learning batch endpoint | Queue-based endpoint with `/run` and webhooks | Jobs queue and run asynchronously. You pass data references in the request. | +| Azure Machine Learning jobs, compute clusters, pipelines | Pods, or the Runpod SDK `@app.task` | `@app.task` runs one ephemeral Pod per call. Bring your own orchestrator for multi-step pipelines. | +| Azure Machine Learning compute instance | Pod with JupyterLab | Official Runpod images start JupyterLab on port 8888. | +| Azure OpenAI in Microsoft Foundry Models, Foundry serverless API deployments | [Public Endpoints](/public-endpoints/overview), or a [Serverless vLLM endpoint](/serverless/vllm/get-started) | Applies to open-weight models. See [What has no direct equivalent](#what-has-no-direct-equivalent). | +| Azure Container Apps with serverless GPUs | [Load balancing endpoint](/serverless/load-balancing/overview) | Both run your HTTP server and scale to zero. The health check and port contract differ. | +| Azure Container Instances GPU | Pods or Serverless | Microsoft's documentation lists GPU container groups as retired as of July 14, 2025. | +| AKS GPU node pools | Serverless or Pods, or [SkyPilot](/integrations/skypilot) and [dstack](/integrations/dstack) | Runpod runs your containers directly, so there's no cluster to manage. SkyPilot and dstack keep a scheduler in front. | +| Azure Batch | Queue-based endpoint or `@app.task` | | +| Azure CycleCloud with Slurm | [Instant Clusters with Slurm](/instant-clusters/slurm-clusters) | Multi-node clusters with a shared network. Stock varies by data center. | +| Azure Container Registry | Any registry, with [registry credentials](/runpodctl/reference/runpodctl-registry) for private images | Runpod pulls from ACR using a token or service principal as username and password. | +| Azure Machine Learning environments and curated environments | Container images and [templates](/pods/templates/overview) | Runpod PyTorch images cover the common training stack. | +| OS disk, temporary disk | Container disk | Erased when the Pod stops. | +| Managed Disks (data disks) | Volume disk or [network volume](/storage/network-volumes) | A network volume exists independently of any Pod. | +| Azure Files, Azure NetApp Files, Azure Managed Lustre | Network volume, including [high-performance storage](/storage/high-performance-storage) | A network volume lives in one data center. | +| Blob Storage, BlobFuse mounts | Keep using Blob Storage over HTTPS, or copy to a network volume or [global volume](/storage/globalvolume/overview) | Global volumes mount in any data center and are in beta. | +| VNet, NSGs, public IPs, Private Link | Exposed ports, HTTP proxy, [global networking](/pods/networking) | Runpod has no VNet, NSG, or Private Link equivalent. | +| Key Vault secrets | [Secrets](/pods/templates/secrets) | Referenced from environment variables as `{{ RUNPOD_SECRET_name }}`. | +| Regions and availability zones | Data centers | List them with `runpodctl datacenter list`. | +| Pay-as-you-go, Spot VMs, reservations, savings plan | Per-second on-demand billing | `runpodctl` and REST API v2 create on-demand Pods. See [pricing](/pods/pricing). | +| Bicep, ARM templates, Terraform (`azurerm`, `azapi`) | [Runpod Terraform provider](/integrations/terraform/overview) | Bicep and ARM have no Runpod equivalent. Use Terraform or REST API v2. | +| Azure CLI, `azure-ai-ml` SDK, Azure Resource Manager | `runpodctl`, the [Runpod SDK](/sdk/overview), [REST API v2](/api-reference-v2/overview), [MCP server](/get-started/mcp-servers) | | +| Microsoft Entra ID, managed identities, Azure RBAC | [API keys](/get-started/credentials), [organizations and roles](/organizations/orgs-overview) | Runpod has no workload identity. Workloads authenticate to other services with secrets. | +| Subscriptions, resource groups, management groups | Accounts and organizations | Organizations are in beta. | +| Azure Monitor, Log Analytics, Application Insights | Console logs and metrics, `runpodctl pod logs`, `runpodctl serverless logs`, REST API v2 metrics routes | Ship logs to your own system if you need long retention or alerting. | + +## What's different + +### What works differently + +**You choose a GPU, not a VM size.** An Azure size such as `Standard_NC24ads_A100_v4` fixes the GPU, vCPUs, memory, and disk together. On Runpod you pick a GPU type and count, and the Pod reports the vCPUs and memory it was allocated. Use the `min_vcpu_count_per_gpu` and `min_ram_per_gpu` placement filters when your workload needs a floor. + +**Pods are containers.** There's no cloud-init, no kernel modules, no systemd, and no NVIDIA driver extension to install: the host provides the driver and your image provides CUDA libraries. Anything your cloud-init script did moves into the image or a start command. Docker Compose and UDP ports aren't supported. + +**Stopping means something different.** A stopped Azure VM that isn't deallocated still bills for compute. A stopped Pod doesn't bill for compute, and its container disk is erased. Its volume disk is kept and still bills. See [Pod pricing](/pods/pricing). + +**Serverless scales on the queue, not on Azure Monitor rules.** A managed online deployment runs a fixed `instance_count` unless you attach Azure Monitor autoscale rules. A Runpod endpoint has a minimum and maximum worker count and scales on queue delay or request count. The minimum defaults to zero, so an idle endpoint has no running workers. + +**Requests are wrapped.** A queue-based endpoint takes `{"input": {...}}` and returns `{"id", "status", "output"}`. Your scoring script's `run(raw_data)` receives the raw body, so both the handler and every client change shape. A load balancing endpoint keeps your own request and response shapes. + +**Limits differ.** A managed online endpoint caps a request at 180 seconds. A queue-based endpoint's execution timeout defaults to 10 minutes in the console and can be raised, with a 10 MB payload limit on `/run` and 20 MB on `/runsync`. A load balancing endpoint has a 5.5 minute processing timeout and a 30 MB payload limit. + +**Rollouts replace workers instead of splitting traffic.** Azure Machine Learning splits traffic by percentage across deployments and can mirror traffic. A Runpod endpoint update is a [rolling release](/serverless/endpoints/rolling-releases): idle workers are replaced immediately and busy workers after their current job. For a percentage split, run two endpoints and split in your client or gateway. + +**Auth is one API key.** Runpod endpoints accept `Authorization: Bearer YOUR_API_KEY`. There's no equivalent of `aad_token` auth or per-endpoint keys. A Pod's HTTP proxy URL is public and unauthenticated, so put your own auth in front of anything you expose on a Pod. + +**Images must be `linux/amd64`.** Build with `docker build --platform linux/amd64`, especially on Apple silicon or ARM build agents. + +**Data transfer.** Azure charges for outbound data transfer to the internet beyond a monthly free allowance, so budget for the one-time copy of datasets and model weights out of Azure. Runpod doesn't charge for data ingress or egress. + +### What has no direct equivalent + +**On Runpod:** + +- **OpenAI's proprietary GPT chat models.** Runpod doesn't host the GPT deployments you create in Azure OpenAI. The migration applies when you move to an open-weight model, either one listed in the [Public Endpoint model reference](/public-endpoints/reference) or one you serve yourself with vLLM. If you need to keep a proprietary model, keep that call on Azure. +- **VNet, NSGs, and Private Link.** Runpod has no customer-defined private network or private connectivity to Azure. Pods in your account can reach each other over [global networking](/pods/networking) at `POD_ID.runpod.internal`. Traffic between Runpod and Azure goes over the public internet with TLS, so Azure resources behind private endpoints need a public endpoint with firewall rules, or a proxy you run. +- **Managed identities and Entra ID RBAC for workloads.** A Runpod worker can't assume an Azure identity. Store a least-privilege service principal secret or a SAS token as a Runpod secret and pass it as an environment variable. +- **Key Vault.** Runpod secrets store values and inject them at boot. They don't provide key operations, certificates, or versioning. Keep Key Vault for those and copy only the values workloads need. +- **Windows and VDI.** NV and NG family workloads, Windows GPU VMs, and Data Science VM Windows images stay on Azure. +- **Managed Kubernetes.** Runpod runs containers directly instead of offering a Kubernetes control plane. Run a scheduler on top of Runpod with [SkyPilot](/integrations/skypilot) or [dstack](/integrations/dstack), or move each workload to Pods or Serverless. +- **Model registry, pipelines, and experiment tracking.** These Azure Machine Learning workspace features stay on Azure, or move to a tool you host. Runpod runs the compute. +- **Traffic mirroring and percentage splits** across deployments of one endpoint. +- **Bicep and ARM templates.** Use Terraform or REST API v2. +- **Spot capacity through the current API.** REST API v2, `runpodctl`, and the Terraform provider create on-demand Pods. Plan Spot VM workloads as on-demand Pods or Serverless. + +**On Azure:** + +- **A job queue built into the inference endpoint.** A queue-based Runpod endpoint includes the queue, status polling, streaming, retries, and webhooks. On Azure you'd typically pair an online endpoint with a separate queue service, or use a batch endpoint. +- **GPU access without per-family quota requests.** Azure gates GPU capacity with per-region, per-VM-family quota. Runpod uses an account-level spend limit and per-data-center stock instead, so check availability before you plan around a GPU type. +- **One volume shared across data centers.** A global volume mounts on Pods and workers in any data center. + +## Requirements + +- An Azure account with permission to read the resources you plan to move, and the [Azure CLI](https://learn.microsoft.com/cli/azure/install-azure-cli) with the `ml` and `containerapp` extensions if you use those services. +- A [Runpod account](https://console.runpod.io) with credits and an [API key](/get-started/credentials). +- Docker, to rebuild or retag images for `linux/amd64`. +- Python 3.10 or later if you use the Runpod SDK. +- [Terraform](https://developer.hashicorp.com/terraform/install) 1.6 or later if you manage infrastructure as code. + +## Step 1: Take inventory of what you run on Azure + +List every GPU resource, then export its configuration. Start with virtual machines. This command lists N-series VMs across the subscription with their size, power state, and location: + +```bash +az vm list -d \ + --query "[?starts_with(hardwareProfile.vmSize, 'Standard_N')].{name:name, rg:resourceGroup, size:hardwareProfile.vmSize, state:powerState, location:location}" \ + --output table +``` + +For each VM, record the size, image, OS and data disk sizes, open ports, and the cloud-init or custom script it boots with. `az vm show -g RESOURCE_GROUP -n VM_NAME` returns the full definition. VMs in the Stopped state without being deallocated are still billing, so note those too. + +Next, list Azure Machine Learning resources in each workspace: + +```bash +az ml online-endpoint list -g RESOURCE_GROUP -w WORKSPACE --output table +az ml online-deployment list --endpoint-name ENDPOINT_NAME -g RESOURCE_GROUP -w WORKSPACE +az ml batch-endpoint list -g RESOURCE_GROUP -w WORKSPACE --output table +az ml compute list -g RESOURCE_GROUP -w WORKSPACE --output table +az ml job list -g RESOURCE_GROUP -w WORKSPACE --query "[].{name:name, status:status}" --output table +``` + +For each online deployment, record `instance_type`, `instance_count`, the model and environment references, `environment_variables`, `request_settings`, and whether it uses a scoring script or a custom container with `inference_config` routes. Check the endpoint's `auth_mode` and `traffic` with `az ml online-endpoint show`, and any Azure Monitor autoscale settings attached to the deployment. + +List Container Apps, then read each GPU app's definition: + +```bash +az containerapp list -g RESOURCE_GROUP --output table +az containerapp show -n APP_NAME -g RESOURCE_GROUP +``` + +From the `show` output, record the workload profile name, image, CPU and memory, environment variables and secrets, ingress target port, minimum and maximum replicas, and scale rules. + +Then list the storage and images those workloads depend on: + +```bash +az disk list --output table +az storage account list --output table +az acr repository list --name REGISTRY_NAME --output table +``` + +Finally, find the code and configuration that calls Azure. Run this from the root of each repository: + +```bash +grep -rnE \ + "azure\.ai\.ml|azureml|AzureOpenAI|azure_endpoint|openai\.azure\.com|services\.ai\.azure\.com|inference\.ml\.azure\.com|azurecontainerapps\.io|azurecr\.io|blob\.core\.windows\.net|AZUREML_MODEL_DIR|DefaultAzureCredential|az ml |az vm |az containerapp |azcopy|azurerm_|azapi_" \ + --include="*.py" --include="*.ts" --include="*.js" --include="*.go" --include="*.sh" \ + --include="*.yml" --include="*.yaml" --include="*.tf" --include="*.bicep" --include="*.json" --include="Dockerfile*" . +``` + +Sort the matches into three groups: code that runs inside the workload (scoring scripts, training scripts), code that calls the workload (clients), and automation that manages it (CI, Terraform, scripts). + +## Step 2: Set up Runpod + +Create an account at [console.runpod.io](https://console.runpod.io), add credits, and [create an API key](/get-started/credentials). Then install the tools you'll use. + + + + Install the CLI and check that it can reach your account: + + ```bash + curl -fsSL https://cli.runpod.net | sudo bash + export RUNPOD_API_KEY="YOUR_API_KEY" + runpodctl doctor + ``` + + On macOS you can also run `brew install runpod/runpodctl/runpodctl`. + + + Install the Python SDK, which includes the `rp` CLI, and log in: + + ```bash + pip install runpod + rp login + ``` + + `rp login` stores your key in `~/.runpod/config.toml`. The SDK also reads `RUNPOD_API_KEY`. + + + Add the Runpod provider next to your existing Azure providers. See [Migrate your Terraform](#migrate-your-terraform). + + + +To let a coding agent manage Runpod for you, connect the [Runpod MCP server](/get-started/mcp-servers). + +## Step 3: Choose where each workload lands + +Pick a target for each workload based on how it's invoked, not on which Azure service it runs on today. + +| Azure workload | Runpod target | Why | +| --- | --- | --- | +| GPU VM you SSH into for development or training | [Pod](/pods/overview) | Long-lived, interactive, persistent storage. | +| Compute instance with notebooks | Pod with JupyterLab | Same workflow, port 8888 over the HTTP proxy. | +| Azure Machine Learning command job or Azure Batch task | Pod, or `@app.task` in the [Runpod SDK](/sdk/overview) | Runs to completion, then releases the GPU. | +| Managed online endpoint with a scoring script | [Queue-based endpoint](/serverless/workers/handler-functions) | `init()` and `run()` translate directly to a handler. | +| Managed online endpoint with a custom container and `scoring_route` | [Load balancing endpoint](/serverless/load-balancing/overview) | Keeps your HTTP server and routes. | +| Batch endpoint | Queue-based endpoint with `/run` and a webhook | Asynchronous, queued, retried. | +| Container Apps GPU app | Load balancing endpoint | Closest match: your HTTP server, scale to zero. | +| LLM served on a VM, AKS, or a managed compute deployment | [Serverless vLLM endpoint](/serverless/vllm/get-started) | OpenAI-compatible route with no server to manage. | +| Open-weight model called through a Foundry serverless API | [Public Endpoint](/public-endpoints/reference), if the model is listed | No deployment. | +| CycleCloud Slurm cluster or multi-node training on ND VMs | [Instant Cluster](/instant-clusters/slurm-clusters) | Multi-node with Slurm, PyTorch, or Ray. | +| AKS GPU node pool | Serverless or Pods per workload, or [SkyPilot](/integrations/skypilot) or [dstack](/integrations/dstack) | Depends on whether you want to keep an orchestrator. | + +Use this table to translate current Azure GPU sizes to Runpod GPU type IDs. GPU counts and memory come from Microsoft's size documentation, and the Runpod IDs come from the [GPU types reference](/references/gpu-types). + +| Azure size | GPUs | Runpod GPU type ID | Serverless pool | +| --- | --- | --- | --- | +| `Standard_NC4as_T4_v3`, `NC8as_T4_v3`, `NC16as_T4_v3` | 1 x T4 (16 GB) | `NVIDIA RTX A4000` or `NVIDIA RTX 2000 Ada Generation` (16 GB), or `NVIDIA L4` (24 GB) | `AMPERE_16` or `AMPERE_24` | +| `Standard_NC64as_T4_v3` | 4 x T4 (16 GB) | Four 16 GB GPUs, or fewer larger GPUs | `AMPERE_16` | +| `Standard_NC24ads_A100_v4`, `NC48ads_A100_v4`, `NC96ads_A100_v4` | 1, 2, or 4 x A100 (80 GB) | `NVIDIA A100 80GB PCIe` or `NVIDIA A100-SXM4-80GB` | `AMPERE_80` | +| `Standard_NC40ads_H100_v5`, `NC80adis_H100_v5` | 1 or 2 x H100 (94 GB) | `NVIDIA H100 NVL` | `ADA_80_PRO` (80 GB H100 pool) | +| `Standard_ND96amsr_A100_v4` | 8 x A100 (80 GB) | `NVIDIA A100-SXM4-80GB` | `AMPERE_80` | +| `Standard_ND96isr_H100_v5` | 8 x H100 (80 GB) | `NVIDIA H100 80GB HBM3` | `ADA_80_PRO` | +| `Standard_ND96isr_H200_v5` | 8 x H200 (141 GB) | `NVIDIA H200` | `HOPPER_141` | +| `Standard_ND96isr_MI300X_v5` | 8 x MI300X | `AMD Instinct MI300X OAM` | Not in a listed pool. Use Pods. | +| NC RTX PRO 6000 Blackwell Server Edition v6 sizes | Fractional, 1, or 2 x RTX PRO 6000 (24, 48, or 96 GB per GPU) | `NVIDIA RTX PRO 6000 Blackwell Server Edition`, or its `MIG 1g.24gb` and `MIG 2g.48gb` types | Not in a listed pool. Use Pods. | +| NVadsA10 v5 sizes (`Standard_NV6ads_A10_v5` to `Standard_NV72ads_A10_v5`) | Fractional, 1, or 2 x A10 (24 GB) | For Linux compute only: `NVIDIA L4`, `NVIDIA RTX A5000`, or `NVIDIA GeForce RTX 4090` (24 GB) | `AMPERE_24` or `ADA_24` | + +Runpod's list doesn't include the T4 or A10, so those rows give the closest memory match. Choose by what the workload needs (GPU memory, then throughput), and check live stock before you commit to a type: run `runpodctl gpu list`, or call `GET /v2/catalog/gpus`. Multi-GPU counts and specific types aren't available in every data center. + +## Step 4: Move your images and code + +### Use your Azure Container Registry images + +You can keep images in ACR. Runpod pulls private images with a saved username and password, so create a credential that works without an interactive login. Microsoft documents two options for unattended pulls: a repository-scoped token and a service principal. The registry admin user also works, but Microsoft doesn't recommend sharing it because it has full push and pull access. + + + + Create a token that can only read the repository Runpod needs: + + ```bash + az acr token create --name runpod-pull --registry myregistry \ + --repository inference/worker content/read \ + --output json + ``` + + The output includes two generated passwords, shown once. The username is the token name. Save it on Runpod, reading the password from an environment variable so it stays out of your shell history: + + ```bash + printenv ACR_TOKEN_PASSWORD | runpodctl registry create \ + --name acr-myregistry --username runpod-pull --password-stdin + ``` + + + Create a service principal with the `acrpull` role on the registry: + + ```bash + ACR_REGISTRY_ID=$(az acr show --name myregistry --query id --output tsv) + SP_PASSWORD=$(az ad sp create-for-rbac --name runpod-acr-pull \ + --scopes "$ACR_REGISTRY_ID" --role acrpull --query password --output tsv) + SP_APP_ID=$(az ad sp list --display-name runpod-acr-pull --query "[].appId" --output tsv) + ``` + + The username is the application ID and the password is the client secret: + + ```bash + printf '%s' "$SP_PASSWORD" | runpodctl registry create \ + --name acr-myregistry --username "$SP_APP_ID" --password-stdin + ``` + + Service principal secrets expire (one year by default), so add a rotation reminder. + + + +`runpodctl registry create` returns a registry credential ID. Pass it as `--registry-auth-id` when you create a template or Pod, or as `registry` in REST API v2 and Terraform. Reference the image by its full name, such as `myregistry.azurecr.io/inference/worker:v1`. + + +Runpod pulls images over the public internet. If your registry only allows private endpoint access, enable public access with authentication, or push a copy of the image to a registry that's reachable from outside your VNet. + + +### Check the image + +- **Architecture:** Runpod runs `linux/amd64` images only. +- **CUDA:** Your image ships the CUDA runtime, and the host ships the driver. Set a CUDA floor when you create the Pod or endpoint (`--min-cuda-version 12.8`, or `gpu.minCudaVersion` in REST API v2) so the host driver supports your image. +- **Azure base images:** Images built from Azure Machine Learning inference or curated environments include the Azure inference server, which expects the Azure Machine Learning runtime around it. Rebuild from a Runpod PyTorch image such as `runpod/pytorch:1.0.3-cu1281-torch291-ubuntu2404`, or from an `nvidia/cuda` base, and install your dependencies from the environment's conda or pip file. +- **Start command:** A Pod runs the image's `CMD`. Override it with `--docker-args` in `runpodctl`, `args` in Terraform, or `cmd` in REST API v2. + +### Translate a scoring script to a handler + +An Azure Machine Learning scoring script has two functions. `init()` runs once when the container starts and loads the model from `AZUREML_MODEL_DIR`. `run(raw_data)` runs for every request. Here's a typical GPU scoring script: + +```python score.py +import json +import os + +import torch + + +def init(): + global model + model_path = os.path.join(os.getenv("AZUREML_MODEL_DIR"), "model.pt") + model = torch.jit.load(model_path, map_location="cuda").eval() + + +def run(raw_data): + data = json.loads(raw_data)["data"] + with torch.no_grad(): + result = model(torch.tensor(data, device="cuda")) + return result.tolist() +``` + +On Runpod, module-level code replaces `init()` and a handler function replaces `run()`. The handler receives a job whose `input` field is already parsed JSON: + +```python handler.py +import os + +import runpod +import torch + +MODEL_DIR = os.getenv("MODEL_DIR", "/models") + +# Runs once per worker, like init(). +model = torch.jit.load(os.path.join(MODEL_DIR, "model.pt"), map_location="cuda").eval() + + +def handler(job): + data = job["input"]["data"] + with torch.no_grad(): + result = model(torch.tensor(data, device="cuda")) + return result.tolist() + + +runpod.serverless.start({"handler": handler}) +``` + +Three things changed. `AZUREML_MODEL_DIR` doesn't exist on Runpod, so the model path is a variable you control. The request body is no longer a raw string, so `json.loads` goes away. And the return value is placed in the `output` field of the job result. To return an error, raise an exception (the job is marked `FAILED`) or return a dictionary with an `error` key. + +Azure Machine Learning mounts the registered model for you. On Runpod, download the model files first and bake them into the image: + +```bash +az ml model download --name my-model --version 1 --download-path ./model \ + -g RESOURCE_GROUP -w WORKSPACE +``` + +Check the layout of the downloaded folder and make sure `model.pt` ends up directly under `./model`. Then build the worker image: + +```dockerfile Dockerfile +FROM runpod/pytorch:1.0.3-cu1281-torch291-ubuntu2404 + +RUN pip install --no-cache-dir --break-system-packages --ignore-installed cryptography runpod + +COPY model/ /models/ +COPY handler.py /handler.py + +CMD ["python3", "-u", "/handler.py"] +``` + +The two extra `pip` flags are needed on this base image: Ubuntu 24.04 marks the system Python as externally managed, and the image's `cryptography` package is installed by the OS, so `pip` can't upgrade it in place. + +Build and push it for the right platform: + +```bash +docker build --platform linux/amd64 -t myregistry.azurecr.io/inference/worker:v1 . +docker push myregistry.azurecr.io/inference/worker:v1 +``` + +Baking works well for models up to a few gigabytes. For larger models, keep the image small and load weights from a [network volume](/storage/network-volumes) at `/runpod-volume`, or use [cached models](/serverless/endpoints/model-caching) for models hosted on Hugging Face. + +### Translate a custom container or Container Apps app + +If your deployment is a custom container with `liveness_route`, `readiness_route`, and `scoring_route`, or a Container Apps app with ingress, your image already runs an HTTP server. A load balancing endpoint runs it with three changes: + +1. **Health check.** Runpod polls `GET /ping` on the worker. `200` means healthy, `204` means still initializing, and anything else means unhealthy. If your server already has a readiness path, set the `HEALTH_CHECK_PATH` environment variable to it instead of adding a route. +2. **Port.** Runpod routes to the port in the `PORT` environment variable (default `80`) and health checks the port in `PORT_HEALTH` (default `80`). Set both to your server's port, and list that port in the endpoint's `ports`. +3. **Auth and URL.** Callers send `Authorization: Bearer YOUR_API_KEY` to `https://ENDPOINT_ID.api.runpod.ai/YOUR_PATH`. Your routes and payloads stay the same. + +A FastAPI app that ran on Container Apps needs only the health route and the port: + +```python app.py +import os + +from fastapi import FastAPI +from pydantic import BaseModel + +app = FastAPI() + + +class GenerateRequest(BaseModel): + prompt: str + + +@app.get("/ping") +async def ping(): + return {"status": "healthy"} + + +@app.post("/generate") +async def generate(request: GenerateRequest): + return {"text": f"Generated for: {request.prompt}"} + + +if __name__ == "__main__": + import uvicorn + + uvicorn.run(app, host="0.0.0.0", port=int(os.getenv("PORT", "8000"))) +``` + +Return `204` from the health route while the model is still loading so the load balancer waits instead of marking the worker unhealthy. + +### Translate a training job or cloud-init script + +For an Azure Machine Learning command job, the job YAML's `command`, `environment`, and `compute` become a start command, an image, and a GPU type. For a VM, move what cloud-init installed into a Dockerfile and what it ran into the image's `CMD`. To keep the job as Python and skip the image entirely, use `@app.task` from the Runpod SDK, which runs each call on an ephemeral Pod: + +```python train.py +import runpod + +app = runpod.App("training") +checkpoints = runpod.NetworkVolume("checkpoints", size=100) + + +@app.task(gpu="H100", mounts={"/checkpoints": checkpoints}, dependencies=["torch"]) +def train(epochs: int = 10): + import torch + + model = torch.nn.Linear(16, 1).to("cuda") + torch.save(model.state_dict(), checkpoints.path / "model.pt") + return {"epochs": epochs} + + +@runpod.local_entrypoint +def main(): + print(train.remote(epochs=10)) +``` + +Run it with `rp flash dev train.py`. The Pod is created for the call and terminated when the function returns. The network volume keeps the checkpoint, and keeps billing for storage until you delete it. + +## Step 5: Move your data and models + +### Keep data in Blob Storage + +In a hybrid setup, the simplest option is to leave data in Blob Storage and read it over HTTPS from Runpod. Create a SAS token scoped to the container and the permissions the workload needs, or a service principal with a narrow role assignment, and store it as a Runpod secret: + +```bash +curl -s -X POST https://api.runpod.io/v2/account/secrets \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"name": "azure_container_sas_url", "value": "https://ACCOUNT.blob.core.windows.net/CONTAINER?SAS_TOKEN"}' +``` + +Reference the secret from an environment variable on the Pod, template, or endpoint by setting the value to `{{ RUNPOD_SECRET_azure_container_sas_url }}`. Inside the workload, the Azure SDK reads it like any other URL: + +```python +import os + +from azure.storage.blob import ContainerClient + +container = ContainerClient.from_container_url(os.environ["AZURE_CONTAINER_SAS_URL"]) +with open("/workspace/data/train.parquet", "wb") as f: + f.write(container.download_blob("datasets/train.parquet").readall()) +``` + +Code that uses `DefaultAzureCredential` keeps working if you set `AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, and `AZURE_CLIENT_SECRET` from secrets, because the credential chain reads a service principal from those variables. Managed identity only works on Azure-hosted compute. If the storage account's firewall restricts access to selected networks, requests from Runpod arrive from the public internet, so plan for public network access protected by the SAS token or service principal. + + +Secret names can't start with `runpod`. A running Pod or worker keeps the value it booted with, so restart it after you rotate a SAS token or client secret. + + +### Copy data to a network volume + +For training data and model weights that workers read often, copy the data to a network volume. Create the volume in the data center where you plan to run, because a network volume and the Pods or workers that mount it must be in the same data center: + +```bash +runpodctl network-volume create --name training-data --size 500 --data-center-id US-KS-2 +``` + +Pick the data center after you check GPU stock there. Then choose a transfer method: + + + + Start a Pod with the volume attached, install [AzCopy](https://learn.microsoft.com/azure/storage/common/storage-use-azcopy-v10) from Microsoft's Linux download, and sync straight from Blob Storage to the volume: + + ```bash + azcopy sync \ + "https://ACCOUNT.blob.core.windows.net/CONTAINER/datasets?SAS_TOKEN" \ + "/workspace/datasets" + ``` + + The data goes directly from Azure to Runpod. `azcopy sync` is incremental, so you can rerun it before cutover to pick up changes. A CPU Pod is enough for the copy. + + + In the console, open a Pod with the volume attached, click **Cloud Sync**, and select **Azure Blob Storage**. Enter the storage account name, an account key, and the container path. See [Cloud Sync](/pods/storage/cloud-sync#microsoft-azure-blob-storage). Cloud Sync uses the account key, which grants full access to the storage account, so rotate the key afterward or use AzCopy with a SAS token. + + + From any machine that has the files, such as an Azure VM with the disk or file share mounted, upload through the volume's [S3-compatible API](/storage/s3-api) with no Pod running: + + ```bash + aws s3 sync ./datasets s3://VOLUME_ID/datasets \ + --region US-KS-2 \ + --endpoint-url https://s3api-us-ks-2.runpod.io/ + ``` + + This needs the AWS CLI configured with an S3 API key from the Credentials page, and a volume in a [data center that supports the S3-compatible API](/storage/s3-api#datacenter-availability). Use the region and endpoint URL listed for your volume's data center. Rerun the command to resume an interrupted upload. `aws s3 sync` can fail on directories with more than 10,000 files, so split very large trees into batches. + + + +rclone's Azure Blob backend also works from inside a Pod if you already use rclone. + +### Map each storage type + +| Azure storage | Move it to | How | +| --- | --- | --- | +| OS disk contents | The container image | Rebuild as a Dockerfile. A VM disk image can't be imported. | +| Managed data disk | Network volume | Mount the disk on a VM, then use `aws s3 sync` against the volume's S3-compatible API, or rsync over SSH to a Pod. | +| Azure Files, Azure NetApp Files, Managed Lustre | Network volume, or [high-performance storage](/storage/high-performance-storage) | Copy from a VM that mounts the share. | +| Blob container mounted with BlobFuse | Network volume, or a [global volume](/storage/globalvolume/overview) for read-mostly data shared across data centers | Copy with AzCopy from a Pod. Global volumes are in beta and aren't fully POSIX compliant. | +| Azure Machine Learning registered model | Image, network volume, or [cached model](/serverless/endpoints/model-caching) | `az ml model download`, then copy. | +| Models pulled from Hugging Face | Cached models | Set a model reference on the endpoint. | + +Network volumes mount at `/workspace` on Pods by default and at `/runpod-volume` on Serverless workers, so use an environment variable for model and data paths if the same code runs on both. + +## Step 6: Deploy on Runpod + +### Pod (from a GPU VM) + +On Azure, a GPU VM that boots a training container looks like this: + +```bash +az vm create -g my-rg -n trainer \ + --image Ubuntu2204 --size Standard_NC24ads_A100_v4 \ + --admin-username azureuser --generate-ssh-keys \ + --data-disk-sizes-gb 500 \ + --custom-data cloud-init.txt +``` + +The cloud-init file prepares the host and runs the container. This one is trimmed to the last step, after the GPU driver install, Docker install, and data disk mount: + +```yaml cloud-init.txt +#cloud-config +runcmd: + - docker run --gpus all -v /mnt/data:/workspace -e EPOCHS=10 myregistry.azurecr.io/train:v1 python /app/train.py +``` + +On Runpod, the Pod is the container. The image, environment, volume, and command go straight on the create call: + +```bash +runpodctl pod create \ + --name trainer \ + --image myregistry.azurecr.io/train:v1 \ + --registry-auth-id REGISTRY_AUTH_ID \ + --gpu-id "NVIDIA A100 80GB PCIe" \ + --min-cuda-version 12.8 \ + --container-disk-in-gb 50 \ + --network-volume-id VOLUME_ID \ + --data-center-ids US-KS-2 \ + --ports "22/tcp,8888/http" \ + --env '{"EPOCHS":"10"}' \ + --docker-args "python /app/train.py" +``` + +The network volume mounts at `/workspace`. If the create fails for lack of stock, try another GPU type that fits, or another data center before you've put data on a volume there. Follow the Pod with `runpodctl pod logs POD_ID --follow` and wait for `runtimeStatus` to read `running` in `runpodctl pod get POD_ID`. The same create is available as `POST https://api.runpod.io/v2/pods`, and in the console under **Pods > Deploy**. + + +A Pod bills until you stop or delete it, even after its command exits. End a batch job by deleting the Pod (`runpodctl pod delete POD_ID`) from your pipeline, or use `@app.task`, which terminates the Pod for you. + + +### Queue-based endpoint (from a managed online endpoint) + +On Azure, the endpoint and deployment are two YAML files: + +```yaml endpoint.yml +$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineEndpoint.schema.json +name: my-endpoint +auth_mode: key +``` + +```yaml blue-deployment.yml +$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineDeployment.schema.json +name: blue +endpoint_name: my-endpoint +model: azureml:my-model:1 +code_configuration: + code: ./src + scoring_script: score.py +environment: azureml:my-pytorch-env:1 +instance_type: Standard_NC4as_T4_v3 +instance_count: 1 +request_settings: + request_timeout_ms: 60000 +``` + +You create them with `az ml online-endpoint create -f endpoint.yml` and `az ml online-deployment create -f blue-deployment.yml --all-traffic`. + +On Runpod, the image from [Step 4](#translate-a-scoring-script-to-a-handler) holds the code, model, and environment. Create a Serverless template for it, then an endpoint: + +```bash +runpodctl template create \ + --name my-endpoint-worker \ + --image myregistry.azurecr.io/inference/worker:v1 \ + --registry-auth-id REGISTRY_AUTH_ID \ + --container-disk-in-gb 20 \ + --serverless + +runpodctl serverless create \ + --name my-endpoint \ + --template-id TEMPLATE_ID \ + --gpu-id AMPERE_16 \ + --min-cuda-version 12.8 \ + --workers-min 0 \ + --workers-max 3 \ + --execution-timeout 60 +``` + +`--gpu-id` takes a pool ID here, so workers can land on any 16 GB GPU in the pool. `--workers-min 0` scales to zero. Set it to 1 or more to keep a worker running, which is the closest match to `instance_count: 1`. `--workers-max` is the ceiling the autoscaler can reach. Send a first request to check it: + +```bash +curl -s -X POST https://api.runpod.ai/v2/ENDPOINT_ID/runsync \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"input": {"data": [[1.0, 2.0, 3.0, 4.0]]}}' +``` + +The first request includes a cold start while the worker pulls the image and loads the model. If workers don't start, read `runpodctl serverless logs ENDPOINT_ID`. + +### Load balancing endpoint (from Container Apps or a custom container) + +On Azure, a Container Apps GPU app is created against a GPU workload profile: + +```bash +az containerapp create \ + --name gpu-api --resource-group my-rg --environment my-env \ + --image myregistry.azurecr.io/gpu-api:v1 \ + --registry-server myregistry.azurecr.io \ + --registry-username "$SP_APP_ID" --registry-password "$SP_PASSWORD" \ + --workload-profile-name gpu-t4 \ + --cpu 8.0 --memory 56.0Gi \ + --target-port 8000 --ingress external \ + --min-replicas 0 --max-replicas 3 \ + --scale-rule-name http-rule --scale-rule-type http --scale-rule-http-concurrency 4 +``` + +On Runpod, create the endpoint through REST API v2. The endpoint type is set at creation and can't be changed later, and `runpodctl serverless create` creates queue-based endpoints: + +```bash +curl -s -X POST https://api.runpod.io/v2/serverless \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "gpu-api", + "type": "LOAD_BALANCER", + "image": "myregistry.azurecr.io/gpu-api:v1", + "registry": "REGISTRY_AUTH_ID", + "gpu": {"pools": ["AMPERE_16"], "count": 1, "minCudaVersion": "12.8"}, + "disk": 20, + "ports": ["8000/http"], + "env": {"PORT": "8000", "PORT_HEALTH": "8000"}, + "workers": {"min": 0, "max": 3, "idleTimeout": 30}, + "scaling": {"type": "REQUEST_COUNT", "requestCount": 4}, + "timeout": 300000 + }' +``` + +The field mapping is direct: `--target-port` becomes `ports` plus `PORT` and `PORT_HEALTH`, `--min-replicas` and `--max-replicas` become `workers.min` and `workers.max`, and the HTTP concurrency scale rule becomes `REQUEST_COUNT` scaling, which is the only scaler a load balancing endpoint accepts. `timeout` is the per-request execution timeout in milliseconds. The response includes `requestUrls.base`, the URL your clients call. In the console, choose **Load Balancer** as the endpoint type when you create the endpoint. + +Load balancing endpoints don't queue. If no worker is ready within 2 minutes the request fails, so clients need a retry with backoff for cold starts, and long-running or must-not-drop work belongs on a queue-based endpoint. + +### Public Endpoint or vLLM endpoint (from Azure OpenAI or Foundry models) + +If the open-weight model you need is in the [Public Endpoint model reference](/public-endpoints/reference), there's nothing to deploy: go to [Step 7](#openai-compatible-clients). To serve a different open-weight model, deploy the vLLM worker from the Hub with `runpodctl`: + +```bash +runpodctl hub search vllm +runpodctl serverless create --hub-id HUB_ID --name my-llm \ + --env MODEL_NAME=Qwen/Qwen2.5-0.5B-Instruct +``` + +The endpoint serves an OpenAI-compatible API at `https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1`. See [Get started with vLLM](/serverless/vllm/get-started) for GPU sizing and gated models. + +### Instant Cluster (from CycleCloud or multi-node ND VMs) + +Create a Slurm cluster from the console under **Instant Clusters > Create Cluster**, selecting **Slurm Cluster**, the GPU type, the Pod count, and an optional network volume in the same region. Slurm and munge are preconfigured, and managed Slurm Clusters require an official Runpod PyTorch image. See [Slurm Clusters](/instant-clusters/slurm-clusters). Multi-node stock for a given GPU type varies, so confirm it in the console before you schedule a run. + +## Step 7: Update the code that calls Azure + +### Scoring requests + +Azure clients call the scoring URI with an endpoint key, or go through the SDK: + + + +```python REST (before) +import os + +import requests + +response = requests.post( + os.environ["AZUREML_SCORING_URI"], + headers={ + "Authorization": f"Bearer {os.environ['AZUREML_ENDPOINT_KEY']}", + "Content-Type": "application/json", + }, + json={"data": [[1.0, 2.0, 3.0, 4.0]]}, + timeout=60, +) +response.raise_for_status() +prediction = response.json() +``` + +```python SDK (before) +from azure.ai.ml import MLClient +from azure.identity import DefaultAzureCredential + +ml_client = MLClient( + DefaultAzureCredential(), "SUBSCRIPTION_ID", "RESOURCE_GROUP", "WORKSPACE" +) +raw = ml_client.online_endpoints.invoke( + endpoint_name="my-endpoint", + deployment_name="blue", + request_file="sample-request.json", +) +``` + + + +On a queue-based endpoint, wrap the payload in `input` and read the result from `output`: + + + +```python REST (after) +import os + +import requests + +response = requests.post( + f"https://api.runpod.ai/v2/{os.environ['RUNPOD_ENDPOINT_ID']}/runsync", + headers={ + "Authorization": f"Bearer {os.environ['RUNPOD_API_KEY']}", + "Content-Type": "application/json", + }, + json={"input": {"data": [[1.0, 2.0, 3.0, 4.0]]}}, + timeout=120, +) +response.raise_for_status() +job = response.json() +if job["status"] != "COMPLETED": + raise RuntimeError(f"Job {job['id']} ended as {job['status']}") +prediction = job["output"] +``` + +```python SDK (after) +import os + +import runpod + +endpoint = runpod.Endpoint(os.environ["RUNPOD_ENDPOINT_ID"]) +prediction = endpoint.run_sync({"data": [[1.0, 2.0, 3.0, 4.0]]}) +``` + + + +Check `status` as well as the HTTP code, because a job that fails in the handler reports `"status": "FAILED"` in the response body. `/runsync` waits 90 seconds by default (adjust with the `?wait=` query parameter, in milliseconds). If the job is still running when the wait ends, the status is `IN_QUEUE` or `IN_PROGRESS`, and you poll `/status/JOB_ID` for the result. + +For a load balancing endpoint, only the base URL and the auth header change: + +```python +import os + +import requests + +response = requests.post( + f"https://{os.environ['RUNPOD_ENDPOINT_ID']}.api.runpod.ai/generate", + headers={"Authorization": f"Bearer {os.environ['RUNPOD_API_KEY']}"}, + json={"prompt": "Hello"}, + timeout=120, +) +response.raise_for_status() +``` + +### Batch and long-running requests + +A batch endpoint invocation (`az ml batch-endpoint invoke --name my-batch-endpoint --input ...`) starts a job against data in storage. On Runpod, submit one `/run` request per unit of work, pass the data location in the input, and let a webhook notify you: + +```python +import os + +import requests + +response = requests.post( + f"https://api.runpod.ai/v2/{os.environ['RUNPOD_ENDPOINT_ID']}/run", + headers={"Authorization": f"Bearer {os.environ['RUNPOD_API_KEY']}"}, + json={ + "input": {"blob_path": "batches/2026-11-14/part-0001.jsonl"}, + "webhook": "https://example.com/hooks/runpod", + "policy": {"executionTimeout": 3600000, "ttl": 86400000}, + }, + timeout=30, +) +job_id = response.json()["id"] +``` + +The handler reads the blob with the SAS URL from [Step 5](#keep-data-in-blob-storage) and writes results back to Blob Storage. Runpod posts the job result to the webhook when the job finishes, and retries the call up to two more times if your server doesn't return `200`. Results from `/run` jobs also stay available at `/status/JOB_ID` for 30 minutes after the job completes. Azure Machine Learning parallelizes a batch job across files for you. On Runpod you fan out by submitting one job per file or shard, and the endpoint's worker count sets the parallelism. + +### OpenAI-compatible clients + +Azure OpenAI code uses either the `AzureOpenAI` client with `azure_endpoint`, `api_version`, and a deployment name, or the newer v1 API through the standard `OpenAI` client: + + + +```python AzureOpenAI client (before) +import os + +from openai import AzureOpenAI + +client = AzureOpenAI( + azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"], + api_key=os.environ["AZURE_OPENAI_API_KEY"], + api_version="2024-10-21", +) + +response = client.chat.completions.create( + model="my-deployment-name", + messages=[{"role": "user", "content": "Hello"}], +) +``` + +```python v1 API (before) +import os + +from openai import OpenAI + +client = OpenAI( + base_url="https://YOUR-RESOURCE-NAME.openai.azure.com/openai/v1/", + api_key=os.environ["AZURE_OPENAI_API_KEY"], +) + +response = client.chat.completions.create( + model="my-deployment-name", + messages=[{"role": "user", "content": "Hello"}], +) +``` + + + +On Runpod, use the standard `OpenAI` client with a Runpod base URL and API key. There's no `api_version`, and `model` is the model name, not a deployment name: + + + +```python Public Endpoint (after) +import os + +from openai import OpenAI + +client = OpenAI( + base_url="https://api.runpod.ai/v2/qwen3-32b-awq/openai/v1", + api_key=os.environ["RUNPOD_API_KEY"], +) + +response = client.chat.completions.create( + model="Qwen/Qwen3-32B-AWQ", + messages=[{"role": "user", "content": "Hello"}], +) +``` + +```python Your vLLM endpoint (after) +import os + +from openai import OpenAI + +client = OpenAI( + base_url=f"https://api.runpod.ai/v2/{os.environ['RUNPOD_ENDPOINT_ID']}/openai/v1", + api_key=os.environ["RUNPOD_API_KEY"], +) + +response = client.chat.completions.create( + model="Qwen/Qwen2.5-0.5B-Instruct", + messages=[{"role": "user", "content": "Hello"}], +) +``` + + + +vLLM workers serve `/chat/completions`, `/completions`, and `/models`. If your code uses the Responses API, Azure content filtering results, or Entra ID token providers, replace those parts: call chat completions, apply your own moderation, and authenticate with the API key. Changing models changes output quality and behavior, so rerun your evaluations before you switch production traffic. + +### Control-plane automation + +Replace Azure CLI and SDK calls in CI and scripts with `runpodctl`, REST API v2, or the Runpod SDK. The [CLI command map](#cli-command-map) and [API map](#api-map) list the equivalents. Store `RUNPOD_API_KEY` in your CI secret store in place of the service principal credentials used for `az login`. + +## Step 8: Verify, cut over, and shut down + +1. **Send real requests from outside Runpod.** Use production-shaped payloads, not a health check. For Pods, connect over SSH or the proxy URL and run the actual job. +2. **Compare outputs.** Send the same inputs to both sides and diff the results. Expect small numeric differences if the GPU model changed. +3. **Measure latency, including cold starts.** Check the endpoint's **Metrics** tab for delay time, execution time, and cold start time. If cold starts are too slow, raise the minimum workers or the idle timeout, or enable FlashBoot. +4. **Run both in parallel.** Shift a share of traffic in your client or gateway, watch error rates, then raise the share. Keep the Azure deployment until you've seen a full traffic cycle. +5. **Set a cost guard on Runpod.** Cap `workers-max` on every endpoint, enable [low balance notifications](/accounts-billing/billing), and review `runpodctl billing pods` and `runpodctl billing serverless` after the first day. + +When Runpod has carried full traffic for long enough, stop the matching Azure charges. Deleting a resource group removes everything in it, so only do that if nothing else lives there. Otherwise, go resource by resource: + +```bash +# VMs: a stopped VM that isn't deallocated still bills for compute. +az vm deallocate -g RESOURCE_GROUP -n VM_NAME +az vm delete -g RESOURCE_GROUP -n VM_NAME --yes + +# Azure Machine Learning: deployments bill for their instances until deleted. +az ml online-endpoint delete --name ENDPOINT_NAME -g RESOURCE_GROUP -w WORKSPACE --yes +az ml compute stop --name COMPUTE_INSTANCE -g RESOURCE_GROUP -w WORKSPACE +az ml compute delete --name COMPUTE_NAME -g RESOURCE_GROUP -w WORKSPACE --yes + +# Container Apps +az containerapp delete -g RESOURCE_GROUP -n APP_NAME --yes +``` + +Then check for what's commonly left behind: + +- **Managed disks and snapshots.** Depending on each VM's delete options, deleting the VM can leave its OS disk, data disks, NIC, and public IP. List unattached disks with ``az disk list --query '[?managedBy==`null`].[id]' -o tsv``, and review `az snapshot list`. +- **Public IP addresses.** Review `az network public-ip list --output table` and delete addresses that are no longer attached. +- **Compute instances and clusters.** A compute instance bills while it's running, and a stopped one keeps holding quota, so delete the ones you won't restart. Compute clusters with a minimum node count above zero keep nodes running. +- **Online deployments.** Deleting the endpoint deletes its deployments. If you keep the endpoint, delete unused deployments with `az ml online-deployment delete`. +- **GPU workload profiles.** Dedicated GPU workload profiles in a Container Apps environment are allocated per node, not per app, so remove the profile (or the environment) when no app needs it. +- **ACR storage.** Delete repositories you no longer pull from with `az acr repository delete`, or keep the registry if Runpod still pulls from it. +- **Log Analytics workspaces and Application Insights.** Workspaces created alongside Container Apps environments and Azure Machine Learning workspaces outlive the workloads and can keep accruing charges for the data they hold. List them with `az monitor log-analytics workspace list`. +- **Storage accounts.** Delete Blob containers and file shares only after you've verified the copy on Runpod and you have another backup. +- **Reservations and savings plans.** These are commitments that continue after the resources are gone. Review them in the Azure portal. +- **Azure OpenAI and Foundry deployments.** Delete provisioned deployments you no longer call. + +## Migrate your Terraform + +The [Runpod Terraform provider](/integrations/terraform/overview) 2.x is built on REST API v2 and manages Pods, Serverless endpoints, templates, network and global volumes, registry credentials, and Instant Clusters. If you use Bicep or ARM templates, there's no Runpod equivalent: use Terraform or call REST API v2 from your pipeline. + +### Resource map + +| Azure resource | Runpod resource | Notes | +| --- | --- | --- | +| `azurerm_linux_virtual_machine` (GPU size) | `runpod_pod` | `size` becomes `gpu_type_id` and `gpu_count`. `custom_data` becomes the image and `args`. | +| `azurerm_linux_virtual_machine_scale_set` | `runpod_serverless_endpoint` | For request-driven scaling. Use `count` or `for_each` on `runpod_pod` for a fixed fleet. | +| `azurerm_network_interface`, `azurerm_public_ip`, `azurerm_network_security_group`, `azurerm_virtual_network`, `azurerm_subnet` | `ports` on `runpod_pod` | No network resources to declare. | +| `azurerm_managed_disk` | `persistent_volume` on `runpod_pod`, or `runpod_network_volume` | Prefer a network volume for data you can't recreate. | +| `azurerm_storage_account`, `azurerm_storage_share` | `runpod_network_volume` or `runpod_global_volume` | Or keep the storage account and read it from Runpod. | +| `azurerm_container_registry` | `runpod_registry` | Stores pull credentials. The registry itself can stay on Azure. | +| `azurerm_machine_learning_workspace` | None | Keep it if you still use the workspace for tracking or the model registry. | +| `azurerm_machine_learning_compute_cluster` | `runpod_pod`, or the Runpod SDK `@app.task` outside Terraform | Runpod has no idle cluster to declare. | +| `azurerm_machine_learning_compute_instance` | `runpod_pod` with `start_jupyter = true` | | +| `azapi_resource` of type `Microsoft.MachineLearningServices/workspaces/onlineEndpoints` and `.../deployments` | `runpod_serverless_endpoint` | The `azurerm` provider has no online endpoint resource, so these are usually `azapi_resource` blocks or `az ml` calls. | +| `azurerm_container_app_environment`, `azurerm_container_app` (GPU workload profile) | `runpod_template` and `runpod_serverless_endpoint` with `endpoint_type = "LOAD_BALANCER"` | | +| `azurerm_kubernetes_cluster_node_pool` (GPU) | `runpod_serverless_endpoint` or `runpod_pod` per workload | | +| `azurerm_key_vault_secret` | Runpod secret, referenced as `{{ RUNPOD_SECRET_name }}` in `env` | Create the secret with REST API v2 or the console. | + +### Run both providers side by side + +During the transition, declare both providers in one configuration so a single plan shows what's added on Runpod and what's removed from Azure: + +```hcl providers.tf +terraform { + required_version = ">= 1.6" + + required_providers { + azurerm = { + source = "hashicorp/azurerm" + version = "~> 5.0" + } + runpod = { + source = "runpod/runpod" + version = "~> 2.0" + } + } +} + +provider "azurerm" { + features {} + subscription_id = var.subscription_id +} + +# Reads the key from the RUNPOD_API_KEY environment variable. +provider "runpod" {} +``` + +Keep whatever `azurerm` version constraint you already use. Add the Runpod resources first and apply, verify the workload, move traffic, and only then remove the Azure resource blocks in a separate change, so `terraform destroy` behavior for the Azure side is reviewed on its own. Resources from the two providers can reference each other, for example an Azure storage account name in a Runpod endpoint's `env`. + +### Example 1: GPU Linux VM to a Pod + +Before, a GPU VM needs a network interface, a disk, and the VM itself, on top of an existing resource group, virtual network, and subnet: + +```hcl azure-vm.tf +resource "azurerm_network_interface" "trainer" { + name = "trainer-nic" + location = azurerm_resource_group.ml.location + resource_group_name = azurerm_resource_group.ml.name + + ip_configuration { + name = "internal" + subnet_id = azurerm_subnet.ml.id + private_ip_address_allocation = "Dynamic" + } +} + +resource "azurerm_linux_virtual_machine" "trainer" { + name = "trainer" + resource_group_name = azurerm_resource_group.ml.name + location = azurerm_resource_group.ml.location + size = "Standard_NC24ads_A100_v4" + admin_username = "azureuser" + network_interface_ids = [azurerm_network_interface.trainer.id] + custom_data = base64encode(file("${path.module}/cloud-init.txt")) + + admin_ssh_key { + username = "azureuser" + public_key = file("~/.ssh/id_rsa.pub") + } + + os_disk { + caching = "ReadWrite" + storage_account_type = "Premium_LRS" + disk_size_gb = 256 + } + + source_image_reference { + publisher = "Canonical" + offer = "0001-com-ubuntu-server-jammy" + sku = "22_04-lts" + version = "latest" + } +} +``` + +After, the Pod, its storage, and the credential for pulling from ACR are three resources: + +```hcl runpod-pod.tf +variable "acr_pull_username" { + type = string +} + +variable "acr_pull_password" { + type = string + sensitive = true +} + +resource "runpod_registry" "acr" { + name = "acr-myregistry" + username = var.acr_pull_username + password = var.acr_pull_password +} + +resource "runpod_network_volume" "training" { + name = "training-data" + size = 500 + data_center = "US-KS-2" +} + +resource "runpod_pod" "trainer" { + name = "trainer" + image = "myregistry.azurecr.io/train:v1" + registry = runpod_registry.acr.id + gpu_type_id = "NVIDIA A100 80GB PCIe" + gpu_count = 1 + min_cuda_version = "12.8" + cloud = "SECURE" + disk = 50 + data_center_ids = [runpod_network_volume.training.data_center] + ports = ["22/tcp", "8888/http"] + start_ssh = true + args = "python /app/train.py" + + network_volumes = [{ + volume_id = runpod_network_volume.training.id + path = "/workspace" + }] + + env = { + EPOCHS = "10" + AZURE_CONTAINER_SAS_URL = "{{ RUNPOD_SECRET_azure_container_sas_url }}" + } +} + +output "pod_id" { + value = runpod_pod.trainer.id +} +``` + +`start_ssh` injects the SSH keys registered on your Runpod account, which replaces `admin_ssh_key`, and needs an image that honors the convention (the official Runpod images do). `disk` is the ephemeral container disk, so anything that must survive goes on the network volume. Changing `gpu_type_id`, `gpu_count`, `cloud`, or `data_center_ids` replaces the Pod. To stop the Pod without removing it from the configuration, set `desired_state = "EXITED"`. + +### Example 2: Container Apps GPU app to a Serverless endpoint + +Before, a GPU app needs an environment with a GPU workload profile and the app with its registry secret, ingress, and scale settings: + +```hcl azure-container-app.tf +resource "azurerm_container_app_environment" "gpu" { + name = "gpu-env" + location = azurerm_resource_group.ml.location + resource_group_name = azurerm_resource_group.ml.name + log_analytics_workspace_id = azurerm_log_analytics_workspace.ml.id + + workload_profile { + name = "gpu-t4" + workload_profile_type = "Consumption-GPU-NC8as-T4" + } +} + +resource "azurerm_container_app" "api" { + name = "gpu-api" + container_app_environment_id = azurerm_container_app_environment.gpu.id + resource_group_name = azurerm_resource_group.ml.name + revision_mode = "Single" + workload_profile_name = "gpu-t4" + + secret { + name = "acr-password" + value = var.acr_pull_password + } + + registry { + server = azurerm_container_registry.ml.login_server + username = var.acr_pull_username + password_secret_name = "acr-password" + } + + template { + min_replicas = 0 + max_replicas = 3 + + container { + name = "api" + image = "${azurerm_container_registry.ml.login_server}/gpu-api:v1" + cpu = 8.0 + memory = "56Gi" + + env { + name = "MODEL_NAME" + value = "my-model" + } + } + + http_scale_rule { + name = "http-rule" + concurrent_requests = "4" + } + } + + ingress { + external_enabled = true + target_port = 8000 + + traffic_weight { + latest_revision = true + percentage = 100 + } + } +} +``` + +After, a Serverless template holds the container settings and the endpoint holds the GPU pool and scaling. It reuses `runpod_registry.acr` from the first example: + +```hcl runpod-endpoint.tf +resource "runpod_template" "api" { + name = "gpu-api" + image = "${azurerm_container_registry.ml.login_server}/gpu-api:v1" + registry = runpod_registry.acr.id + serverless = true + disk = 20 + ports = ["8000/http"] + start_ssh = false + start_jupyter = false + + env = { + MODEL_NAME = "my-model" + PORT = "8000" + PORT_HEALTH = "8000" + } +} + +resource "runpod_serverless_endpoint" "api" { + name = "gpu-api" + template_id = runpod_template.api.id + endpoint_type = "LOAD_BALANCER" + gpu_pools = ["AMPERE_16"] + gpu_count = 1 + min_cuda_version = "12.8" + scaling_type = "REQUEST_COUNT" + scaling_value = 4 + workers_min = 0 + workers_max = 3 + idle_timeout = 30 + flashboot = "FLASHBOOT" + timeout = 300000 +} + +output "api_base_url" { + value = runpod_serverless_endpoint.api.request_urls.base +} +``` + +`endpoint_type` is required and changing it replaces the endpoint. A load balancing endpoint only accepts `REQUEST_COUNT` scaling. The `request_urls.base` output replaces the Container App's ingress FQDN in your clients. + +For an Azure Machine Learning online deployment declared with `azapi_resource`, the target is the same pair of resources with a queue-based endpoint: + +```hcl runpod-queue-endpoint.tf +resource "runpod_template" "scoring" { + name = "my-endpoint-worker" + image = "myregistry.azurecr.io/inference/worker:v1" + registry = runpod_registry.acr.id + serverless = true + disk = 20 + start_ssh = false + start_jupyter = false +} + +resource "runpod_serverless_endpoint" "scoring" { + name = "my-endpoint" + template_id = runpod_template.scoring.id + endpoint_type = "QUEUE" + gpu_pools = ["AMPERE_16"] + scaling_type = "QUEUE_DELAY" + scaling_value = 4 + workers_min = 0 + workers_max = 3 + timeout = 60000 +} + +output "scoring_runsync_url" { + value = runpod_serverless_endpoint.scoring.request_urls.run_sync +} +``` + +`instanceType` and `sku.capacity` from the `azapi_resource` body become `gpu_pools` and the `workers_min` to `workers_max` range. `timeout` is in milliseconds, like `request_timeout_ms`. For every argument, see the [provider reference on the Terraform Registry](https://registry.terraform.io/providers/runpod/runpod/latest/docs). To report a provider problem, [open an issue on GitHub](https://github.com/runpod/terraform-provider-runpod/issues). + +## Reference + +### CLI command map + +| Azure CLI | Runpod | +| --- | --- | +| `az login` | `export RUNPOD_API_KEY=...`, or `rp login` for the SDK | +| `az vm list-skus -l LOCATION` | `runpodctl gpu list`, `runpodctl datacenter list` | +| `az vm create` | `runpodctl pod create` | +| `az vm list -d` | `runpodctl pod list --all` | +| `az vm show` | `runpodctl pod get POD_ID` | +| `az vm start`, `az vm restart` | `runpodctl pod start POD_ID`, `runpodctl pod restart POD_ID` | +| `az vm deallocate` | `runpodctl pod stop POD_ID` | +| `az vm delete` | `runpodctl pod delete POD_ID` | +| `ssh azureuser@IP` | `runpodctl ssh info POD_ID`, then `ssh` | +| `az ml online-endpoint create` plus `az ml online-deployment create` | `runpodctl template create --serverless` plus `runpodctl serverless create` | +| `az ml online-endpoint list`, `show` | `runpodctl serverless list`, `runpodctl serverless get ENDPOINT_ID` | +| `az ml online-deployment update --set instance_count=N` | `runpodctl serverless update ENDPOINT_ID --workers-min N --workers-max M` | +| `az ml online-endpoint update --traffic "blue=90 green=10"` | No equivalent. Split across two endpoints in your client. | +| `az ml online-endpoint invoke` | `curl https://api.runpod.ai/v2/ENDPOINT_ID/runsync` | +| `az ml online-deployment get-logs` | `runpodctl serverless logs ENDPOINT_ID` | +| `az ml online-endpoint delete` | `runpodctl serverless delete ENDPOINT_ID` | +| `az ml job create`, `az ml job stream` | `runpodctl pod create`, `runpodctl pod logs POD_ID --follow`, or `rp flash dev` | +| `az ml compute create`, `stop`, `delete` | `runpodctl pod create`, `stop`, `delete` | +| `az containerapp create`, `az containerapp up` | `POST /v2/serverless` with `"type": "LOAD_BALANCER"` | +| `az containerapp update --image` | `runpodctl template update TEMPLATE_ID --image ...` | +| `az containerapp logs show` | `runpodctl serverless logs ENDPOINT_ID` | +| `az containerapp delete` | `runpodctl serverless delete ENDPOINT_ID` | +| `az acr login`, `az acr token create` | `runpodctl registry create` | +| `azcopy sync`, `azcopy copy` | `aws s3 sync` against the network volume [S3-compatible API](/storage/s3-api), or `azcopy` inside a Pod | +| `az monitor` metrics and log queries | `runpodctl pod logs`, `runpodctl serverless logs`, the endpoint **Metrics** tab | +| `az disk create` | `runpodctl network-volume create` | + +### API map + +| Azure call | Runpod REST API v2 | +| --- | --- | +| `PUT .../Microsoft.Compute/virtualMachines/NAME` | `POST https://api.runpod.io/v2/pods` | +| VM `start`, `deallocate`, `restart` actions | `POST /v2/pods/POD_ID/action` with `{"action": "start"}`, `"stop"`, or `"restart"` | +| `DELETE .../virtualMachines/NAME` | `DELETE /v2/pods/POD_ID` | +| `MLClient.online_endpoints.begin_create_or_update` plus `online_deployments.begin_create_or_update` | `POST /v2/serverless` | +| `MLClient.online_deployments.begin_create_or_update` (update) | `PATCH /v2/serverless/ENDPOINT_ID` | +| `MLClient.online_endpoints.invoke`, scoring URI | `POST https://api.runpod.ai/v2/ENDPOINT_ID/runsync` or `/run` | +| `MLClient.online_deployments.get_logs` | `GET /v2/serverless/ENDPOINT_ID/workers/WORKER_ID/logs` | +| `MLClient.online_endpoints.begin_delete` | `DELETE /v2/serverless/ENDPOINT_ID` | +| `MLClient.batch_endpoints.invoke` | `POST https://api.runpod.ai/v2/ENDPOINT_ID/run` with a `webhook` | +| Azure OpenAI `/openai/v1/chat/completions` | `https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1/chat/completions` | +| ACR pull credentials | `POST /v2/registries` | +| Key Vault `set secret` | `POST /v2/account/secrets` | +| Managed disk create | `POST /v2/network-volumes` | +| Resource SKUs list | `GET /v2/catalog/gpus`, `GET /v2/catalog/datacenters` | +| Cost Management queries | `GET /v2/billing/pods`, `/v2/billing/serverless`, `/v2/billing/network-volumes` | + +### Configuration field map + +| Azure field | Runpod field (REST API v2) | `runpodctl` flag | +| --- | --- | --- | +| VM `size`, deployment `instance_type` | `gpu.id` and `gpu.count` (Pods), `gpu.pools` and `gpu.count` (Serverless) | `--gpu-id`, `--gpu-count` | +| `location` | `dataCenterIds` | `--data-center-ids` | +| VM image, deployment `environment` | `image` | `--image` | +| `custom_data`, job `command` | `cmd`, `entrypoint`, or `args` | `--docker-args` (Pods), `--docker-start-cmd` (templates) | +| `environment_variables`, `--env-vars` | `env` | `--env` | +| OS disk size | `disk` | `--container-disk-in-gb` | +| Data disk | `mounts.network` or `mounts.persistent` | `--network-volume-id`, `--volume-in-gb` | +| NSG rules, `--target-port`, `--ingress` | `ports` | `--ports` | +| `instance_count`, `--min-replicas`, `--max-replicas` | `workers.min`, `workers.max` | `--workers-min`, `--workers-max` | +| Azure Monitor autoscale rule, HTTP scale rule | `scaling.type` with `queueDelay` or `requestCount` | `--scale-by`, `--scale-threshold` | +| `request_settings.request_timeout_ms` | `timeout` (milliseconds) | `--execution-timeout` (seconds) | +| `liveness_route`, `readiness_route` | `/ping`, or `HEALTH_CHECK_PATH` in `env` | Not applicable | +| `scoring_route.port`, `--target-port` | `PORT` and `PORT_HEALTH` in `env`, plus `ports` | Not applicable | +| ACR registry credentials | `registry` | `--registry-auth-id` | +| `auth_mode` | Not applicable. All endpoints use the Runpod API key. | Not applicable | +| Spot `priority`, `eviction_policy` | Not applicable in REST API v2 | Not applicable | + +### Environment variable map + +| Azure (inside the container) | Runpod | Notes | +| --- | --- | --- | +| `AZUREML_MODEL_DIR` | A path you choose, such as `/models` or `/runpod-volume/models` | Runpod doesn't mount a registered model. | +| `CONTAINER_APP_PORT` | `PORT` | Set on load balancing endpoints. Defaults to `80`. | +| `CONTAINER_APP_NAME`, `CONTAINER_APP_REPLICA_NAME` | `RUNPOD_POD_ID` | Identifies the Pod or Serverless worker. | +| Azure region from instance metadata | `RUNPOD_DC_ID` | Data center ID. | +| GPU count from the VM size | `RUNPOD_GPU_COUNT` | | +| Public IP and SSH port | `RUNPOD_PUBLIC_IP`, `RUNPOD_TCP_PORT_22` | Available when the Pod has a public IP. | +| `AZURE_CLIENT_ID`, `AZURE_TENANT_ID`, `AZURE_CLIENT_SECRET` | The same names, set from Runpod secrets | For calling back into Azure with a service principal. | + +### Status map + +| Azure VM power state | Runpod Pod status | Compute billing | +| --- | --- | --- | +| Creating | `PROVISIONING` | | +| Starting | `STARTING` | | +| Running | `RUNNING` | Billed on both. | +| Stopped (allocated) | No equivalent | Billed on Azure. | +| Deallocated | `EXITED` | Not billed on either. Disks on Azure and the volume disk on Runpod still bill. | +| Failed provisioning state | `ERROR` | A Pod in `ERROR` still exists, so delete it. | +| Deleted | `TERMINATED` | | + +Serverless jobs move through `IN_QUEUE`, `IN_PROGRESS`, and then `COMPLETED`, `FAILED`, `CANCELLED`, or `TIMED_OUT`. Workers report `INITIALIZING`, `IDLE`, `RUNNING`, `THROTTLED`, or `UNHEALTHY`. `THROTTLED` means the worker is waiting on GPU capacity. + +## Migrate with a coding agent + +A coding agent can run the inventory, propose the mapping, and migrate one workload at a time. Install the Runpod skills and MCP server first. + + + + The plugin also connects the hosted MCP server: + + ```bash + claude plugin marketplace add runpod/runpod-plugins-official + claude plugin install runpod@runpod + ``` + + + Install the skills, then the hosted MCP server. Replace `AGENT_ID` and `CLIENT_ID` with your agent's identifiers: + + ```bash + npx skills add runpod/runpod-plugins-official --skill '*' --agent AGENT_ID --global + npx @runpod/mcp-server@latest add --client CLIENT_ID + ``` + + + +The MCP server signs in with OAuth ("Sign in with Runpod"), so you don't need an API key to start. See [Agent skills](/get-started/agent-skills) and [MCP servers](/get-started/mcp-servers) for details. + +Then give the agent a prompt like this one: + +```text +Use the runpod-cloud-migrate skill to migrate this project's GPU workloads from Azure to Runpod. + +1. Inventory first. Scan this repo for Azure usage (azure.ai.ml, AzureOpenAI, score.py with + init()/run(), az ml / az vm / az containerapp commands, azurerm_ and azapi_ Terraform resources, + azurecr.io images, blob.core.windows.net URLs). If the Azure CLI is logged in, list GPU VMs, + Azure Machine Learning online and batch endpoints, compute, and Container Apps GPU apps + read-only. Don't change anything on Azure. +2. Show me a plan: each workload, its Runpod target (Pod, queue-based endpoint, load balancing + endpoint, Public Endpoint, Instant Cluster, or @app.task), the GPU type or pool, where data and + images will live, what stays on Azure, and an hourly cost estimate from live Runpod pricing. + List every difference that needs my decision (private networking, managed identity, proprietary + models, traffic splitting). +3. Wait for my approval before creating anything billable on Runpod. +4. Migrate one workload first. Translate the code, build a linux/amd64 image, deploy it, and verify + it with a real request. Show me the output next to the Azure output. +5. Update the client code and Terraform for that workload, with both providers side by side. +6. Never stop, deallocate, or delete Azure resources, and never rotate or delete Azure credentials, + without my explicit approval. When I approve, give me the shutdown commands to review first. +``` + +## Troubleshooting + +**The Pod or worker fails to pull an image from ACR.** The registry credential is wrong, expired, or scoped to another repository, or the registry blocks public network access. Check that `docker login myregistry.azurecr.io` works with the same username and password from outside Azure, that the token has `content/read` on that repository, and that the template or Pod references the registry credential ID. + +**The container exits immediately with an exec format error.** The image was built for ARM. Rebuild with `docker build --platform linux/amd64`. + +**CUDA or driver errors at startup that didn't happen on Azure.** The host's driver is older than your image's CUDA runtime expects. Set `--min-cuda-version` (or `gpu.minCudaVersion`) to match the CUDA version in your image. + +**The handler fails with a `TypeError` or `KeyError` on the input.** The code still treats the request as a raw JSON string or expects the payload at the top level. On Runpod, read `job["input"]`, and make sure clients send `{"input": {...}}`. + +**`os.getenv("AZUREML_MODEL_DIR")` returns `None`.** That variable is set only by Azure Machine Learning. Point the code at the path where you baked or mounted the model. + +**A load balancing worker shows as running but requests time out or return `502`.** The health check isn't passing. Make sure the server listens on `0.0.0.0`, the port is listed in `ports`, both `PORT` and `PORT_HEALTH` are set to it, and `GET /ping` (or your `HEALTH_CHECK_PATH`) returns `200`. + +**`{"error":"no workers available"}` on the first request.** The endpoint scaled to zero and the worker wasn't ready in time. Retry with backoff, or set the minimum workers to 1 for latency-sensitive endpoints. + +**`DefaultAzureCredential` fails inside a Pod or worker.** There's no managed identity outside Azure. Set `AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, and `AZURE_CLIENT_SECRET` from Runpod secrets, or switch to a SAS URL. + +**Requests to Blob Storage, Key Vault, or a database time out.** The Azure resource only allows traffic from a VNet or private endpoint. Enable its public endpoint with authentication and firewall rules, or keep the component that needs private access on Azure and call it through an API you expose. + +**Data written by a job is gone after the Pod restarts.** It was written to the container disk. Write to `/workspace` on a volume disk or network volume, or to `/runpod-volume` on Serverless. + +**A Pod create fails for lack of capacity.** The GPU type isn't in stock in that data center right now. Try another type that fits the workload or another data center, or deploy from the console with **Deploy when available**. If a network volume pins you to a data center, check stock there before you copy data. + +## Next steps + + + + How endpoints, workers, and scaling work. + + + Serve your own HTTP API on Serverless. + + + Manage Runpod resources as code. + + + Persistent storage for Pods and Serverless. + + diff --git a/migrate/baseten.mdx b/migrate/baseten.mdx new file mode 100644 index 000000000..0ba5d3896 --- /dev/null +++ b/migrate/baseten.mdx @@ -0,0 +1,1375 @@ +--- +title: "Migrate from Baseten to Runpod" +sidebarTitle: "Baseten" +description: "Move Truss models, custom servers, Model APIs calls, Chains, and training jobs from Baseten to Runpod Serverless, Public Endpoints, and Pods." +--- + +This guide is for teams that run production inference on [Baseten](https://docs.baseten.co/overview) and want to run it on Runpod. A [Truss](https://docs.baseten.co/development/model/overview) model with a `Model` class becomes a [Serverless](/serverless/overview) queue endpoint with a handler, a [custom Docker server](https://docs.baseten.co/development/model/custom-server) becomes a load balancing endpoint or the Runpod vLLM worker, [Model APIs](https://docs.baseten.co/inference/model-apis/overview) calls move to [Public Endpoints](/public-endpoints/overview) or your own vLLM endpoint, [Chains](https://docs.baseten.co/development/chain/overview) become several endpoints behind an orchestrator, and [Training Jobs](https://docs.baseten.co/training/overview) move to [Pods](/pods/overview). At the end you'll have each workload deployed on Runpod, your clients pointed at the new URLs, and your Baseten deployments deactivated. + + +You can hand this migration to a coding agent. See [Migrate with a coding agent](#migrate-with-a-coding-agent) for the setup commands and a prompt. + + +## How Baseten maps to Runpod + +| Baseten concept | Runpod equivalent | Notes | +| --- | --- | --- | +| Truss model (`config.yaml` + `model/model.py`) | Serverless queue endpoint with a [handler function](/serverless/workers/handler-functions) | You build the image yourself (Dockerfile), or skip the Dockerfile with the [Runpod SDK](/sdk/overview) and `@app.queue`. | +| Custom server (`docker_server`) | Serverless [load balancing endpoint](/serverless/load-balancing/overview) | Your HTTP server is exposed at its own paths. No `/predict` remapping. | +| vLLM custom server or engine | [vLLM worker](/serverless/vllm/overview) on a queue endpoint | Includes an OpenAI-compatible route. | +| Model APIs (`https://inference.baseten.co/v1`) | [Public Endpoints](/public-endpoints/overview) | Only for models listed in the [model reference](/public-endpoints/reference). Otherwise deploy the model on a vLLM endpoint. | +| Chains (`truss_chains`) | Several endpoints called from a CPU orchestrator endpoint | Runpod has no pipeline product. You write the calls between steps. | +| Training Jobs (`baseten train push`) | A [Pod](/pods/overview), or `@app.task` in the Runpod SDK | `@app.task` runs one Pod per call and terminates it when the function returns. | +| Multi-node (`resources.node_count`) | [Instant Clusters](/instant-clusters) | Stock varies by data center. | +| Deployment (one version of a model) | An endpoint release | Changing an endpoint's image or env creates a release and a [rolling release](/serverless/endpoints/rolling-releases). | +| Environments (production, staging, custom) and promotion | One endpoint per environment, or `rp flash deploy --env` for SDK apps | Each endpoint has its own ID and URL. | +| Replica | Worker | | +| Min replicas / max replicas | Active workers / max workers | Both platforms default to a minimum of 0 (scale to zero). | +| Scale-down delay | Idle timeout | 1 to 3,600 seconds on Runpod. | +| Concurrency target | Request count scaler value | See [What works differently](#what-works-differently). | +| `runtime.predict_concurrency` | `concurrency_modifier` in the handler, or `max_concurrency` on `@app.queue` | | +| `weights` and `model_cache` (Hugging Face sources) | [Cached models](/serverless/endpoints/model-caching) | One cached model per endpoint. | +| `weights` from S3, GCS, R2, or Azure, and BDN volumes | [Network volume](/storage/network-volumes), mounted at `/runpod-volume` on Serverless | A network volume is writable and pinned to one data center. | +| `data/` directory (bundled files) | Files copied into the image | | +| Baseten secrets | Runpod [secrets](/pods/templates/secrets), referenced from env vars | Baseten passes secrets to `__init__` or mounts files at `/secrets/NAME`. Runpod injects them as environment variables. | +| Regions and regional environments | Data centers (`--data-center-ids`, `dataCenterIds`) | | +| Instance types (`L4:4x16`, `H100`) | GPU pools (`AMPERE_24`, `ADA_80_PRO`) and CPU instance IDs (`cpu3g-4-16`) | See the [GPU mapping](#gpu-mapping). | +| Billing | Per second while a worker runs | Baseten's docs describe per-minute billing per running replica. See [Serverless pricing](/serverless/pricing). | +| Terraform provider `basetenlabs/baseten` | Terraform provider `runpod/runpod` (`~> 2.0`) | See [Migrate your Terraform](#migrate-your-terraform). | +| `baseten` CLI and `truss` CLI | `runpodctl` and `rp` | `runpodctl` manages resources. `rp` ships with `pip install runpod` and runs `rp flash dev` and `rp flash deploy`. | +| Management API (`https://api.baseten.co/v1`) | [REST API v2](/api-reference-v2/overview) (`https://api.runpod.io/v2`) | | +| Inference API (`https://model-MODEL_ID.api.baseten.co`) | `https://api.runpod.ai/v2/ENDPOINT_ID` (queue) or `https://ENDPOINT_ID.api.runpod.ai` (load balancing) | | +| `Authorization: Bearer` or legacy `Api-Key` with `BASETEN_API_KEY` | `Authorization: Bearer` with `RUNPOD_API_KEY` | | +| Teams, roles, SSO | [Organizations](/organizations/orgs-overview) with roles and groups | | +| Logs and metrics (`baseten model deployment logs`, metrics export) | Console **Logs** and **Metrics** tabs, and the REST v2 worker log and metrics routes | See [Logs](/serverless/development/logs). | + +## What's different + +### What works differently + +- **You own the image build.** `baseten model push` uploads a Truss and Baseten builds the image. On Runpod you build and push a `linux/amd64` image, deploy from a [GitHub repository](/serverless/workers/github-integration) that contains a Dockerfile, or use the Runpod SDK, which packages your code without a container build. +- **The request body is wrapped.** A Truss `predict` receives the JSON body as is. A Runpod queue endpoint expects `{"input": {...}}` and your handler reads `job["input"]`. +- **The response is wrapped.** `/predict` returns your model's output directly. `/runsync` returns a job object, and your output is under the `output` key. +- **Load at import time instead of in `load()`.** Code at the top of `handler.py` runs once when the worker starts. That's where the work from `__init__` and `load()` goes. +- **Secrets are environment variables.** Replace `kwargs["secrets"]["hf_access_token"]` and reads of `/secrets/hf_access_token` with `os.environ`. +- **Sync requests have a shorter wait window.** Baseten's docs give a default 1,200 second predict timeout for sync requests. Runpod's `/runsync` waits 90 seconds by default and up to 5 minutes with `?wait=`. For anything longer, including requests that might land on a cold start, use `/run` with `/status` or a webhook. +- **Async results are stored.** Baseten delivers async output only to your webhook and doesn't store it. Runpod keeps `/run` results for 30 minutes and returns them from `/status/JOB_ID`, with or without a webhook. +- **Streaming on queue endpoints is polled.** A Truss generator streams over the open HTTP response. A Runpod generator handler publishes chunks that clients read from `/stream/JOB_ID`. For a streamed HTTP response (for example server-sent events), use a load balancing endpoint or the vLLM worker's OpenAI-compatible route. +- **Autoscaling has fewer knobs.** Runpod scales a queue endpoint by queue delay (add workers when requests wait longer than a threshold, 4 seconds by default) or by request count (`ceil((queued + in progress) / scaler value)`). Set the request count scaler value to your Baseten concurrency target. To keep headroom like a target utilization below 100%, use a lower scaler value. Runpod has no autoscaling window, max scale-down rate, or schedule settings. +- **Idle timeout and request count scaling don't combine on queue endpoints.** REST API v2 rejects an idle timeout on a queue endpoint that scales by request count. Pick queue delay scaling when you need a long idle timeout. +- **Promotion is an image update.** Instead of promoting a deployment into an environment, you update the endpoint's image tag. Runpod replaces workers progressively: workers without a job are replaced right away, and busy workers finish their job first. +- **Cancel works on running jobs.** Baseten cancels only `QUEUED` async requests. Runpod's `/cancel/JOB_ID` stops queued and in-progress jobs. +- **Health checks.** A `docker_server` declares readiness and liveness endpoints. A load balancing worker has one health path (`/ping` by default, configurable with `HEALTH_CHECK_PATH`). Return `200` when ready and `204` while initializing. +- **Ports.** Baseten reserves port 8080 and sets `PORT=8080`. On a load balancing endpoint, `PORT` and `PORT_HEALTH` both default to `80`, and you set both if your server listens elsewhere. +- **Load balancing timeouts.** A load balancing endpoint allows 5.5 minutes of processing per request and a 30 MB payload. Longer requests belong on a queue endpoint. +- **Payload limits on queue endpoints.** 10 MB for `/run` and 20 MB for `/runsync`. Send large inputs as URLs. +- **Idle endpoints scale themselves down.** After 3 days without requests, Runpod reduces an endpoint's max workers to 2, and after 7 days to 0. Raise max workers again before you send traffic to an endpoint that sat unused. + +### What has no direct equivalent + +| On Baseten | On Runpod | Workaround | +| --- | --- | --- | +| Engine builder (`trt_llm`), BEI, and BIS-LLM managed engines | No managed engine builder | Use the [vLLM worker](/serverless/vllm/overview), or run your own TensorRT-LLM or embeddings server image on a load balancing endpoint. Benchmark before you cut over, because throughput per GPU can differ. | +| Chains (typed Chainlet-to-Chainlet calls, one deploy) | No pipeline product | Deploy each step as an endpoint and call them from a CPU orchestrator endpoint. See [Chains](#chains). | +| Loops (managed LoRA fine-tuning API) | No managed fine-tuning API | Run your training script on a Pod or with `@app.task`. | +| Environments with a stable URL and promotion | No environment object on endpoints | One endpoint per environment. Keep the endpoint ID in your client config per environment. | +| Rolling deployments you can pause, resume, or cancel | Rolling releases run automatically | For staged traffic shifting, run two endpoints and split traffic in your client or gateway. | +| Development deployments with live patching (`baseten model push --watch`) | `rp flash dev` for SDK apps | For Dockerfile workers, test locally with `python handler.py --test_input`. See [Local testing](/serverless/development/local-testing). | +| Async request priority (`priority` 0 to 2) and `max_time_in_queue_seconds` | `policy.lowPriority` and `policy.ttl` | `lowPriority` stops a job from triggering scale-up. `ttl` covers queue time plus run time. | +| Webhook signatures (`X-BASETEN-SIGNATURE`) | Runpod's webhook docs describe no signature header | Put an unguessable token in the webhook URL and verify it in your receiver, then confirm the result with `/status/JOB_ID`. | +| gRPC and WebSocket transports on the model URL | Expose HTTP/TCP ports on the endpoint | Persistent connections go to the worker's public IP and port, not through the endpoint URL. See [Endpoint settings](/serverless/endpoints/endpoint-configurations). | +| Single-tenant, self-hosted, and hybrid hosting | Runpod runs in Runpod data centers | For compliance details see [Security and compliance](/references/security-and-compliance). | +| Prometheus metrics export and OpenTelemetry tracing | Console metrics and REST v2 metrics routes | Export from your own code if you need traces. | + +Going the other way, these Runpod features don't have a counterpart in the Baseten docs this guide was checked against: Pods you can SSH into and keep running as long-lived GPU machines, writable shared [network volumes](/storage/network-volumes) (Baseten's BDN volumes are immutable and mounted read-only), and job results you can fetch from `/status` after the fact. + +## Requirements + +- A [Runpod account](/accounts-billing/manage-accounts) with credits and an [API key](/get-started/credentials). +- Access to your Baseten workspace, with the `baseten` CLI authenticated (`baseten auth login` or `BASETEN_API_KEY`). +- [Docker](https://docs.docker.com/get-started/get-docker/) and a container registry account (for example Docker Hub), unless you migrate with the Runpod SDK only. +- Python 3.10 to 3.14 for the Runpod SDK. +- The source of each Truss (`config.yaml`, `model/`, `data/`, `packages/`). If you don't have it locally, Step 1 shows how to download it. + +## Step 1: Take inventory of what you run on Baseten + +List every model, then every deployment and environment for each model. The `baseten` CLI prints JSON with `--output json`, which is easier to diff later. + +```bash +# Models, deployments, and environments +baseten model list --output json +baseten model deployment list --model-id MODEL_ID --output json +baseten model environment list --model-id MODEL_ID --output json + +# The config.yaml a deployment was built from, and its autoscaling settings +baseten model deployment config --model-id MODEL_ID --deployment-id DEPLOYMENT_ID +baseten model deployment describe --model-id MODEL_ID --deployment-id DEPLOYMENT_ID --jq '.autoscaling_settings' + +# The full Truss source of a deployment +baseten model deployment download --model-id MODEL_ID --deployment-id DEPLOYMENT_ID --out-dir ./exported/MODEL_ID +``` + +For each deployment, record the instance type, region, min and max replicas, scale-down delay, concurrency target, and which environment points at it. If the download fails (a push can disable archive download), fall back to your repository. + +Then list the resources that aren't models: + +```bash +baseten api management chains # Chains (GET /v1/chains) +baseten train job list # Training jobs +baseten org secret list # Secret names (values are never returned) +baseten org regions # Regions your workspace can deploy to +baseten api management instance_types # Instance types available to you +baseten volume ls bdn:NAMESPACE # BDN volumes in a namespace +baseten org billing usage --output json # Usage per resource, to find what is actually in use +``` + +Secret values can't be exported from Baseten. Collect them from wherever you keep the originals before you start. + +Next, find every Truss in your repositories and every place your code calls Baseten: + +```bash +# Truss directories +grep -rlE "^(model_name|docker_server|trt_llm|model_cache|weights):" --include="config.yaml" . + +# Client code, CI, and infrastructure that call Baseten +grep -rnE "api\.baseten\.co|inference\.baseten\.co|BASETEN_API_KEY|Api-Key|truss_chains|import truss|truss push|baseten model|async_predict|run_remote|basetenlabs/baseten" \ + --include="*.py" --include="*.ts" --include="*.js" --include="*.go" \ + --include="*.yaml" --include="*.yml" --include="*.tf" --include="*.sh" --include="*.env*" . +``` + +The second command catches inference URLs, Model APIs base URLs, the legacy `Api-Key` header, Chains code, CI jobs that push models, and Terraform. + +## Step 2: Set up Runpod + +Create an API key in the console (see [Manage API keys](/get-started/credentials)), then install the tools. + +```bash +# runpodctl: manages endpoints, templates, volumes, and registries +curl -fsSL https://cli.runpod.net | sudo bash +runpodctl doctor + +# Runpod SDK and the rp CLI +pip install runpod +rp login +``` + +`runpodctl doctor` walks you through first-time setup, including the API key. `rp login` stores credentials in `~/.runpod/config.toml`, and the SDK also reads `RUNPOD_API_KEY`. On macOS you can install the CLI with `brew install runpod/runpodctl/runpodctl`. + +If you manage infrastructure as code, add the [Terraform provider](/integrations/terraform/overview). To let an agent manage resources, connect the [MCP server](/get-started/mcp-servers). + +## Step 3: Choose where each workload lands + +| Baseten workload | Runpod target | Why | +| --- | --- | --- | +| Truss with a `Model` class, called through `/predict` or `/async_predict` | Serverless queue endpoint | Same request-in, result-out shape, with a queue, retries, and stored results. | +| Truss that returns a generator | Queue endpoint with a streaming handler, or a load balancing endpoint | Use load balancing if clients need a streamed HTTP response. | +| Truss that implements `chat_completions` or other `/v1` methods | Load balancing endpoint, or the vLLM worker | Clients keep using OpenAI SDKs with a new base URL. | +| `docker_server` running vLLM | vLLM worker on a queue endpoint | Prebuilt, with cached model support and an OpenAI-compatible route. | +| `docker_server` running SGLang, TGI, Triton, NIM, Ollama, or your own server | Load balancing endpoint | Runs your image and routes HTTP straight to it. | +| Model APIs calls for a model Runpod hosts | Public Endpoint | No deployment. | +| Model APIs calls for any other model | vLLM worker on a queue endpoint | You choose the GPU and the model. | +| Chain | One endpoint per Chainlet plus a CPU orchestrator endpoint | | +| Training job, single node | Pod, or `@app.task` | | +| Training job or deployment with `node_count` above 1 | Instant Cluster | | +| Engine builder (`trt_llm`) or BEI deployment | vLLM worker, or your own server image on a load balancing endpoint | Requires a rebuild on a different serving stack. Test output quality and throughput. | + +### GPU mapping + +Serverless endpoints select GPUs by pool. Pick the pool by the VRAM your model needs, not by name. To request several GPUs per worker (for example `H100:2`), set the GPU count. + +| Baseten accelerator | VRAM | Runpod GPU pool | GPUs in the pool | +| --- | --- | --- | --- | +| `T4` | 16 GB | `AMPERE_16` | RTX A4000, RTX A4500, RTX 4000 Ada, RTX 2000 Ada | +| `L4` | 24 GB | `AMPERE_24` | L4, RTX A5000, RTX 3090 | +| `A10G` | 24 GB | `AMPERE_24` or `ADA_24` | L4, RTX A5000, RTX 3090, or RTX 4090 | +| `H100_40GB` (`H100MIG`) | 40 GB | `AMPERE_48` or `ADA_48_PRO` | A40, RTX A6000, or L40, L40S, RTX 6000 Ada | +| `L40S` | 48 GB | `ADA_48_PRO` | L40, L40S, RTX 6000 Ada | +| `A100` | 80 GB | `AMPERE_80` | A100 80GB | +| `H100` | 80 GB | `ADA_80_PRO` | H100 | +| `RTX_PRO_6000` | 96 GB | `BLACKWELL_96` | RTX PRO 6000 Blackwell | +| `H200` | 141 GB | `HOPPER_141` | H200 | +| `B200` | 180 GB | `BLACKWELL_180` | B200 | + +A pool can hold a different GPU model than the one you ran on Baseten, so rerun your latency benchmarks. GPU stock varies by data center and over time. Check what's available before you commit to a pool with `runpodctl gpu list`, and list more than one pool where your model fits both. For the full list of GPU IDs and pools, see [GPU types](/references/gpu-types). + +CPU-only instances map to CPU instance IDs in the form `cpu{generation}{type}-{vCPU}-{memory GB}`: + +| Baseten instance | Runpod CPU instance | +| --- | --- | +| `1x2` | `cpu3c-1-2` | +| `1x4` | `cpu3g-1-4` | +| `2x8` | `cpu3g-2-8` | +| `4x16` | `cpu3g-4-16` | +| `8x32` | `cpu3g-8-32` | + +## Step 4: Move your images and code + +Runpod runs `linux/amd64` images. If you build on an Apple Silicon machine, pass `--platform linux/amd64`. Use versioned tags instead of `:latest`, because workers can keep serving a cached copy of a mutable tag. + +### Translate a Truss model to a handler + +This is a complete Truss for a text generation model. `config.yaml` declares the environment and `model/model.py` holds the `Model` class. + + +```yaml config.yaml +model_name: text-generator +python_version: py311 +requirements: + - torch==2.8.0 + - transformers==4.56.1 + - accelerate==1.10.1 +resources: + accelerator: L4 + cpu: "4" + memory: 16Gi + use_gpu: true +secrets: + hf_access_token: null +environment_variables: + MODEL_ID: HuggingFaceTB/SmolLM2-1.7B-Instruct +runtime: + predict_concurrency: 1 +``` + +```python model/model.py +import os + +import torch +from transformers import pipeline + + +class Model: + def __init__(self, **kwargs): + self._secrets = kwargs["secrets"] + self._pipe = None + + def load(self): + self._pipe = pipeline( + "text-generation", + model=os.environ["MODEL_ID"], + token=self._secrets["hf_access_token"], + torch_dtype=torch.bfloat16, + device_map="auto", + ) + + def predict(self, model_input): + prompt = model_input["prompt"] + max_new_tokens = model_input.get("max_new_tokens", 256) + result = self._pipe(prompt, max_new_tokens=max_new_tokens, return_full_text=False) + return {"output": result[0]["generated_text"]} +``` + + +On Runpod, the same model is three files. The handler loads the pipeline at module scope, which replaces `__init__` and `load()`, and `handler(job)` replaces `predict(model_input)`. + + +```python handler.py +import os + +import runpod +import torch +from transformers import pipeline + +# Runs once when the worker starts. This replaces __init__() and load(). +pipe = pipeline( + "text-generation", + model=os.environ["MODEL_ID"], + token=os.environ.get("HF_TOKEN"), + torch_dtype=torch.bfloat16, + device_map="auto", +) + + +def handler(job): + # job["input"] is the dict that predict() received as model_input. + model_input = job["input"] + prompt = model_input.get("prompt") + if not prompt: + return {"error": "Missing required input: prompt"} + + max_new_tokens = model_input.get("max_new_tokens", 256) + result = pipe(prompt, max_new_tokens=max_new_tokens, return_full_text=False) + return {"output": result[0]["generated_text"]} + + +runpod.serverless.start({"handler": handler}) +``` + +```dockerfile Dockerfile +FROM runpod/pytorch:1.0.2-cu1281-torch280-ubuntu2404 + +WORKDIR / + +COPY requirements.txt . +RUN pip install --no-cache-dir -r requirements.txt + +COPY handler.py . + +CMD ["python3", "-u", "/handler.py"] +``` + +```txt requirements.txt +runpod>=2.0,<3 +transformers==4.56.1 +accelerate==1.10.1 +``` + + +What matters in the translation: + +- The base image already contains PyTorch 2.8 and CUDA 12.8, so `torch` isn't in `requirements.txt`. If your Truss pins a different `torch`, choose a base image that matches it, or start from an `nvidia/cuda` image and install it yourself. See [Create a Dockerfile](/serverless/workers/create-dockerfile). +- The secret arrives as the `HF_TOKEN` environment variable. You wire it to a Runpod secret when you create the endpoint in Step 6. +- `preprocess` and `postprocess` don't exist as separate hooks. Call them in order inside the handler: `postprocess(predict(preprocess(job["input"])))`. +- An exception raised in the handler marks the job `FAILED`. To fail with your own message, return `{"error": "..."}`. +- Test before you build: `python handler.py --test_input '{"input": {"prompt": "Hello"}}'`. + +### Field-by-field map of config.yaml + +| `config.yaml` field | Dockerfile worker | Runpod SDK (`@app.queue`, `@app.api`) | +| --- | --- | --- | +| `model_name` | Endpoint and template `--name` | `runpod.App("NAME")` and the function name | +| `python_version` | Python version of the base image in `FROM` | `rp flash deploy --python-version 3.11` | +| `requirements`, `requirements_file` | `requirements.txt` and `RUN pip install -r requirements.txt` | `dependencies=[...]` and the project's `requirements.txt` | +| `system_packages` | `RUN apt-get update && apt-get install -y PACKAGE` | `system_dependencies=[...]` | +| `build_commands` | `RUN` lines | Build a custom image and pass `image=` | +| `base_image.image` | `FROM` | `image="..."` | +| `base_image.docker_auth` | `runpodctl registry create`, then `--registry-auth-id` on the template | `rp registry add`, then `registry_auth="..."` | +| `docker_server.start_command` | `CMD` in the Dockerfile, `--docker-start-cmd` on the template, or `cmd` in REST v2 | Not applicable. Use a Dockerfile worker. | +| `docker_server.server_port` | `PORT` and `PORT_HEALTH` env vars, and the endpoint's `ports` | Not applicable | +| `docker_server.predict_endpoint` | Call the path directly on `https://ENDPOINT_ID.api.runpod.ai` | `@runpod.post("/path")` on an `@app.api` class | +| `docker_server.readiness_endpoint`, `liveness_endpoint` | `HEALTH_CHECK_PATH` env var (default `/ping`) | A `@runpod.get` health route | +| `resources.accelerator` | `--gpu-id POOL`, `gpu.pools` in REST v2 | `gpu="..."` | +| `resources.accelerator: L4:4` (count) | `--gpu-count 4`, `gpu.count` | `gpu_count=4` | +| `resources.cpu`, `resources.memory`, `instance_type` (CPU-only) | `--compute-type CPU --instance-id cpu3g-4-16` | `cpu="cpu3g-4-16"` | +| `resources.node_count`, `resources.fabric` | Instant Clusters | Not applicable | +| `secrets` | Create a Runpod secret, then set an env var to `{{ RUNPOD_SECRET_name }}` | `env={"HF_TOKEN": runpod.Secret("name")}` | +| `environment_variables` | Template `--env`, or `ENV` in the Dockerfile | `env={...}` | +| `runtime.predict_concurrency` | `concurrency_modifier` in `runpod.serverless.start` | `max_concurrency=N` | +| `runtime.health_checks.startup_threshold_seconds` | `RUNPOD_INIT_TIMEOUT` env var (default 420 seconds) | Same env var through `env` | +| `weights` or `model_cache` with a Hugging Face source | `--model-reference` on `runpodctl serverless create`, or the **Model** field in the console | `model=runpod.Model("org/name")` | +| `weights` from S3, GCS, R2, or Azure | Network volume, or bake the files into the image | `mounts={"/runpod-volume": runpod.NetworkVolume("name")}` | +| `external_data` | `RUN wget` in the Dockerfile | Download in code, or use a volume | +| `data/` directory (`data_dir`) | `COPY data/ /data/` | Project files are packaged with the app | +| `packages/`, `external_package_dirs` | `COPY` the directories and set `PYTHONPATH` | Project files are packaged with the app | +| `model_metadata.example_model_input` | `test_input.json` next to the handler | The `@runpod.local_entrypoint` function | +| `trt_llm` | No equivalent. See [What has no direct equivalent](#what-has-no-direct-equivalent). | No equivalent | + +GPU endpoints have no separate vCPU or memory field. You choose a GPU pool and the container disk size. If your Truss asks for a lot of system RAM next to a small GPU, check the pool's hosts with `runpodctl gpu list` before you settle on it. + +### Streaming + +A Truss `predict` that returns a generator becomes a handler that yields. + + +```python model/model.py (Baseten) +from threading import Thread + +import torch +from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer + +CHECKPOINT = "HuggingFaceTB/SmolLM2-1.7B-Instruct" + + +class Model: + def __init__(self, **kwargs): + self.tokenizer = None + self.model = None + + def load(self): + self.tokenizer = AutoTokenizer.from_pretrained(CHECKPOINT) + self.model = AutoModelForCausalLM.from_pretrained( + CHECKPOINT, torch_dtype=torch.bfloat16, device_map="auto" + ) + + def predict(self, model_input): + inputs = self.tokenizer(model_input["prompt"], return_tensors="pt").to("cuda") + streamer = TextIteratorStreamer(self.tokenizer, skip_prompt=True) + thread = Thread( + target=self.model.generate, + kwargs={**inputs, "max_new_tokens": 256, "streamer": streamer}, + ) + thread.start() + + def stream_output(): + for text in streamer: + yield text + thread.join() + + return stream_output() +``` + +```python handler.py (Runpod) +from threading import Thread + +import runpod +import torch +from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer + +CHECKPOINT = "HuggingFaceTB/SmolLM2-1.7B-Instruct" + +tokenizer = AutoTokenizer.from_pretrained(CHECKPOINT) +model = AutoModelForCausalLM.from_pretrained( + CHECKPOINT, torch_dtype=torch.bfloat16, device_map="auto" +) + + +def handler(job): + inputs = tokenizer(job["input"]["prompt"], return_tensors="pt").to("cuda") + streamer = TextIteratorStreamer(tokenizer, skip_prompt=True) + thread = Thread( + target=model.generate, + kwargs={**inputs, "max_new_tokens": 256, "streamer": streamer}, + ) + thread.start() + + for text in streamer: + yield text + thread.join() + + +runpod.serverless.start({ + "handler": handler, + "return_aggregate_stream": True, +}) +``` + + +Clients read the chunks from `/stream/JOB_ID`. With `return_aggregate_stream` set, the full output is also available from `/run` and `/runsync`. See [Streaming handlers](/serverless/workers/handler-functions#streaming-handlers). + +### Concurrency + +`runtime.predict_concurrency` with an `async def predict` becomes an async handler plus a `concurrency_modifier`. + + +```python model/model.py (Baseten) +# config.yaml sets: +# runtime: +# predict_concurrency: 8 +import asyncio + + +class Model: + def __init__(self, **kwargs): + pass + + def load(self): + pass + + async def predict(self, model_input): + await asyncio.sleep(model_input.get("delay", 1)) + return {"processed": model_input} +``` + +```python handler.py (Runpod) +import asyncio + +import runpod + + +async def handler(job): + job_input = job["input"] + await asyncio.sleep(job_input.get("delay", 1)) + return {"processed": job_input} + + +def concurrency_modifier(current_concurrency): + # The number of jobs this worker takes at once. + return 8 + + +runpod.serverless.start({ + "handler": handler, + "concurrency_modifier": concurrency_modifier, +}) +``` + + +The function can return a different number over time to adapt to load. See [Build a concurrent handler](/serverless/workers/concurrent-handler). + +### Skip the Dockerfile with the Runpod SDK + +The Runpod SDK is the closest match to the Truss workflow: you declare hardware and dependencies in Python, and `rp flash deploy` packages and deploys the code. This file replaces `config.yaml`, `model/model.py`, the Dockerfile, and the endpoint settings from the example above. + +```python main.py +import runpod + +app = runpod.App("text-generator") + +MODEL_ID = "HuggingFaceTB/SmolLM2-1.7B-Instruct" +weights = runpod.Model(MODEL_ID) + +_pipe = None + + +@app.queue( + gpu="AMPERE_24", + workers=(0, 3), + idle_timeout=300, + dependencies=["transformers==4.56.1", "accelerate==1.10.1"], + model=weights, + env={"HF_TOKEN": runpod.Secret("hf_access_token")}, + max_concurrency=1, + execution_timeout_ms=600000, +) +def generate(prompt: str, max_new_tokens: int = 256): + global _pipe + if _pipe is None: + import torch + from transformers import pipeline + + _pipe = pipeline( + "text-generation", + model=MODEL_ID, + torch_dtype=torch.bfloat16, + device_map="auto", + ) + + result = _pipe(prompt, max_new_tokens=max_new_tokens, return_full_text=False) + return {"output": result[0]["generated_text"]} + + +@runpod.local_entrypoint +def main(): + print(generate.remote("Write a haiku about GPUs.")) +``` + +Run it against temporary endpoints with `rp flash dev main.py`, then deploy with `rp flash deploy`. A few things to know: + +- `model=weights` stages the Hugging Face weights on the host before the worker starts, and `transformers` finds them through the standard cache. This replaces `weights` or `model_cache` in `config.yaml`. +- Imports of worker-only packages go inside the function, so your machine doesn't need them. `torch` is preinstalled on the built-in GPU images. +- The SDK docs describe a once-per-worker setup hook, `@runpod.init`, for `@app.api` classes. For an `@app.queue` function, this example keeps the pipeline in a module-level variable. Confirm in `rp flash dev` that a second call skips the load. If it doesn't, move the model to an `@app.api` class with `@runpod.init`, which mirrors `load()` and `predict()` directly. +- Environments map to `rp flash deploy --env staging` and `rp flash deploy --env production`. Each environment gets its own endpoints. See [Apps and environments](/sdk/apps/apps-and-environments). + +For every decorator argument, see [Resource parameters](/sdk/configuration/parameters). + +### Custom servers + +A `docker_server` Truss already is a container with an HTTP server, so the image moves as is. What changes is the wiring around it. + +```yaml config.yaml +base_image: + image: vllm/vllm-openai:v0.29.0 +docker_server: + start_command: sh -c "HF_TOKEN=$(cat /secrets/hf_access_token) vllm serve Qwen/Qwen2.5-3B-Instruct --port 8000" + server_port: 8000 + predict_endpoint: /v1/chat/completions + readiness_endpoint: /health + liveness_endpoint: /health +resources: + accelerator: L4 +secrets: + hf_access_token: null +``` + +You have two options on Runpod: + +- **vLLM worker (recommended for vLLM).** Deploy Runpod's vLLM worker from the Hub and set `MODEL_NAME`. You get cached models, the queue, and an OpenAI-compatible route at `https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1`. Any `vllm serve` flag can be set as an environment variable in upper snake case, for example `MAX_MODEL_LEN=8192`. See [vLLM environment variables](/serverless/vllm/environment-variables). +- **Load balancing endpoint (any server).** Run the same image and start command. Set `PORT` and `PORT_HEALTH` to your server port, set `HEALTH_CHECK_PATH` to your readiness route, and expose the port. Requests to `https://ENDPOINT_ID.api.runpod.ai/v1/chat/completions` reach `/v1/chat/completions` on your server. There's no `/predict` alias and no `/sync/` prefix. + +Step 6 has the deploy commands for both. If your image depends on Baseten's runtime contract, check these before you move it: + +| Baseten runtime detail | On Runpod | +| --- | --- | +| Runs as user `app` (UID 60000) unless `run_as_user_id` is set | Runs as the user your image defines | +| Secrets as files in `/secrets/NAME` | Environment variables | +| `PORT=8080` reserved for Baseten's proxy | `PORT` is yours to set on load balancing endpoints (default `80`) | +| `/app/data` holds the Truss `data/` directory | Copy the files into your own image | +| `/etc/b10_dynamic_config/environment` holds the environment name | Set your own variable, for example `APP_ENV=production`, per endpoint | + +### Chains + +A Chain deploys several Chainlets together and lets one call another through `run_remote`. On Runpod, each Chainlet becomes its own endpoint with its own hardware and scaling, and the entrypoint becomes a small CPU endpoint that calls the others. + + +```python hello.py (Baseten) +import random + +import truss_chains as chains + + +class RandInt(chains.ChainletBase): + async def run_remote(self, max_value: int) -> int: + return random.randint(1, max_value) + + +@chains.mark_entrypoint +class HelloWorld(chains.ChainletBase): + def __init__(self, rand_int=chains.depends(RandInt, retries=3)) -> None: + self._rand_int = rand_int + + async def run_remote(self, max_value: int) -> str: + num_repetitions = await self._rand_int.run_remote(max_value) + return "Hello World! " * num_repetitions +``` + +```python rand_int/handler.py (Runpod) +import random + +import runpod + + +def handler(job): + return random.randint(1, job["input"]["max_value"]) + + +runpod.serverless.start({"handler": handler}) +``` + +```python hello_world/handler.py (Runpod) +import os + +import requests +import runpod + +RAND_INT_URL = f"https://api.runpod.ai/v2/{os.environ['RAND_INT_ENDPOINT_ID']}/runsync" +HEADERS = {"Authorization": f"Bearer {os.environ['ORCHESTRATOR_API_KEY']}"} + + +def call_rand_int(max_value, retries=3): + for attempt in range(retries): + response = requests.post( + RAND_INT_URL, + headers=HEADERS, + json={"input": {"max_value": max_value}}, + timeout=120, + ) + if response.ok and response.json().get("status") == "COMPLETED": + return response.json()["output"] + raise RuntimeError("rand_int did not complete") + + +def handler(job): + num_repetitions = call_rand_int(job["input"]["max_value"]) + return "Hello World! " * num_repetitions + + +runpod.serverless.start({"handler": handler}) +``` + + +Deploy `rand_int` and `hello_world` as two endpoints (the orchestrator on a CPU instance), store a Runpod API key as a secret, and pass it to the orchestrator as `ORCHESTRATOR_API_KEY` along with `RAND_INT_ENDPOINT_ID`. Secret names can't start with `runpod`, so don't name the secret `RUNPOD_API_KEY`. + +What you give up compared with Chains: typed stubs between steps, a single `truss chains push` for the whole pipeline, and Chainlet-level retries declared in `chains.depends`. What stays the same: each step scales separately on its own hardware. Keep the orchestrator's wait under the `/runsync` window, or switch the inner calls to `/run` and poll `/status`. + +## Step 5: Move your data and models + +**Hugging Face weights.** If your Truss uses `weights` with an `hf://` source, or `model_cache` with a Hugging Face `repo_id`, use a [cached model](/serverless/endpoints/model-caching). Runpod schedules workers on hosts that already hold the model and doesn't bill worker time for the download. Cached models are mounted at `/runpod-volume/huggingface-cache/hub/` in the Hugging Face cache layout, so the path changes from your Truss `mount_location` or `/app/model_cache/VOLUME_FOLDER`. [Use Hugging Face models](/serverless/development/huggingface-models#use-cached-models) has a helper that resolves the snapshot path. Each endpoint takes one cached model. + +**Weights in S3, GCS, R2, or Azure, and BDN volumes.** Put them on a [network volume](/storage/network-volumes), which mounts at `/runpod-volume` on Serverless workers. A network volume lives in one data center, and an endpoint that attaches it runs only there, so choose a data center that stocks the GPU pool you need. + +```bash +# Create the volume in the data center where your endpoint will run +runpodctl network-volume create --name model-weights --size 100 --data-center-id US-KS-2 + +# Copy weights in over the S3-compatible API (bucket name = volume ID) +aws s3 sync ./weights s3://NETWORK_VOLUME_ID/weights \ + --region US-KS-2 \ + --endpoint-url https://s3api-us-ks-2.runpod.io/ +``` + +The S3-compatible API uses its own access key and secret, which you create in the console, and is available in the data centers listed in [S3-compatible API](/storage/s3-api). To pull a BDN volume down first, run `baseten volume pull bdn:NAMESPACE/VOLUME ./weights`. For large transfers, start a Pod with the volume attached and copy from your bucket inside the data center instead of through your laptop. + +**Small files.** Anything in the Truss `data/` directory, or fetched by `external_data`, can be copied or downloaded into the image in the Dockerfile. This keeps the worker self-contained but increases image size. + +**Weights without a data center pin.** A [global volume](/storage/globalvolume/overview) (beta) isn't tied to a data center and attaches to GPU endpoints at `/runpod-volume`. It bills for storage and requests and has no S3-compatible API, so load it from a Pod. + +**Training checkpoints.** Baseten stores checkpoints from Training Jobs. Download the ones you need before you shut down, and write new ones to a network volume on Runpod. + +## Step 6: Deploy on Runpod + +### Queue endpoint from your image + +First build and push the image from Step 4, and create the secret the handler reads. + +```bash +docker build --platform linux/amd64 -t DOCKER_USERNAME/text-generator:v1.0.0 . +docker push DOCKER_USERNAME/text-generator:v1.0.0 + +# Prompts for the value +rp secret add hf_access_token +``` + +On Baseten, one command deploys the Truss and sets autoscaling: + +```bash +baseten model push --wait \ + --min-replica 0 --max-replica 3 \ + --scale-down-delay 300 --concurrency-target 1 +``` + +On Runpod, create a Serverless template for the image, then an endpoint that uses it. + + + + ```bash + runpodctl template create \ + --name text-generator \ + --image docker.io/DOCKER_USERNAME/text-generator:v1.0.0 \ + --serverless \ + --container-disk-in-gb 20 \ + --env '{"MODEL_ID":"HuggingFaceTB/SmolLM2-1.7B-Instruct","HF_TOKEN":"{{ RUNPOD_SECRET_hf_access_token }}"}' + + runpodctl serverless create \ + --name text-generator \ + --template-id TEMPLATE_ID \ + --gpu-id AMPERE_24 \ + --workers-min 0 --workers-max 3 \ + --idle-timeout 300 \ + --scale-by delay --scale-threshold 4 \ + --execution-timeout 600 \ + --min-cuda-version 12.8 \ + --model-reference https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct:main + ``` + + `TEMPLATE_ID` comes from the output of the first command. Each Serverless template binds to one endpoint, so create one template per endpoint. For a private image, run `runpodctl registry create` and add `--registry-auth-id` to the template. + + + ```bash + curl -X POST https://api.runpod.io/v2/serverless \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "text-generator", + "image": "docker.io/DOCKER_USERNAME/text-generator:v1.0.0", + "type": "QUEUE", + "gpu": {"pools": ["AMPERE_24"], "count": 1, "minCudaVersion": "12.8"}, + "workers": {"min": 0, "max": 3, "idleTimeout": 300}, + "scaling": {"type": "QUEUE_DELAY", "queueDelay": 4}, + "flashboot": "FLASHBOOT", + "timeout": 600000, + "disk": 20, + "env": { + "MODEL_ID": "HuggingFaceTB/SmolLM2-1.7B-Instruct", + "HF_TOKEN": "{{ RUNPOD_SECRET_hf_access_token }}" + } + }' + ``` + + `name`, `type`, and `scaling` are required. REST v2 defaults differ from the console: FlashBoot is `OFF` and the execution timeout is 300,000 ms unless you set them. See [Create a Serverless endpoint](/api-reference-v2/serverless/create-a-serverless-endpoint). + + + Go to **Serverless**, click **New Endpoint**, choose **Import from Docker Registry**, enter the image, and set the GPU, workers, environment variables, and **Model** field. See [Deploy a worker](/serverless/workers/deploy#deploy-an-endpoint). + + + +The autoscaling flags map as follows: + +| Baseten setting | Runpod setting | Notes | +| --- | --- | --- | +| `--min-replica` | `--workers-min` (active workers) | Active workers are billed continuously. | +| `--max-replica` | `--workers-max` | Baseten defaults to 1, Runpod to 3. | +| `--scale-down-delay` (default 900 s) | `--idle-timeout` | A worker is billed while it waits. The console default is 5 seconds. | +| `--concurrency-target` | `--scale-by requests --scale-threshold N` | Or keep queue delay scaling and set per-worker concurrency in the handler. | +| `--target-utilization-percentage` | No setting | Lower the request count threshold to leave headroom. | +| `--autoscaling-window`, `--max-scale-down-rate` | No setting | | +| `--instance-type` | `--gpu-id`, `--gpu-count`, or `--compute-type CPU --instance-id` | | +| `--region` | `--data-center-ids` | | +| `--environment` | A separate endpoint per environment | | + +### Queue endpoint from the Runpod SDK + +If you wrote `main.py` in Step 4, there's no template or image to create: + +```bash +rp secret add hf_access_token +rp flash dev main.py --once # Run the entrypoint once against temporary endpoints +rp flash deploy --env production +``` + +`rp flash deploy` prints a link to each endpoint. To remove them later, run `rp flash undeploy --app text-generator --env production`. + +### vLLM worker + +Find the vLLM worker in the Hub and create an endpoint from it: + +```bash +runpodctl hub search vllm + +runpodctl serverless create \ + --hub-id HUB_ID \ + --name qwen-chat \ + --gpu-id AMPERE_24 \ + --env MODEL_NAME=Qwen/Qwen2.5-3B-Instruct \ + --env MAX_MODEL_LEN=8192 \ + --model-reference https://huggingface.co/Qwen/Qwen2.5-3B-Instruct:main +``` + +For a gated model, also set `HF_TOKEN` from a secret. The endpoint answers Runpod-native requests on `/run` and `/runsync`, and OpenAI SDK requests at `https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1`. See [Get started with vLLM](/serverless/vllm/get-started). + +### Load balancing endpoint + +`runpodctl serverless create` creates queue endpoints. Create a load balancing endpoint with REST API v2 (or the console, by choosing **Load Balancer** as the endpoint type). This is the `docker_server` example from Step 4: + +```bash +curl -X POST https://api.runpod.io/v2/serverless \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "qwen-vllm-lb", + "image": "vllm/vllm-openai:v0.29.0", + "cmd": ["Qwen/Qwen2.5-3B-Instruct", "--port", "8000"], + "type": "LOAD_BALANCER", + "ports": ["8000/http"], + "gpu": {"pools": ["AMPERE_24"], "count": 1}, + "workers": {"min": 0, "max": 3, "idleTimeout": 300}, + "scaling": {"type": "REQUEST_COUNT", "requestCount": 32}, + "disk": 40, + "env": { + "PORT": "8000", + "PORT_HEALTH": "8000", + "HEALTH_CHECK_PATH": "/health", + "HF_TOKEN": "{{ RUNPOD_SECRET_hf_access_token }}" + } + }' +``` + +`cmd` is the argument list passed to the image's entrypoint, so check what your image's `ENTRYPOINT` is and set `entrypoint` too if you need to replace it. Load balancing endpoints scale by request count only. Set `requestCount` to the concurrency target you used on Baseten. The response includes the endpoint ID, and your server is then reachable at `https://ENDPOINT_ID.api.runpod.ai`. + +### Pods for training + +A Baseten Training Job packages your code and runs it on a GPU. On Runpod, the SDK equivalent is `@app.task`, which starts a Pod for the call and terminates it when the function returns: + +```python train.py +import runpod + +app = runpod.App("finetune") +checkpoints = runpod.NetworkVolume("checkpoints", size=100) + + +@app.task(gpu="H100", gpu_count=1, mounts={"/checkpoints": checkpoints}) +def train(steps: int = 1000): + # Your training loop. Write checkpoints under checkpoints.path. + return {"steps": steps} + + +@runpod.local_entrypoint +def main(): + print(train.remote(steps=1000)) +``` + +Run it with `rp flash dev train.py --once`. For interactive work, create a Pod from the console or with `runpodctl pod create` and connect over SSH. See [Pods](/pods/overview). + +## Step 7: Update the code that calls Baseten + +### Sync requests + +The URL, the body wrapper, and the response wrapper change. Both platforms accept `Authorization: Bearer`. If your code still sends Baseten's legacy `Authorization: Api-Key ...` header, change the scheme to `Bearer`. + + +```python Baseten +import os + +import requests + +response = requests.post( + f"https://model-{os.environ['BASETEN_MODEL_ID']}.api.baseten.co/environments/production/predict", + headers={"Authorization": f"Bearer {os.environ['BASETEN_API_KEY']}"}, + json={"prompt": "Hello, world!"}, + timeout=60, +) +response.raise_for_status() +print(response.json()["output"]) +``` + +```python Runpod +import os + +import requests + +response = requests.post( + f"https://api.runpod.ai/v2/{os.environ['ENDPOINT_ID']}/runsync", + headers={"Authorization": f"Bearer {os.environ['RUNPOD_API_KEY']}"}, + json={"input": {"prompt": "Hello, world!"}}, + timeout=120, +) +response.raise_for_status() +job = response.json() +if job["status"] != "COMPLETED": + raise RuntimeError(f"Job {job['id']} is {job['status']}") +print(job["output"]["output"]) +``` + + +The Runpod response is a job object with `id`, `status`, `output`, `delayTime`, and `executionTime`. The first `output` key is Runpod's, and the second is the key the handler returned. An HTTP 200 doesn't mean the job succeeded, so check `status`. + +### Async requests and webhooks + + +```python Baseten +import os + +import requests + +response = requests.post( + f"https://model-{os.environ['BASETEN_MODEL_ID']}.api.baseten.co/environments/production/async_predict", + headers={"Authorization": f"Bearer {os.environ['BASETEN_API_KEY']}"}, + json={ + "model_input": {"prompt": "Hello, world!"}, + "webhook_endpoint": "https://example.com/webhook", + }, + timeout=30, +) +request_id = response.json()["request_id"] + +status = requests.get( + f"https://model-{os.environ['BASETEN_MODEL_ID']}.api.baseten.co/async_request/{request_id}", + headers={"Authorization": f"Bearer {os.environ['BASETEN_API_KEY']}"}, + timeout=30, +).json() +print(status["status"]) # Lifecycle only. The output goes to the webhook. +``` + +```python Runpod +import os +import time + +import requests + +BASE_URL = f"https://api.runpod.ai/v2/{os.environ['ENDPOINT_ID']}" +HEADERS = {"Authorization": f"Bearer {os.environ['RUNPOD_API_KEY']}"} + +response = requests.post( + f"{BASE_URL}/run", + headers=HEADERS, + json={ + "input": {"prompt": "Hello, world!"}, + "webhook": "https://example.com/webhook", + }, + timeout=30, +) +job_id = response.json()["id"] + +interval = 1 +while True: + job = requests.get(f"{BASE_URL}/status/{job_id}", headers=HEADERS, timeout=30).json() + if job["status"] in ("COMPLETED", "FAILED", "CANCELLED", "TIMED_OUT"): + break + time.sleep(interval) + interval = min(interval * 2, 30) + +print(job["status"], job.get("output")) +``` + + +Update your webhook receiver too. Baseten posts `{"request_id", "model_id", "deployment_id", "type", "time", "data", "errors"}` and signs it with `X-BASETEN-SIGNATURE`. Runpod posts the job object (the same shape `/status` returns) and expects a `200`, retrying up to 2 more times with a 10 second delay. Read `output` instead of `data`, `id` instead of `request_id`, and replace the signature check as described in [What has no direct equivalent](#what-has-no-direct-equivalent). The `webhook` field is optional: you can poll `/status` alone. Per-request options move as follows: `max_time_in_queue_seconds` becomes `policy.ttl` (milliseconds, covering queue plus run time), and `priority: 1` or `2` becomes `policy.lowPriority: true`. See [Send API requests](/serverless/endpoints/send-requests). + +### Streaming + +On a queue endpoint, submit with `/run`, then read chunks with the Runpod SDK's endpoint client: + +```python +import os + +import runpod + +runpod.api_key = os.environ["RUNPOD_API_KEY"] +endpoint = runpod.Endpoint(os.environ["ENDPOINT_ID"]) + +run_request = endpoint.run({"input": {"prompt": "Tell me about falcons"}}) +for chunk in run_request.stream(): + print(chunk, end="", flush=True) +``` + +This replaces a `requests.post(..., stream=True)` loop over a Baseten `/predict` response. If a browser or another service consumes the HTTP stream directly, serve it from a load balancing endpoint instead. + +### OpenAI-compatible clients + +For Model APIs, and for deployments you called through `/environments/production/sync/v1`, only the base URL, API key, and model name change. + + +```python Baseten Model APIs +import os + +from openai import OpenAI + +client = OpenAI( + base_url="https://inference.baseten.co/v1", + api_key=os.environ["BASETEN_API_KEY"], +) +response = client.chat.completions.create( + model="openai/gpt-oss-120b", + messages=[{"role": "user", "content": "Reply with exactly: Paris."}], +) +print(response.choices[0].message.content) +``` + +```python Runpod Public Endpoint +import os + +from openai import OpenAI + +client = OpenAI( + base_url="https://api.runpod.ai/v2/gpt-oss-120b/openai/v1", + api_key=os.environ["RUNPOD_API_KEY"], +) +response = client.chat.completions.create( + model="openai/gpt-oss-120b", + messages=[{"role": "user", "content": "Reply with exactly: Paris."}], +) +print(response.choices[0].message.content) +``` + +```python Runpod vLLM endpoint +import os + +from openai import OpenAI + +client = OpenAI( + base_url=f"https://api.runpod.ai/v2/{os.environ['ENDPOINT_ID']}/openai/v1", + api_key=os.environ["RUNPOD_API_KEY"], +) +response = client.chat.completions.create( + model="Qwen/Qwen2.5-3B-Instruct", + messages=[{"role": "user", "content": "Reply with exactly: Paris."}], +) +print(response.choices[0].message.content) +``` + + +Each Public Endpoint has its own base URL that contains the model's endpoint ID, unlike Baseten's single base URL with a model slug per request. The OpenAI-compatible text models are [GPT-OSS 120B](/public-endpoints/models/gpt-oss-120b) (`gpt-oss-120b`, model `openai/gpt-oss-120b`; the model page's own examples use `gpt-oss-120b`, so try that name if a request is rejected) and [Qwen3 32B AWQ](/public-endpoints/models/qwen3-32b) (`qwen3-32b-awq`, model `Qwen/Qwen3-32B-AWQ`). For any other Model APIs slug, deploy that model on a vLLM endpoint. The vLLM worker serves `/chat/completions`, `/completions`, and `/models`. If you use Baseten's Responses or Anthropic Messages protocols, move those calls to chat completions or serve the protocol from your own server on a load balancing endpoint. For a load balancing endpoint that runs an OpenAI-compatible server, the base URL is `https://ENDPOINT_ID.api.runpod.ai/v1`. + +### Control-plane automation + +Replace CI steps and scripts that call the `baseten` or `truss` CLIs or the Management API. The common ones: + +| Task | Baseten | Runpod | +| --- | --- | --- | +| Deploy a new version | `baseten model push`, `truss push --promote` | Build and push an image tag, then `PATCH /v2/serverless/ENDPOINT_ID` with the new `image`. For SDK apps, `rp flash deploy`. | +| Change scaling | `baseten model deployment update-autoscaling` | `runpodctl serverless update ENDPOINT_ID --workers-min N --workers-max N --idle-timeout N` | +| Read logs | `baseten model deployment logs --tail` | Console **Logs** tab, or `GET /v2/serverless/ENDPOINT_ID/workers/WORKER_ID/logs` | +| Set a secret | `baseten org secret set` | `rp secret add NAME`, or `POST /v2/account/secrets` | + +Updating the image creates a release, and workers move to it as a [rolling release](/serverless/endpoints/rolling-releases). To roll a new image, patch the endpoint itself: + +```bash +curl -X PATCH https://api.runpod.io/v2/serverless/ENDPOINT_ID \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"image": "docker.io/DOCKER_USERNAME/text-generator:v1.1.0"}' +``` + +The full maps are in the [Reference](#reference) section. + +## Step 8: Verify, cut over, and shut down + + + + Call the new endpoint from a machine outside Runpod and check the job status and output. + + ```bash + curl -X POST https://api.runpod.ai/v2/ENDPOINT_ID/runsync \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"input": {"prompt": "Hello, world!"}}' + + curl https://api.runpod.ai/v2/ENDPOINT_ID/health \ + -H "Authorization: Bearer $RUNPOD_API_KEY" + ``` + + The first request after a deploy includes a cold start. Send a second one to see warm latency. + + + Replay a fixed set of production inputs against both platforms with the same seed and sampling settings, and diff the outputs. In the Runpod job object, `delayTime` is queue plus cold start time and `executionTime` is handler time, both in milliseconds. Compare `executionTime` with your Baseten inference latency, and tune active workers and idle timeout against `delayTime`. + + + Keep the Baseten deployment active and send a small share of traffic to Runpod from your client or gateway. Raise max workers before each increase. Watch the endpoint's **Metrics** tab for delay time, cold starts, and failed requests. + + + Max workers caps how far an endpoint scales, and the execution timeout stops runaway jobs. Keep active workers at 0 unless you need warm capacity. Check spend with `runpodctl billing --help` or the REST v2 `/v2/billing/serverless` route. + + + When traffic has been at zero for long enough to trust it, deactivate first (reversible), then delete. + + ```bash + # Stop compute spend but keep the deployment + baseten model deployment deactivate --model-id MODEL_ID --deployment-id DEPLOYMENT_ID --yes + + # Stop any running training jobs + baseten train job list --status running + baseten train job stop --job-id JOB_ID --yes + + # Later: delete models, Chains, and BDN volumes + baseten model delete --model-id MODEL_ID --yes + curl -X DELETE https://api.baseten.co/v1/chains/CHAIN_ID \ + -H "Authorization: Bearer $BASETEN_API_KEY" + baseten volume rm --recursive --yes bdn:NAMESPACE/VOLUME + + # Confirm nothing is still accruing + baseten org billing usage + ``` + + Deleting a model or volume can't be undone. Deactivate instead of only scaling to zero, because a scaled-to-zero deployment stays live and any stray request starts a replica. Finally, delete the Baseten API keys your services used (`baseten org api-key list`) and remove `BASETEN_API_KEY` from your secret stores. + + + +## Migrate your Terraform + +Baseten publishes the `basetenlabs/baseten` provider with a `baseten_model` resource (which can push a Truss directory and manage environment autoscaling) and a `baseten_secret` resource. The Runpod provider, `runpod/runpod` at `~> 2.0`, is built on REST API v2. + +| Baseten | Runpod | Notes | +| --- | --- | --- | +| `baseten_model` with `push` | `runpod_serverless_endpoint` with `image` | The Runpod provider deploys an image you already built. Build and push in CI before `terraform apply`. | +| `baseten_model.environments.NAME.autoscaling` | `workers_min`, `workers_max`, `idle_timeout`, `scaling_type`, `scaling_value` | One `runpod_serverless_endpoint` per environment. | +| `baseten_model.environments.NAME.promotion.rolling_deploy` | No arguments | Rolling releases run on every image change. | +| `baseten_secret` | No secret resource | Create the secret in the console or with `POST /v2/account/secrets`, and reference it in `env` as `{{ RUNPOD_SECRET_name }}`. | +| Registry auth in `config.yaml` | `runpod_registry`, referenced by `registry` | | +| BDN volume | `runpod_network_volume` | | + + +```hcl Baseten +terraform { + required_providers { + baseten = { + source = "basetenlabs/baseten" + version = "~> 0.1" + } + } +} + +provider "baseten" {} + +resource "baseten_model" "text_generator" { + name = "text-generator" + + push = { + config_dir = "${path.module}/text-generator" + environment = "production" + wait = true + } + + environments = { + production = { + autoscaling = { + min_replica = 0 + max_replica = 3 + scale_down_delay = 300 + } + } + } +} +``` + +```hcl Runpod +terraform { + required_version = ">= 1.6" + required_providers { + runpod = { + source = "runpod/runpod" + version = "~> 2.0" + } + } +} + +provider "runpod" {} + +resource "runpod_serverless_endpoint" "text_generator" { + name = "text-generator" + image = "docker.io/DOCKER_USERNAME/text-generator:v1.0.0" + endpoint_type = "QUEUE" + + gpu_pools = ["AMPERE_24"] + gpu_count = 1 + min_cuda_version = "12.8" + disk = 20 + + workers_min = 0 + workers_max = 3 + idle_timeout = 300 + scaling_type = "QUEUE_DELAY" + scaling_value = 4 + flashboot = "FLASHBOOT" + timeout = 600000 + + env = { + MODEL_ID = "HuggingFaceTB/SmolLM2-1.7B-Instruct" + HF_TOKEN = "{{ RUNPOD_SECRET_hf_access_token }}" + } +} + +output "run_url" { + value = runpod_serverless_endpoint.text_generator.request_urls.run +} +``` + + +Both providers read the API key from the environment (`BASETEN_API_KEY` and `RUNPOD_API_KEY`). A new image tag in `image` rolls the endpoint. `timeout` is in milliseconds and `idle_timeout` in seconds. For every argument, see the [provider reference](https://registry.terraform.io/providers/runpod/runpod/latest/docs). + +## Reference + +### CLI command map + +| Baseten | Runpod | +| --- | --- | +| `baseten auth login`, `truss login` | `runpodctl doctor`, `rp login` | +| `truss init` | `rp flash init my-app` (SDK apps) | +| `baseten model push`, `truss push` | `docker build` and `docker push`, then `runpodctl template create --serverless` and `runpodctl serverless create`. For SDK apps, `rp flash deploy`. | +| `baseten model push --watch`, `truss watch` | `rp flash dev main.py` | +| `baseten model push --environment staging` | `rp flash deploy --env staging`, or a separate endpoint | +| `baseten model deployment promote` | `PATCH /v2/serverless/ENDPOINT_ID` with a new `image` | +| `baseten model list` | `runpodctl serverless list` | +| `baseten model describe`, `baseten model deployment describe` | `runpodctl serverless get ENDPOINT_ID --include-template` | +| `baseten model deployment update-autoscaling` | `runpodctl serverless update ENDPOINT_ID` | +| `baseten model predict` | `curl` to `/runsync` or `/run` | +| `baseten model deployment logs` | Console **Logs** tab, or the REST v2 worker logs route | +| `baseten model deployment deactivate` | `runpodctl serverless update ENDPOINT_ID --workers-max 0` on a current `runpodctl`, then confirm with `runpodctl serverless get` | +| `baseten model delete` | `runpodctl serverless delete ENDPOINT_ID` | +| `baseten org secret set`, `list` | `rp secret add`, `rp secret list` | +| `baseten org regions` | `runpodctl datacenter list` | +| `baseten api management instance_types` | `runpodctl gpu list` | +| `baseten volume push`, `pull` | `runpodctl network-volume create`, then `aws s3 sync` with the S3-compatible API | +| `truss chains push` | Deploy each step as its own endpoint | +| `baseten train push` | `rp flash dev train.py --once` with `@app.task`, or `runpodctl pod create` | +| `baseten org billing usage` | `runpodctl billing serverless`, `runpodctl billing pods`, `runpodctl billing network-volume` | + +### API map + +| Baseten | Runpod | +| --- | --- | +| `POST https://model-MODEL_ID.api.baseten.co/environments/production/predict` | `POST https://api.runpod.ai/v2/ENDPOINT_ID/runsync` | +| `POST .../environments/production/async_predict` | `POST https://api.runpod.ai/v2/ENDPOINT_ID/run` | +| `GET https://model-MODEL_ID.api.baseten.co/async_request/REQUEST_ID` | `GET https://api.runpod.ai/v2/ENDPOINT_ID/status/JOB_ID` | +| Cancel a queued async request | `POST https://api.runpod.ai/v2/ENDPOINT_ID/cancel/JOB_ID` | +| Streamed `/predict` response | `GET https://api.runpod.ai/v2/ENDPOINT_ID/stream/JOB_ID` | +| `.../environments/production/sync/PATH` (custom server routes) | `https://ENDPOINT_ID.api.runpod.ai/PATH` (load balancing) | +| `.../environments/production/sync/v1` (OpenAI base URL) | `https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1` (vLLM worker) | +| `https://inference.baseten.co/v1` (Model APIs) | `https://api.runpod.ai/v2/PUBLIC_ENDPOINT_ID/openai/v1` | +| `https://chain-CHAIN_ID.api.baseten.co/production/run_remote` | `/runsync` on your orchestrator endpoint | +| Wake endpoint | No equivalent call. A request starts a worker. | +| `GET https://api.baseten.co/v1/models` | `GET https://api.runpod.io/v2/serverless` | +| `PATCH /v1/models/MODEL_ID/deployments/DEPLOYMENT_ID/autoscaling_settings` | `PATCH /v2/serverless/ENDPOINT_ID` (`workers`, `scaling`) | +| `POST /v1/models/MODEL_ID/deployments/DEPLOYMENT_ID/promote` | `PATCH /v2/serverless/ENDPOINT_ID` (`image`) | +| `POST /v1/models/MODEL_ID/deployments/DEPLOYMENT_ID/deactivate` | `PATCH /v2/serverless/ENDPOINT_ID` (`workers.max` set to 0) | +| `DELETE /v1/models/MODEL_ID` | `DELETE /v2/serverless/ENDPOINT_ID` | +| `POST /v1/secrets` | `POST /v2/account/secrets` | +| `GET /v1/instance_types` | `GET /v2/catalog/gpus`, `GET /v2/catalog/cpus` | + +### Request and response field map + +| Baseten | Runpod | +| --- | --- | +| Request body is the model input | `{"input": MODEL_INPUT}` | +| `model_input` (async) | `input` | +| `webhook_endpoint` | `webhook` | +| `max_time_in_queue_seconds` | `policy.ttl` (ms, queue plus run time) | +| `priority` | `policy.lowPriority` | +| Response body is the model output | `output` inside the job object | +| `request_id` | `id` | +| Webhook `data` | Webhook `output` | +| `X-Baseten-Request-Id` header | `job["id"]` in the handler | + +### Environment variable map + +| Inside a Baseten container | Inside a Runpod worker | +| --- | --- | +| `BT_MODEL_ID` | `RUNPOD_ENDPOINT_ID` | +| `BT_MODEL_DEPLOYMENT_ID` | No equivalent. Put the version in your image tag or an env var. | +| Replica identity | `RUNPOD_POD_ID` (the worker ID) | +| `kwargs["secrets"]["NAME"]`, `/secrets/NAME` | An env var set to `{{ RUNPOD_SECRET_NAME }}` | +| `kwargs["environment"]["name"]` | An env var you set per endpoint | +| `kwargs["data_dir"]` (`/app/data`) | A path you `COPY` to in the image | +| `/app/model_cache/VOLUME_FOLDER`, `weights` `mount_location` | `/runpod-volume/huggingface-cache/hub/` (cached models) or `/runpod-volume` (network volume) | +| `PORT` (always `8080`) | `PORT`, `PORT_HEALTH` on load balancing endpoints (default `80`) | + +### Status map + +| Baseten async request | Runpod job | +| --- | --- | +| `QUEUED` | `IN_QUEUE` | +| `IN_PROGRESS` | `IN_PROGRESS` | +| `SUCCEEDED` | `COMPLETED` | +| `FAILED` | `FAILED` | +| `EXPIRED` | `TIMED_OUT` | +| `CANCELED` | `CANCELLED` | +| `WEBHOOK_FAILED` | `COMPLETED`. The result is still available from `/status`. | + +Baseten deployment statuses map more loosely: **Active** is an endpoint with a running worker, **Scaled to zero** is an endpoint with no running workers, **Waking up** and **Loading model** are a worker that is initializing, and **Inactive** is an endpoint with max workers set to 0. + +## Migrate with a coding agent + +Install the Runpod plugin or skills and connect the MCP server. + + + + ```bash + # The plugin also connects the hosted MCP server + claude plugin marketplace add runpod/runpod-plugins-official + claude plugin install runpod@runpod + ``` + + + ```bash + # Codex, Cursor, and others: skills, then the hosted MCP server + npx skills add runpod/runpod-plugins-official --skill '*' --agent AGENT_ID --global + npx @runpod/mcp-server@latest add --client CLIENT_ID + ``` + + + +The MCP server signs in with OAuth ("Sign in with Runpod"), so you don't need an API key to start. See [Agent skills](/get-started/agent-skills) and [MCP servers](/get-started/mcp-servers) for the supported agents and client IDs. + +Then give the agent this prompt from the root of your repository: + +```text +Migrate this project from Baseten to Runpod using the runpod-cloud-migrate skill. + +1. Inventory first. Find every Truss (config.yaml + model/model.py), every docker_server + config, every Chain (truss_chains), every training config, every call to + *.api.baseten.co or inference.baseten.co, every use of BASETEN_API_KEY, and any + basetenlabs/baseten Terraform. If the baseten CLI is authenticated, also list models, + deployments, environments, autoscaling settings, and secret names. Don't change + anything yet. +2. Show me a plan: for each workload, the Runpod target (queue endpoint with a handler, + load balancing endpoint, vLLM worker, Public Endpoint, Pod, or @app.task), the GPU + pool, the worker and idle timeout settings mapped from my Baseten autoscaling, how + weights and secrets move, and an estimated cost from current Runpod pricing. List + anything with no direct equivalent (engine builder, Chains, Loops, environments, + webhook signatures) and what you propose instead. +3. Wait for my approval before you create anything billable on Runpod. +4. Migrate one workload end to end: translate the Truss to handler.py, a Dockerfile, and + requirements.txt (or a runpod.App if I ask for no Dockerfile), deploy it with max + workers of 1, and send a real request. Show me the job output next to the Baseten + output for the same input. +5. Update the client code for that workload: URL, {"input": ...} wrapper, response + parsing, async polling or webhook payload, and auth header. +6. Stop and report. Don't deactivate or delete any Baseten model, deployment, Chain, + volume, secret, or API key unless I approve that exact action. +``` + +## Troubleshooting + +**The handler raises `KeyError` on the input.** The client is still sending the bare model input. Wrap it as `{"input": {...}}`. If the handler receives a nested `input` inside `job["input"]`, the client wrapped it twice. + +**The client gets a 200 but no result.** `/runsync` returned before the job finished, or the code reads the top level of the response. Check `status`, read `output`, and switch long requests to `/run` plus `/status`. + +**The model reloads on every request.** The load is inside `handler` (a direct port of `predict` that calls `self.load()`-style code). Move it to module scope. + +**The secret is empty.** Code still reads `kwargs["secrets"]` or `/secrets/NAME`. Read the environment variable, and check that its value on the endpoint is `{{ RUNPOD_SECRET_name }}` with the exact secret name. Workers pick up a changed secret the next time they start. + +**The worker never becomes ready on a load balancing endpoint, and requests return 502.** The server listens on a port the load balancer doesn't check. Set `PORT` and `PORT_HEALTH` to the server's port, expose that port, and make sure `HEALTH_CHECK_PATH` returns `200`. A server ported from Baseten that reads `PORT` and expects `8080` needs the variable set explicitly. + +**The worker is stopped during startup with a large model.** Initialization exceeded `RUNPOD_INIT_TIMEOUT` (420 seconds by default). Raise it in the endpoint's environment variables, and use a cached model so the download doesn't count against startup. Baseten's default startup threshold is 30 minutes, so a model that loaded there can time out here. + +**Jobs sit in the queue with no workers.** The GPU pool has no stock in the endpoint's data centers, the endpoint is pinned to one data center by a network volume, or max workers is 0 because the endpoint went unused for 7 days. Add a second pool, widen the data centers, or raise max workers. + +**`exec format error` in the worker logs.** The image was built for ARM. Rebuild with `--platform linux/amd64`. + +**The new image isn't running after a push.** The endpoint still references the old tag, or you pushed over a mutable tag such as `:latest`. Push a new version tag and patch the endpoint's `image`. + +**The webhook receiver rejects Runpod's callbacks.** It still verifies `X-BASETEN-SIGNATURE` and parses `data`. Update it for the Runpod job payload. + +## Next steps + + + + Streaming, async, and concurrent handlers, error handling, and payload limits. + + + Workers, scaling, timeouts, FlashBoot, and data centers. + + + Define and deploy endpoints from Python with `runpod.App`. + + + Start workers on hosts that already hold your Hugging Face model. + + diff --git a/migrate/coreweave.mdx b/migrate/coreweave.mdx new file mode 100644 index 000000000..ac32382da --- /dev/null +++ b/migrate/coreweave.mdx @@ -0,0 +1,1279 @@ +--- +title: "Migrate from CoreWeave to Runpod" +sidebarTitle: "CoreWeave" +description: "Move Kubernetes GPU workloads, Slurm clusters, and object storage from CoreWeave to Runpod Pods, Serverless, and Instant Clusters." +--- + +This guide is for teams that run GPU workloads on [CoreWeave Kubernetes Service (CKS)](https://docs.coreweave.com/products/cks), [SUNK](https://docs.coreweave.com/products/sunk), [CoreWeave AI Object Storage](https://docs.coreweave.com/products/storage/object-storage), [Distributed File Storage](https://docs.coreweave.com/products/storage/distributed-file-storage/about), or [CoreWeave Inference](https://docs.coreweave.com/products/inference). Runpod doesn't expose a Kubernetes API, so the migration is a translation: each Kubernetes object becomes a Runpod resource that you create directly. Autoscaled inference Deployments become [Serverless](/serverless/overview) endpoints, long-running single-container workloads become [Pods](/pods/overview), Jobs become queue-based endpoints or SDK tasks, PersistentVolumeClaims and buckets become [network volumes](/storage/network-volumes), and SUNK clusters become [Instant Clusters](/instant-clusters) running Slurm. At the end, your containers run on Runpod, your clients call Runpod URLs, and your CoreWeave resources are shut down. + + +You can hand this migration to a coding agent. See [Migrate with a coding agent](#migrate-with-a-coding-agent) for the setup commands and a prompt. + + +## How CoreWeave maps to Runpod + +| CoreWeave concept | Runpod equivalent | Notes | +| --- | --- | --- | +| CKS cluster (managed control plane in its own VPC) | No cluster object | You create Pods, Serverless endpoints, and Instant Clusters directly with the API, CLI, or Terraform. There's no control plane to provision or upgrade. | +| Node Pool (`NodePool`, whole bare-metal nodes by `instanceType`) | GPU type and GPU count on each Pod or worker | CoreWeave GPU instances are whole nodes, typically with 8 GPUs. On Runpod you request GPUs per container. | +| Node Pool autoscaling (Cluster Autoscaler) | Serverless worker scaling | Serverless scales workers between `workers.min` and `workers.max`, including to zero. Pods don't autoscale. | +| Deployment + Service + HPA, KEDA, or Knative | [Serverless endpoint](/serverless/overview), queue-based or load-balancing | One endpoint replaces the Deployment, the Service, and the autoscaler. | +| Deployment or StatefulSet with a fixed replica | [Pod](/pods/overview) | One container per Pod. A Pod runs until you stop or delete it. | +| Job, CronJob | Queue-based endpoint, or `@app.task` in the [Runpod SDK](/sdk/overview) | Queue jobs are submitted over HTTP. The schedule stays in your own scheduler. | +| Container image, `imagePullSecrets` | Same image, plus a [template](/pods/templates/overview) and registry credentials | Images must be `linux/amd64`. | +| [Dedicated Inference](https://docs.coreweave.com/products/inference/dedicated) (gateway and deployment) | Load-balancing endpoint, or a [vLLM worker](/serverless/vllm/overview) | Both sides offer OpenAI-compatible routes. Dedicated Inference keeps at least one replica running. Serverless can scale to zero. | +| [Serverless Inference](https://docs.coreweave.com/products/inference/serverless) (per-token catalog models) | [Public Endpoints](/public-endpoints/overview) | Both are OpenAI-compatible hosted models. The model catalogs differ. | +| SUNK (Slurm on Kubernetes) | [Instant Clusters with Slurm](/instant-clusters/slurm-clusters) | Slurm nodes are Pods in a Runpod cluster, with no Kubernetes layer underneath. | +| HPC Interconnect (InfiniBand) | Instant Cluster high-speed interfaces | See [Instant Clusters](/instant-clusters) for supported GPUs and network speeds. | +| AI Object Storage (S3-compatible, `cwobject.com`, LOTA) | [Network volume](/storage/network-volumes) with the [S3-compatible API](/storage/s3-api), or a [global volume](/storage/globalvolume/overview) | Runpod volumes are file systems you mount. The S3-compatible API reads and writes the same files. | +| Distributed File Storage (PVC, `shared-vast`, `ReadWriteMany`) | Network volume | A network volume is pinned to one data center. A global volume attaches in any data center. | +| Local Storage (`emptyDir`, node NVMe) | Container disk | Ephemeral on both sides. | +| `LoadBalancer` Service, `.coreweave.app` hostname, Ingress | Pod proxy URL, Pod TCP port, or endpoint URL | Runpod generates the hostname and serves HTTPS on it. | +| VPC, private Services, cluster DNS | [Global networking](/pods/networking) between Pods | Private `POD_ID.runpod.internal` names between Pods in your account. You don't define CIDRs or routes. | +| Kubernetes Secret | [Runpod secret](/pods/templates/secrets) | Referenced from an environment variable as `{{ RUNPOD_SECRET_name }}`. | +| ConfigMap | Environment variables, or files on a volume | Set `env` on the Pod, endpoint, or template. | +| Region and Availability Zone (`US-EAST-04A`) | Data center ID (`US-KS-2`) | List data centers with `runpodctl datacenter list`. | +| Reserved, On-Demand, and Spot capacity plans | On-demand, per-second billing | See [Pod pricing](/pods/pricing) and [Serverless pricing](/serverless/pricing). For committed or reserved capacity, [contact sales](https://ecykq.share.hsforms.com/2MZdZATC3Rb62Dgci7knjbA). | +| CoreWeave Terraform provider (`coreweave/coreweave`) | [Runpod Terraform provider](/integrations/terraform/overview) (`runpod/runpod`) | See [Migrate your Terraform](#migrate-your-terraform). | +| `kubectl`, `cwic`, CoreWeave REST APIs | [`runpodctl`](/runpodctl/overview), [REST API v2](/api-reference-v2/overview), the Runpod SDK, the MCP server | The Terraform provider and the SDK are built on REST API v2. | +| API access token (`CW-SECRET-...`), kubeconfig, Object Storage access key | One Runpod API key (`RUNPOD_API_KEY`), plus an S3 API key for the S3-compatible API | See [Credentials](/get-started/credentials). | +| Organization, IAM access policies, groups | [Teams](/accounts-billing/manage-accounts) or [Organizations](/organizations/orgs-overview) | Roles are assigned per member. | +| CoreWeave Grafana, PromQL and LogQL at `observe.coreweave.com` | Console logs and metrics, REST v2 log and metric routes | Instant Clusters include a [Grafana dashboard](/instant-clusters/cluster-observability). | + +## What's different + +### What works differently + +**You request GPUs, not nodes.** A CoreWeave Node Pool gives you whole bare-metal nodes, and Kubernetes packs your containers onto them. On Runpod, each Pod or Serverless worker asks for a GPU type and a GPU count, and Runpod places it. There's no bin-packing, node affinity, taints, or tolerations to carry over. Replace `nodeSelector` on `node.coreweave.cloud/type` with a GPU type ID or pool. + +**Autoscaling is a property of the endpoint.** On CKS you combine an HPA or KEDA with Node Pool autoscaling, and CoreWeave documents node scale-up as taking 5 to 15 minutes. A Serverless endpoint has one scaler: workers between a minimum and maximum, driven by queue delay or request count, with scale to zero as the default. + +**Requests reach your container through Runpod's URLs.** A load-balancing endpoint serves your HTTP server at `https://ENDPOINT_ID.api.runpod.ai/PATH` and requires your Runpod API key as a bearer token. A queue-based endpoint wraps your code in a handler behind `/run` and `/runsync`. A Pod's HTTP port is served at `https://POD_ID-PORT.proxy.runpod.net`, which is public, unauthenticated, and has a 100-second timeout. + +**Storage is mounted, not claimed.** A network volume mounts at `/workspace` on Pods and at `/runpod-volume` on Serverless workers. A Pod can choose the mount path through the REST API. A Serverless worker can't, so update paths that pointed at your PVC's `mountPath`. + +**Volumes are pinned to a data center.** A network volume lives in one data center, and anything that mounts it runs there. To spread an endpoint across data centers, attach one network volume per data center and copy the data to each, or use a global volume. + +**Health checks are a single HTTP path.** Load-balancing workers answer a health path (default `/ping`) with `200` when ready and `204` while initializing. Readiness, liveness, and startup probes collapse into that one check. + +**Configuration is flat.** Kubernetes Secrets become Runpod secrets referenced from environment variables. ConfigMaps become environment variables or files you bake into the image or place on a volume. + +### What has no direct equivalent + +On the Runpod side: + +- **The Kubernetes API.** Operators, CRDs, DaemonSets, admission webhooks, Helm charts, and anything that calls the Kubernetes API from inside a container don't carry over. Rebuild the behavior with the Runpod API, or keep that component on a Kubernetes cluster elsewhere. +- **Multi-container Pods and sidecars.** A Runpod Pod or worker runs one container image, and Pods don't support Docker Compose. Combine the processes into one image with a process supervisor, or run the sidecar as its own Pod and connect over global networking. +- **VPC-level networking.** CoreWeave gives each cluster a VPC with CIDRs you choose, and offers Direct Connect. Runpod's global networking gives Pods private DNS names inside your account at 100 Mbps, on NVIDIA GPU Pods in supported data centers. Pods don't support UDP. +- **Object storage features.** AI Object Storage has bucket policies, versioning, lifecycle rules, and LOTA caching. A network volume is a file system with a fixed size and an S3-compatible API on top. If you depend on bucket features, keep an object store and mount or sync from it. +- **Whole-node control.** Kernel settings, GPU driver versions per Node Pool, kubelet overrides, and node shell access have no counterpart. Set a CUDA floor with `minCudaVersion` instead of pinning a driver. +- **CronJob schedules.** Keep the schedule in a scheduler you already run (CI, Airflow, cron) and have it submit a job to the endpoint. +- **SUNK features.** SUNK schedules Kubernetes Pods through Slurm, provisions users through SCIM, and gives users login Pods. On Runpod, managed Slurm Clusters come with Slurm and munge installed, and you configure everything else on the controller node. + +On the CoreWeave side: + +- **Scale to zero for inference.** CoreWeave documents that Dedicated Inference doesn't support scale to zero, and on CKS, GPU capacity scales at node granularity through the Cluster Autoscaler. A Serverless endpoint with `workers.min` set to `0` runs no workers while idle. +- **Per-container GPU requests without a cluster.** There's no cluster, VPC, or Node Pool to create before the first container starts. +- **A managed request queue.** Queue-based endpoints give you async submission, status polling, streaming, retries, and webhooks without deploying a queue. + + +If you want to keep a declarative, multi-cloud workflow rather than calling Runpod directly, two orchestration layers already target Runpod. [SkyPilot](/integrations/skypilot) launches tasks from a YAML file with `cloud: runpod`. [dstack](/integrations/dstack) provisions Pods on Runpod from YAML definitions, including volumes. Neither is Kubernetes, but both can soften the move for teams used to manifests. + + +## Requirements + +- Access to your CoreWeave organization: a kubeconfig for each CKS cluster, an API access token, and an AI Object Storage access key if you use buckets. +- `kubectl`, and optionally the [CoreWeave Intelligent CLI](https://github.com/coreweave/cwic) (`cwic`). +- A [Runpod account](https://www.console.runpod.io/home) with a payment method or credits, and a [Runpod API key](/get-started/credentials). +- Docker, to rebuild or retag images for `linux/amd64`. +- The AWS CLI and `jq`. + +## Step 1: Take inventory of what you run on CoreWeave + +Start from the cluster, because most of what you run is described by Kubernetes objects. Point `kubectl` at each CKS cluster and export everything that defines a workload. + +List your Node Pools and nodes to see which instance types you use. The second command is adapted from the node alias in CoreWeave's docs and prints the instance type and GPU model for each node: + +```bash +kubectl get nodepool +kubectl get nodes -o=custom-columns="NAME:metadata.name,TYPE:metadata.labels['node\.coreweave\.cloud\/type'],GPU:metadata.labels['gpu\.nvidia\.com\/model']" +``` + +The `INSTANCE TYPE` and `TARGET` columns of the first command tell you how many GPUs you hold. Each GPU instance type is a whole node, so multiply nodes by the GPUs per node. + +List the workloads, how they scale, and how they're exposed: + +```bash +kubectl get deploy,statefulset,daemonset,job,cronjob -A -o wide +kubectl get hpa -A +kubectl get svc,ingress -A -o wide +kubectl get pvc -A +kubectl get crd | grep -Ev "coreweave|cilium" +``` + +DaemonSets and CRDs that aren't CoreWeave's own are the items most likely to need redesign, so note what each one does. + +Export the full specs so you can translate them offline: + +```bash +mkdir -p inventory +for kind in deploy statefulset job cronjob hpa svc ingress pvc configmap; do + kubectl get "$kind" -A -o yaml > "inventory/$kind.yaml" +done +kubectl get secret -A -o custom-columns="NAMESPACE:metadata.namespace,NAME:metadata.name,TYPE:type" > inventory/secrets.txt +``` + +The secrets export lists names only, so no secret values land on disk. For each workload, record the image, `command` and `args`, `env`, container ports, `nvidia.com/gpu` count, node selector, volume mounts, and replica range. + +If you run SUNK, list the Slurm resources and capture the Slurm configuration from a login node: + +```bash +kubectl get sunkcluster,slurmcluster,nodesets -A +scontrol show config > inventory/slurm-config.txt +sinfo -N -l > inventory/slurm-nodes.txt +``` + +List your AI Object Storage buckets and their sizes. CoreWeave's docs configure the AWS CLI with a `cw` profile that points at `https://cwobject.com` with virtual-hosted addressing: + +```bash +aws s3 ls --profile cw +aws s3 ls s3://BUCKET_NAME --recursive --summarize --human-readable --profile cw | tail -2 +``` + +List Dedicated Inference gateways and deployments in the CoreWeave Cloud Console or through the Dedicated Inference API, and note each deployment's engine, instance type, GPU count, model path, and autoscaling `min` and `max`. + +Finally, find the code and configuration that reference CoreWeave. Run this from the root of each repository: + +```bash +grep -rnE "coreweave\.(com|cloud|app)|cwobject\.com|cwlota\.com|cwinference\.com|api\.inference\.wandb\.ai|CW-SECRET-|COREWEAVE_API_TOKEN|shared-vast|svc\.cluster\.local|KUBERNETES_SERVICE_HOST" \ + --include="*.py" --include="*.ts" --include="*.js" --include="*.go" --include="*.yaml" --include="*.yml" \ + --include="*.tf" --include="*.sh" --include="*.env*" --include="Dockerfile*" . +``` + +Each match is a hostname, credential, storage class, or in-cluster address that changes in Step 7. + +## Step 2: Set up Runpod + +Create a [Runpod account](https://www.console.runpod.io/home), add credits or a payment method, and create an API key on the [Credentials](/get-started/credentials) page. + +Install the CLI and the Python SDK, then set your key: + +```bash +curl -fsSL https://cli.runpod.net | sudo bash +pip install runpod +export RUNPOD_API_KEY="YOUR_API_KEY" +runpodctl doctor +``` + +`runpodctl` manages Pods, endpoints, templates, and volumes. The `runpod` package (2.x, Python 3.10 to 3.14) includes the worker SDK, the endpoint client, and the `rp` CLI. `rp login` stores the key in `~/.runpod/config.toml` if you prefer not to export it. + +If you manage infrastructure as code, install the [Terraform provider](/integrations/terraform/overview). If you work with a coding agent, connect the [MCP server](/get-started/mcp-servers). + +## Step 3: Choose where each workload lands + +Use the shape of each Kubernetes object to pick its target. + +| Kubernetes object on CKS | Runpod target | Why | +| --- | --- | --- | +| Deployment + Service serving HTTP inference, with HPA, KEDA, or Knative | Serverless load-balancing endpoint | Your HTTP server runs unchanged, with custom paths and request-count scaling. | +| Deployment that consumes a queue, or an API that can take `{"input": ...}` | Serverless queue-based endpoint | Built-in queue, async status, retries, and webhooks. | +| Deployment running vLLM for an OpenAI-compatible API | [vLLM worker](/serverless/vllm/overview) or load-balancing endpoint | The vLLM worker exposes `/openai/v1` with no server code of your own. | +| Deployment or StatefulSet pinned at one replica (notebooks, dev boxes, stateful services, anything long-lived) | Pod | Runs until you stop it. Persistent data goes on a network volume. | +| Job | Queue-based endpoint, or `@app.task` | A queue job per work item, or one ephemeral Pod per call that terminates when the function returns. | +| CronJob | Queue-based endpoint triggered by your scheduler | The schedule lives outside Runpod. | +| Multi-node training (MPIJob, PyTorchJob, Ray) or a SUNK cluster | Instant Cluster | Multi-node with high-speed networking, with Slurm or Ray managed for you. | +| Calls to CoreWeave Serverless Inference | Public Endpoint | Hosted model behind an OpenAI-compatible URL. | +| DaemonSet, operator, CRD controller | No target | Redesign or keep on Kubernetes. See [What has no direct equivalent](#what-has-no-direct-equivalent). | + + +Load-balancing endpoints have a 5.5-minute processing limit per request and a 30 MB payload limit, and they drop requests when every worker is busy. Use a queue-based endpoint for long-running or must-not-drop work. + + +### GPU mapping + +CoreWeave instance types are whole nodes. Map the GPU model, then set the GPU count each container requested with `nvidia.com/gpu`. + +| CoreWeave instance type | GPU | Runpod GPU type ID | Serverless pool | +| --- | --- | --- | --- | +| `gd-8xh100ib-i128` | H100, 80 GB | `NVIDIA H100 80GB HBM3`, `NVIDIA H100 PCIe` | `ADA_80_PRO` | +| `gd-8xh200ib-i128` | H200, 141 GB | `NVIDIA H200` | `HOPPER_141` | +| `b200-8x` | B200, 180 GB | `NVIDIA B200` | Check the catalog | +| `b300-8x` | B300 | `NVIDIA B300 SXM6 AC` | Check the catalog | +| `gd-8xl40s-i128` | L40S, 48 GB | `NVIDIA L40S` | `ADA_48_PRO` | +| `gd-8xl40-i128` | L40, 48 GB | `NVIDIA L40` | `ADA_48_PRO` | +| `gd-8xa100-i128` | A100, 80 GB | `NVIDIA A100-SXM4-80GB`, `NVIDIA A100 80GB PCIe` | `AMPERE_80` | +| `rtxp6000-8x`, `rtxp6000-8x-v2` | RTX PRO 6000 Blackwell Server Edition, 96 GB | `NVIDIA RTX PRO 6000 Blackwell Server Edition` | Check the catalog | +| `gb200-4x`, `gb300-4x`, `gb300-4x-e`, `gd-1xgh200` | GB200, GB300, GH200 | Not in the [GPU reference](/references/gpu-types) | Pick by VRAM, or [contact sales](https://ecykq.share.hsforms.com/2MZdZATC3Rb62Dgci7knjbA) | + +Pick the GPU that fits the workload's VRAM and throughput needs. A container that used one GPU of an 8-GPU node doesn't need an 8-GPU Pod. + +GPU stock varies by data center and changes through the day, so check it before you commit to a type: + +```bash +runpodctl gpu list +curl -s "https://api.runpod.io/v2/catalog/gpus?include=AVAILABILITY&product=SERVERLESS" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" | jq . +``` + +The catalog response includes each GPU type's pool and availability for the product you ask about (`POD`, `CLUSTER`, or `SERVERLESS`). + +## Step 4: Move your images and code + +Your container images move as they are, with four checks. + +**Architecture.** Runpod runs `linux/amd64` images. If you built an image for a different architecture, rebuild it: + +```bash +docker buildx build --platform linux/amd64 -t registry.example.com/acme/llm-server:1.4.2 --push . +``` + +**Registry access.** `imagePullSecrets` don't carry over. If the image is private, store the registry credential on Runpod and reference it by ID: + +```bash +printenv REGISTRY_TOKEN | runpodctl registry create --name "acme-registry" --username "acme-puller" --password-stdin +runpodctl registry list +``` + +If your images live in a registry inside the CKS cluster or in CoreWeave's container registry, push them to a registry that stays reachable after the cluster is gone. + +**CUDA.** On CKS, CoreWeave manages the GPU driver for the Node Pool. On Runpod, host driver versions vary by machine, so set a floor that matches your image: `--min-cuda-version 12.8` in `runpodctl`, or `gpu.minCudaVersion` in the REST API. + +**Command, ports, and environment.** Translate the container spec field by field: + +| Kubernetes field | Runpod field (REST v2) | `runpodctl` flag | +| --- | --- | --- | +| `image` | `image` | `--image` | +| `command` (overrides `ENTRYPOINT`) | `entrypoint` | `--docker-entrypoint` on `template create` | +| `args` (overrides `CMD`) | `cmd` | `--docker-start-cmd` on `template create` | +| `env[].value` | `env` | `--env` | +| `env[].valueFrom.secretKeyRef` | `env` with `{{ RUNPOD_SECRET_name }}` | `--env` | +| `envFrom.configMapRef` | `env` (one key per entry) | `--env` | +| `ports[].containerPort` | `ports` (`8000/http`, `22/tcp`) | `--ports` | +| `resources.limits["nvidia.com/gpu"]` | `gpu.count` | `--gpu-count` | +| `nodeSelector` on `node.coreweave.cloud/type` | `gpu.id` (Pods), `gpu.pools` (Serverless) | `--gpu-id` | +| `volumeMounts` for a PVC | `mounts.network` | `--network-volume-id` | +| `emptyDir` | Container disk (`disk`) | `--container-disk-in-gb` | + +Move each Kubernetes Secret your workloads read into a Runpod secret. The value is write-only once stored: + +```bash +curl -s -X POST https://api.runpod.io/v2/account/secrets \ + -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" \ + -d '{"name": "hf_token", "value": "YOUR_HUGGING_FACE_TOKEN"}' +``` + +Reference it from any Pod, endpoint, or template by setting an environment variable's value to `{{ RUNPOD_SECRET_hf_token }}`. + +### Multi-container Pods + +If a Kubernetes Pod has more than one container, decide per container. Init containers that download models become a one-time copy onto a network volume (Step 5). Log and metrics sidecars usually go away, because Runpod collects container logs. A sidecar your app talks to over `localhost` either moves into the same image under a process supervisor, or becomes its own Pod that you reach at `POD_ID.runpod.internal` with [global networking](/pods/networking). + +## Step 5: Move your data and models + +Choose the destination by how the data is used. + +| CoreWeave storage | Runpod destination | Use it when | +| --- | --- | --- | +| Distributed File Storage PVC (`shared-vast`) | Network volume | Workloads read and write files and run in one data center. | +| AI Object Storage bucket holding weights or datasets | Network volume, loaded through the S3-compatible API or from a Pod | You want the files mounted next to compute. | +| Data read by workers in several data centers | Global volume, or one network volume per data center | The endpoint shouldn't be pinned to a single data center. | +| Hugging Face models pulled at startup | [Cached models](/serverless/endpoints/model-caching) on the endpoint | The model is on Hugging Face and you run it on Serverless. | +| `emptyDir`, `/mnt/local` scratch | Container disk | The data doesn't need to survive a restart. | + +Create the network volume first, in a data center that has the GPU you chose in Step 3 and that supports the [S3-compatible API](/storage/s3-api#datacenter-availability): + +```bash +runpodctl network-volume create --name "models" --size 200 --data-center-id "US-KS-2" +``` + +Size it for the data you're copying. A network volume can grow later but can't shrink. + +### Copy from AI Object Storage + +The two S3 endpoints use different credentials, so copy through a machine that can reach both. The most reliable route is a Pod with the network volume mounted, because files are written straight to the file system. + + + + Start a CPU Pod with the volume attached, then [connect to it](/pods/connect-to-a-pod) with the web terminal or SSH: + + ```bash + runpodctl pod create --name "data-copy" --compute-type cpu \ + --image "runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04" \ + --network-volume-id NETWORK_VOLUME_ID --data-center-ids US-KS-2 + ``` + + Inside the Pod, install the AWS CLI, configure the `cw` profile the way CoreWeave's docs describe, and sync the bucket onto the volume: + + ```bash + pip install awscli + + aws configure --profile cw + aws configure set endpoint_url https://cwobject.com --profile cw + aws configure set s3.addressing_style virtual --profile cw + + aws s3 sync s3://BUCKET_NAME /workspace/models --profile cw + ``` + + AI Object Storage requires virtual-hosted addressing, so keep the `addressing_style` line. Use `https://cwobject.com` here, not `http://cwlota.com`, because the LOTA endpoint only works inside a CoreWeave cluster. Delete the Pod when the sync finishes. + + To use Rclone instead, define a remote with CoreWeave's documented settings and copy to the mounted path: + + ```ini ~/.config/rclone/rclone.conf + [coreweave] + type = s3 + provider = Other + access_key_id = YOUR_CW_ACCESS_KEY_ID + secret_access_key = YOUR_CW_SECRET_ACCESS_KEY + endpoint = https://cwobject.com + force_path_style = false + no_check_bucket = true + ``` + + ```bash + rclone copy coreweave:BUCKET_NAME /workspace/models --progress --stats 15s + ``` + + + Create an S3 API key on the Runpod [Credentials](https://console.runpod.io/user/credentials) page and store it in a second AWS CLI profile named `runpod`. Then stage the data locally and upload it. The bucket name on the Runpod side is the network volume ID, and the region is the data center ID: + + ```bash + aws s3 sync s3://BUCKET_NAME ./stage --profile cw + + aws s3 sync ./stage s3://NETWORK_VOLUME_ID/models --profile runpod \ + --region US-KS-2 --endpoint-url https://s3api-us-ks-2.runpod.io/ + ``` + + Runpod's `aws s3 sync` support is limited for directories with more than 10,000 files. For large trees, sync in batches or use the Pod method. No Pod needs to be running for this route. + + + +On a Pod, `/workspace/models/file` is the same file as `s3://NETWORK_VOLUME_ID/models/file` in the S3-compatible API and `/runpod-volume/models/file` on a Serverless worker. + +### Copy from a PVC + +Distributed File Storage volumes are reachable only from the cluster they live in, so push from inside CKS. This Job mounts the PVC and syncs it to the network volume through Runpod's S3-compatible API: + +```yaml pvc-export.yaml +apiVersion: batch/v1 +kind: Job +metadata: + name: pvc-export +spec: + backoffLimit: 1 + template: + spec: + restartPolicy: Never + containers: + - name: sync + image: amazon/aws-cli:latest + args: + - s3 + - sync + - /data + - s3://NETWORK_VOLUME_ID/models + - --region + - US-KS-2 + - --endpoint-url + - https://s3api-us-ks-2.runpod.io/ + env: + - name: AWS_ACCESS_KEY_ID + valueFrom: + secretKeyRef: { name: runpod-s3, key: access_key } + - name: AWS_SECRET_ACCESS_KEY + valueFrom: + secretKeyRef: { name: runpod-s3, key: secret_key } + volumeMounts: + - name: data + mountPath: /data + readOnly: true + volumes: + - name: data + persistentVolumeClaim: + claimName: model-cache +``` + +Create the `runpod-s3` Secret from your Runpod S3 API key before you apply the Job. For a PVC with a very large number of small files, archive them into tar files first, or run `rsync` over SSH to a Runpod Pod as described in [Network volumes](/storage/network-volumes#using-rsync-over-ssh). + +### Global volumes + +A global volume isn't tied to a data center and is designed for data that many Pods or workers read, such as model weights. The S3-compatible API doesn't work with global volumes, so load one by attaching it to a Pod and copying onto the mount with the Pod method above. See [Global volumes](/storage/globalvolume/overview) for mount paths, request costs, and when to prefer a network volume. + +## Step 6: Deploy on Runpod + +This section translates one real set of manifests. The inference server below is the kind of workload CoreWeave's own CKS tutorials deploy: a vLLM server behind a Service, with a PVC for model weights, a Hugging Face token in a Secret, a node selector for H100 nodes, and an HPA. + +```yaml llm.yaml +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: model-cache +spec: + accessModes: ["ReadWriteMany"] + storageClassName: shared-vast + resources: + requests: + storage: 200Gi +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: llama-3-1-8b +spec: + replicas: 1 + selector: + matchLabels: { app: llama-3-1-8b } + template: + metadata: + labels: { app: llama-3-1-8b } + spec: + nodeSelector: + node.coreweave.cloud/type: gd-8xh100ib-i128 + imagePullSecrets: + - name: acme-registry + containers: + - name: vllm-server + image: registry.example.com/acme/llm-server:1.4.2 + command: ["vllm", "serve"] + args: + - meta-llama/Llama-3.1-8B-Instruct + - --host + - 0.0.0.0 + - --port + - "8000" + - --download-dir + - /models + env: + - name: HF_TOKEN + valueFrom: + secretKeyRef: { name: hf-token-secret, key: api_token } + ports: + - containerPort: 8000 + readinessProbe: + httpGet: { path: /health, port: 8000 } + resources: + requests: { nvidia.com/gpu: 1 } + limits: { nvidia.com/gpu: 1 } + volumeMounts: + - name: models + mountPath: /models + volumes: + - name: models + persistentVolumeClaim: + claimName: model-cache +--- +apiVersion: v1 +kind: Service +metadata: + name: llama-3-1-8b + annotations: + service.beta.kubernetes.io/coreweave-load-balancer-type: public + service.beta.kubernetes.io/external-hostname: llama +spec: + type: LoadBalancer + selector: { app: llama-3-1-8b } + ports: + - port: 80 + targetPort: 8000 +--- +apiVersion: autoscaling/v2 +kind: HorizontalPodAutoscaler +metadata: + name: llama-3-1-8b +spec: + scaleTargetRef: { apiVersion: apps/v1, kind: Deployment, name: llama-3-1-8b } + minReplicas: 1 + maxReplicas: 4 + metrics: + - type: Pods + pods: + metric: { name: http_requests_in_flight } + target: { type: AverageValue, averageValue: "4" } +``` + +On CoreWeave this is reachable at `http://llama.ORG_ID-CLUSTER_NAME.coreweave.app`. The steps below assume you created the `models` network volume and the `hf_token` secret in the earlier steps. + +### Serverless load-balancing endpoint + +A load-balancing endpoint runs the same HTTP server and replaces the Deployment, Service, and HPA with one resource. The endpoint type is set at creation with the REST API (`runpodctl serverless create` creates queue-based endpoints): + +```bash +curl -s -X POST https://api.runpod.io/v2/serverless \ + -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" \ + -d '{ + "name": "llama-3-1-8b", + "type": "LOAD_BALANCER", + "image": "registry.example.com/acme/llm-server:1.4.2", + "registry": "REGISTRY_AUTH_ID", + "entrypoint": ["vllm", "serve"], + "cmd": ["meta-llama/Llama-3.1-8B-Instruct", "--host", "0.0.0.0", "--port", "8000", + "--download-dir", "/runpod-volume/models"], + "ports": ["8000/http"], + "disk": 40, + "env": { + "PORT": "8000", + "PORT_HEALTH": "8000", + "HEALTH_CHECK_PATH": "/health", + "HF_TOKEN": "{{ RUNPOD_SECRET_hf_token }}" + }, + "gpu": {"pools": ["ADA_80_PRO"], "count": 1, "minCudaVersion": "12.8"}, + "mounts": {"network": [{"volumeId": "NETWORK_VOLUME_ID"}]}, + "workers": {"min": 0, "max": 4, "idleTimeout": 60}, + "scaling": {"type": "REQUEST_COUNT", "requestCount": 4}, + "timeout": 300000 + }' +``` + +The response contains the endpoint `id` and `requestUrls.base`, which is `https://ENDPOINT_ID.api.runpod.ai`. Here's how each piece of the manifests landed: + +| In `llm.yaml` | In the endpoint | +| --- | --- | +| `command`, `args` | `entrypoint`, `cmd`. The download path changed from `/models` to `/runpod-volume/models` because Serverless mounts volumes at `/runpod-volume`. | +| `containerPort: 8000` and the Service | `ports`, `PORT`, and `PORT_HEALTH`. Set all three to the same port, or the worker starts but never becomes ready. | +| `readinessProbe` on `/health` | `HEALTH_CHECK_PATH`. Omit it if your server answers `/ping`. | +| `secretKeyRef` | `{{ RUNPOD_SECRET_hf_token }}` | +| `imagePullSecrets` | `registry` | +| `nodeSelector` and `nvidia.com/gpu: 1` | `gpu.pools` and `gpu.count` | +| The PVC | `mounts.network` | +| HPA `minReplicas: 1`, `maxReplicas: 4` | `workers.min: 0`, `workers.max: 4`. Set `min` to `1` to keep one worker warm the way the HPA did. | +| HPA target of 4 in-flight requests per replica | `scaling.requestCount: 4` | + +Wait for a worker to become ready, then send the same OpenAI-style request your clients send today: + +```bash +curl -s https://api.runpod.ai/v2/ENDPOINT_ID/health -H "Authorization: Bearer $RUNPOD_API_KEY" + +curl -s https://ENDPOINT_ID.api.runpod.ai/v1/chat/completions \ + -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" \ + -d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}' +``` + +The first request after scale to zero waits for a cold start, so clients need retries. In the console, click **New Endpoint** on the Serverless page and set **Endpoint Type** to **Load Balancer**. See [Load balancing endpoints](/serverless/load-balancing/overview) for health check behavior and limits. + + +If the server is plain vLLM and you don't need custom routes, the [vLLM worker](/serverless/vllm/overview) gives you a queue-based endpoint with an OpenAI-compatible route at `https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1` and no image to maintain. + + +### Pod + +Use a Pod when the workload should stay up as a single long-running container, the way a one-replica Deployment or StatefulSet does. The same manifests translate to one create call. + + + + Create a template that holds the image, command, ports, and environment, then create the Pod from it: + + ```bash + runpodctl template create --name "llama-3-1-8b-pod" \ + --image "registry.example.com/acme/llm-server:1.4.2" \ + --registry-auth-id REGISTRY_AUTH_ID \ + --container-disk-in-gb 40 \ + --ports "8000/http,22/tcp" \ + --docker-entrypoint "vllm,serve" \ + --docker-start-cmd "meta-llama/Llama-3.1-8B-Instruct,--host,0.0.0.0,--port,8000,--download-dir,/workspace/models" \ + --env '{"HF_TOKEN": "{{ RUNPOD_SECRET_hf_token }}"}' + + runpodctl pod create --name "llama-3-1-8b" --template-id TEMPLATE_ID \ + --gpu-id "NVIDIA H100 80GB HBM3" --gpu-count 1 \ + --network-volume-id NETWORK_VOLUME_ID \ + --data-center-ids US-KS-2 \ + --min-cuda-version 12.8 + ``` + + The network volume mounts at `/workspace`, so the download path points there. + + + The REST API takes the container settings inline and lets you choose the mount path, so `/models` can stay as it was: + + ```bash + curl -s -X POST https://api.runpod.io/v2/pods \ + -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" \ + -d '{ + "name": "llama-3-1-8b", + "image": "registry.example.com/acme/llm-server:1.4.2", + "registry": "REGISTRY_AUTH_ID", + "entrypoint": ["vllm", "serve"], + "cmd": ["meta-llama/Llama-3.1-8B-Instruct", "--host", "0.0.0.0", "--port", "8000", + "--download-dir", "/models"], + "ports": ["8000/http", "22/tcp"], + "disk": 40, + "env": {"HF_TOKEN": "{{ RUNPOD_SECRET_hf_token }}"}, + "gpu": {"id": "NVIDIA H100 80GB HBM3", "count": 1, "minCudaVersion": "12.8"}, + "dataCenterIds": ["US-KS-2"], + "mounts": {"network": [{"volumeId": "NETWORK_VOLUME_ID", "path": "/models"}]} + }' + ``` + + + +Wait for the container to report as running, then call it through the proxy URL: + +```bash +until [ "$(runpodctl pod get POD_ID --output json | jq -r '.runtimeStatus')" = "running" ]; do sleep 5; done + +curl -s https://POD_ID-8000.proxy.runpod.net/v1/models +``` + + +The proxy URL is public and unauthenticated, and requests time out after 100 seconds. The `LoadBalancer` Service it replaces was also public, but if you relied on `loadBalancerSourceRanges` or a private Service, add authentication in your server (for example, vLLM's API key option) or keep the port private and reach it from other Pods over global networking. For long requests or non-HTTP protocols, expose a TCP port instead. See [Expose ports](/pods/configuration/expose-ports). + + +A Pod doesn't have an HPA. If the Deployment scaled beyond one replica, use the Serverless endpoint above. In the console, the same Pod is created from **Pods > Deploy**. + +### Queue-based endpoint for a Job + +A Kubernetes Job runs a container to completion. On Runpod, the same code becomes a handler that processes one job per request. This Job runs a GPU batch script once per input shard: + +```yaml embed-job.yaml +apiVersion: batch/v1 +kind: Job +metadata: + name: embed-shard-0001 +spec: + backoffLimit: 2 + activeDeadlineSeconds: 3600 + template: + spec: + restartPolicy: Never + nodeSelector: + node.coreweave.cloud/type: gd-8xl40s-i128 + containers: + - name: embed + image: registry.example.com/acme/embedder:2.1.0 + command: ["python", "embed.py"] + args: ["--input", "/data/shards/0001.jsonl", "--output", "/data/embeddings/0001.parquet"] + resources: + limits: { nvidia.com/gpu: 1 } + volumeMounts: + - name: data + mountPath: /data + volumes: + - name: data + persistentVolumeClaim: + claimName: datasets +``` + +Wrap the script in a handler. The arguments that were hardcoded in the manifest become the job's input: + +```python handler.py +import subprocess + +import runpod + + +def handler(job): + args = job["input"] + result = subprocess.run( + ["python", "embed.py", "--input", args["input"], "--output", args["output"]], + capture_output=True, + text=True, + ) + if result.returncode != 0: + raise RuntimeError(result.stderr[-2000:]) + return {"output": args["output"]} + + +runpod.serverless.start({"handler": handler}) +``` + +Raising an exception marks the job as `FAILED`, which is what a non-zero exit code did for the Job. Build a worker image on top of the existing one: + +```dockerfile Dockerfile +FROM registry.example.com/acme/embedder:2.1.0 +RUN pip install --no-cache-dir runpod +COPY handler.py . +ENTRYPOINT [] +CMD ["python", "-u", "handler.py"] +``` + +```bash +docker buildx build --platform linux/amd64 -t registry.example.com/acme/embedder-worker:2.1.0 --push . +``` + +Create a Serverless template and an endpoint from it: + +```bash +runpodctl template create --name "embedder-worker" --serverless \ + --image "registry.example.com/acme/embedder-worker:2.1.0" \ + --registry-auth-id REGISTRY_AUTH_ID --container-disk-in-gb 20 + +runpodctl serverless create --name "embedder" --template-id TEMPLATE_ID \ + --gpu-id ADA_48_PRO --workers-min 0 --workers-max 8 \ + --network-volume-id NETWORK_VOLUME_ID --execution-timeout 3600 +``` + +Submit one request per shard. The volume that was at `/data` is at `/runpod-volume` on the worker: + +```bash +curl -s -X POST https://api.runpod.ai/v2/ENDPOINT_ID/run \ + -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" \ + -d '{"input": {"input": "/runpod-volume/shards/0001.jsonl", "output": "/runpod-volume/embeddings/0001.parquet"}}' + +curl -s https://api.runpod.ai/v2/ENDPOINT_ID/status/JOB_ID -H "Authorization: Bearer $RUNPOD_API_KEY" +``` + +The Job fields map as follows: + +| Job field | Queue-based endpoint | +| --- | --- | +| `parallelism` | `--workers-max`. To process several jobs on one worker, use a [concurrent handler](/serverless/workers/concurrent-handler). | +| `completions`, indexed Jobs | One `/run` request per work item. | +| `activeDeadlineSeconds` | `--execution-timeout`, or `policy.executionTimeout` on a request. | +| `backoffLimit` | Failed jobs can be requeued with `/retry`. | +| `ttlSecondsAfterFinished` | Results of `/run` jobs are kept for 30 minutes. Write outputs to a volume or pass a `webhook`. | +| CronJob `schedule` | Your scheduler calls `/run`. | + +For large offline workloads that can tolerate multi-hour latency, see [Batch jobs](/serverless/batch-jobs). + +If you'd rather not build a worker image, the Runpod SDK can run the same script as a task. Each call provisions a Pod, runs to completion, and terminates: + +```python main.py +import subprocess + +import runpod + +app = runpod.App("embed-batch") +datasets = runpod.NetworkVolume("datasets", size=200) + + +@app.task( + gpu="L40S", + image="registry.example.com/acme/embedder:2.1.0", + registry_auth="acme-registry", + mounts={"/data": datasets}, +) +def embed(input_path: str, output_path: str): + subprocess.run( + ["python", "embed.py", "--input", input_path, "--output", output_path], + check=True, + ) + return {"output": output_path} + + +@runpod.local_entrypoint +def main(): + print(embed.remote("/data/shards/0001.jsonl", "/data/embeddings/0001.parquet")) +``` + +Run it with `rp flash dev main.py`. A task chooses its own mount path, so `/data` stays as it was. See [Build an app](/sdk/apps/build-app) for tasks, volumes, and deployment. + +### Instant Cluster with Slurm + +A SUNK cluster becomes a managed Slurm Cluster. In the console, open [Instant Clusters](https://console.runpod.io/cluster), click **Create Cluster**, choose **Slurm Cluster**, then set the Pod count, GPU type, region, and an optional network volume. The REST API creates the same thing: + +```bash +curl -s -X POST https://api.runpod.io/v2/clusters \ + -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" \ + -d '{ + "name": "slurm-train", + "type": "SLURM", + "image": "runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04", + "compute": {"gpuTypeId": "NVIDIA H100 80GB HBM3", "gpuCountPerPod": 8, "podCount": 4}, + "mounts": {"network": [{"volumeId": "NETWORK_VOLUME_ID", "path": "/workspace"}]}, + "startSsh": true + }' +``` + +Runpod designates one node as the Slurm controller and the rest as agents, with Slurm and munge configured. Connect to the controller over SSH and submit jobs with the same commands you use on a SUNK login node: + +```bash +sinfo +sbatch your-job-script.sh +squeue +``` + +Before you move a SUNK workload, check these differences: + +- **Images.** Managed Slurm Clusters support only official Runpod PyTorch images. If your `NodeSet` used a custom image, install your dependencies at job start, keep environments on the network volume, or [set up Slurm yourself](/instant-clusters/slurm) on a cluster with your own image. +- **Size.** Instant Clusters are documented at 2 to 8 nodes. For larger clusters, [contact sales](https://ecykq.share.hsforms.com/2MZdZATC3Rb62Dgci7knjbA). Cluster availability depends on GPU stock, so check it for your GPU type and region before you plan a cutover date. +- **Shared storage.** The `homeDir` and `additionalMounts` volumes in a `SunkCluster` become one network volume, mounted at `/workspace` on each node, in the cluster's data center. +- **Users and accounting.** SUNK provisions POSIX users through SCIM and ships an accounting database. On Runpod, you connect with your account's SSH keys and configure users, partitions, and accounting in `/etc/slurm/slurm.conf` on the controller. +- **Job scripts.** `sbatch` scripts carry over. Change partition names, paths under `/mnt/home` or `/mnt/data`, and any `srun --container-image` usage that relied on SUNK's container support. + +## Step 7: Update the code that calls CoreWeave + +### Inference clients + +For an HTTP server moved to a load-balancing endpoint, the request and response bodies don't change. The hostname changes, and every request needs your Runpod API key. + + + + ```python client.py + from openai import OpenAI + + client = OpenAI( + base_url="http://llama.ORG_ID-CLUSTER_NAME.coreweave.app/v1", + api_key="not-used", + ) + + response = client.chat.completions.create( + model="meta-llama/Llama-3.1-8B-Instruct", + messages=[{"role": "user", "content": "Hello"}], + ) + print(response.choices[0].message.content) + ``` + + + ```python client.py + import os + + from openai import OpenAI + + client = OpenAI( + base_url="https://ENDPOINT_ID.api.runpod.ai/v1", + api_key=os.environ["RUNPOD_API_KEY"], + max_retries=5, + ) + + response = client.chat.completions.create( + model="meta-llama/Llama-3.1-8B-Instruct", + messages=[{"role": "user", "content": "Hello"}], + ) + print(response.choices[0].message.content) + ``` + + + +The retries cover cold starts after the endpoint scales to zero. Use the same pattern for the other URLs you may be replacing: + +| Calling today | Call on Runpod | What changes | +| --- | --- | --- | +| `LoadBalancer` or Ingress hostname on `.coreweave.app` | `https://ENDPOINT_ID.api.runpod.ai/PATH` (load-balancing), or `https://POD_ID-PORT.proxy.runpod.net/PATH` (Pod) | Hostname. Endpoints require `Authorization: Bearer RUNPOD_API_KEY`. | +| Dedicated Inference gateway, `https://GATEWAY_NAME.ORG_ID.gw.cwinference.com/v1` | `https://ENDPOINT_ID.api.runpod.ai/v1` (load-balancing), or `https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1` (vLLM worker) | Base URL and API key. The `model` field must match what your server or worker serves. | +| Serverless Inference, `https://api.inference.wandb.ai/v1` | `https://api.runpod.ai/v2/MODEL_SLUG/openai/v1` ([Public Endpoints](/public-endpoints/overview)) | Base URL, API key, and model ID. Check that the model you use is in the catalog. | +| In-cluster Service DNS, `http://NAME.NAMESPACE.svc.cluster.local` | The endpoint URL, or `http://POD_ID.runpod.internal:PORT` between Pods with global networking | In-cluster names don't resolve on Runpod. | + +### Batch clients + +Code that created Kubernetes Jobs now submits queue requests. Async work uses `/run` and polls `/status`, or passes a `webhook` URL to be notified on completion. + + + + ```python submit.py + from kubernetes import client, config + + config.load_kube_config() + + job = client.V1Job( + metadata=client.V1ObjectMeta(name="embed-shard-0001"), + spec=client.V1JobSpec( + backoff_limit=2, + template=client.V1PodTemplateSpec( + spec=client.V1PodSpec( + restart_policy="Never", + containers=[ + client.V1Container( + name="embed", + image="registry.example.com/acme/embedder:2.1.0", + command=["python", "embed.py"], + args=["--input", "/data/shards/0001.jsonl", + "--output", "/data/embeddings/0001.parquet"], + resources=client.V1ResourceRequirements(limits={"nvidia.com/gpu": "1"}), + ) + ], + ) + ), + ), + ) + client.BatchV1Api().create_namespaced_job(namespace="default", body=job) + ``` + + + ```python submit.py + import os + import time + + import requests + + BASE = "https://api.runpod.ai/v2/ENDPOINT_ID" + HEADERS = {"Authorization": f"Bearer {os.environ['RUNPOD_API_KEY']}"} + + job = requests.post( + f"{BASE}/run", + headers=HEADERS, + json={ + "input": { + "input": "/runpod-volume/shards/0001.jsonl", + "output": "/runpod-volume/embeddings/0001.parquet", + }, + "policy": {"executionTimeout": 3600000}, + }, + timeout=30, + ).json() + + while True: + status = requests.get(f"{BASE}/status/{job['id']}", headers=HEADERS, timeout=30).json() + if status["status"] in ("COMPLETED", "FAILED", "CANCELLED", "TIMED_OUT"): + break + time.sleep(5) + + print(status) + ``` + + + +See [Send requests](/serverless/endpoints/send-requests) for `/runsync`, streaming with `/stream`, webhooks, and rate limits. + +### Storage clients + +Code that read from AI Object Storage with an S3 client has two options. The simpler one is to read files from the mounted volume path. If the code must keep using S3 calls, point it at the network volume's S3-compatible API: + +| Setting | CoreWeave | Runpod | +| --- | --- | --- | +| Endpoint URL | `https://cwobject.com`, or `http://cwlota.com` in-cluster | `https://s3api-DATACENTER.runpod.io/` | +| Bucket | Bucket name | Network volume ID | +| Region | Availability Zone, such as `US-EAST-04A` | Data center ID, such as `US-KS-2` | +| Credentials | Object Storage access key | S3 API key from the Runpod console | + +Check the [compatibility reference](/storage/s3-api#s3-api-compatibility-reference) for the S3 operations your code uses. + +### Control-plane automation + +Scripts and CI jobs that drove CoreWeave through `kubectl`, `cwic`, or the CoreWeave API now call `runpodctl` or REST API v2. The [Reference](#reference) section maps the commands. Replace `COREWEAVE_API_TOKEN` and kubeconfig secrets in CI with `RUNPOD_API_KEY`. + +## Step 8: Verify, cut over, and shut down + +**Verify from outside.** Send production-shaped requests to the Runpod endpoint from a machine outside Runpod, and compare the responses with CoreWeave's for the same inputs. Measure latency for warm requests and for a cold start, then decide whether to keep `workers.min` above zero. + +```bash +curl -s -o /dev/null -w "%{http_code} %{time_total}s\n" \ + https://ENDPOINT_ID.api.runpod.ai/v1/chat/completions \ + -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" \ + -d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}' +``` + +**Run both in parallel.** Keep the CoreWeave workload serving while you send a share of traffic to Runpod from your client, gateway, or DNS layer. Watch error rates, cold starts, and worker counts in the endpoint's **Metrics** and **Logs** tabs, or pull them from the API: + +```bash +curl -s "https://api.runpod.io/v2/serverless/ENDPOINT_ID/metrics/requests?interval=1h" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" | jq . +``` + +**Shift traffic.** Move the remaining traffic once the Runpod side holds at your peak load. Raise `workers.max` first if your peak needs more workers than you tested with. + +**Shut down CoreWeave.** Do this only after you've confirmed the data copy and kept the CoreWeave side idle for a period you're comfortable with. Delete in this order: + +```bash +# 1. Workloads, Services, and autoscalers +kubectl delete -f llm.yaml + +# 2. Volumes (after verifying the copy on Runpod) +kubectl delete pvc model-cache + +# 3. Node Pools. CKS removes the nodes without waiting for workloads to finish. +kubectl delete nodepool NODE_POOL_NAME + +# 4. Dedicated Inference: deployments first, then gateways +cwic inference deployment delete DEPLOYMENT_ID +cwic inference gateway delete GATEWAY_ID + +# 5. Buckets. Only empty buckets can be deleted. +aws s3 rm s3://BUCKET_NAME --recursive --profile cw +aws s3api delete-bucket --bucket BUCKET_NAME --profile cw +``` + +Then delete each CKS cluster from the **Clusters** page of the CoreWeave Cloud Console (the cluster must be in the Running, Failed, or Preempted state), and delete API access tokens and Object Storage access keys you no longer need. If you manage CoreWeave with Terraform, run `terraform destroy` on that configuration instead. + + +Deleting resources doesn't end a commitment. CoreWeave bills Reserved Instances at a flat rate for the term whether or not instances are running, so talk to your CoreWeave account team about reserved capacity before you plan the cutover date. Versioned buckets also need their object versions and delete markers removed before the bucket can be deleted. + + +**Guard costs on Runpod.** Keep `workers.min` at `0` on endpoints that can tolerate cold starts, and keep the idle timeout short. Stop or delete Pods you aren't using, because a running Pod bills per second whether or not it's serving. Enable [low balance notifications](/accounts-billing/billing#low-balance-notifications), and review spend by resource: + +```bash +runpodctl pod list --all +runpodctl serverless list +curl -s "https://api.runpod.io/v2/billing?bucketSize=day&lastN=7" -H "Authorization: Bearer $RUNPOD_API_KEY" | jq . +``` + +## Migrate your Terraform + +The CoreWeave provider (`coreweave/coreweave`) manages clusters, VPCs, buckets, and Dedicated Inference. Workloads inside the cluster are usually managed with the Kubernetes or Helm providers. The Runpod provider (`runpod/runpod`, `~> 2.0`) is built on REST API v2 and manages the workloads directly. + +| CoreWeave or Kubernetes resource | Runpod resource | Notes | +| --- | --- | --- | +| `coreweave_networking_vpc` | None | No VPC to declare. Set `global_networking = true` on `runpod_pod` for private Pod-to-Pod traffic. | +| `coreweave_cks_cluster` | None | Workloads are top-level resources. | +| `NodePool` manifest (`kubernetes_manifest`) | `gpu_type_id` on `runpod_pod`, `gpu_pools` on `runpod_serverless_endpoint` | Capacity is requested per resource. | +| `kubernetes_deployment` + `kubernetes_service` + `kubernetes_horizontal_pod_autoscaler` | `runpod_serverless_endpoint` (with `runpod_template` for the command) | `endpoint_type`, `workers_min`, `workers_max`, `scaling_type`, `scaling_value`. | +| `kubernetes_deployment` or `kubernetes_stateful_set` with one replica | `runpod_pod` | `desired_state` starts and stops it. | +| `kubernetes_persistent_volume_claim` | `runpod_network_volume` | `data_center` is required. | +| `coreweave_object_storage_bucket` | `runpod_network_volume` or `runpod_global_volume` | Bucket policy, versioning, lifecycle, and settings resources have no counterpart. | +| `coreweave_object_storage_access_key` | None | Create an S3 API key in the Runpod console. | +| `coreweave_inference_gateway`, `coreweave_inference_deployment`, `coreweave_inference_capacity_claim` | `runpod_serverless_endpoint` | One resource covers routing, scaling, and the container. | +| `coreweave_container_registry_*` | `runpod_registry` | Stores a pull credential for a registry you host elsewhere. | +| `kubernetes_secret` | None | Create the secret with `POST /v2/account/secrets` and reference it as `{{ RUNPOD_SECRET_name }}` in `env`. | +| SUNK (`SunkCluster` manifest or Helm release) | `runpod_cluster` with `type = "SLURM"` | Changing most arguments replaces the cluster. | + +Here's the infrastructure for the inference server from Step 6 on CoreWeave. The Deployment, Service, and HPA would follow in the Kubernetes provider: + +```hcl coreweave.tf +terraform { + required_providers { + coreweave = { + source = "coreweave/coreweave" + } + } +} + +provider "coreweave" { + # Reads the token from the COREWEAVE_API_TOKEN environment variable. +} + +resource "coreweave_networking_vpc" "default" { + name = "default" + zone = "US-EAST-04A" + + host_prefixes = [ + { + name = "primary" + type = "PRIMARY" + prefixes = ["10.16.192.0/18"] + }, + ] + + vpc_prefixes = [ + { name = "pod cidr", value = "10.0.0.0/13" }, + { name = "service cidr", value = "10.16.0.0/22" }, + { name = "internal lb cidr", value = "10.32.4.0/22" }, + ] +} + +resource "coreweave_cks_cluster" "default" { + name = "inference" + version = "v1.35" + zone = "US-EAST-04A" + vpc_id = coreweave_networking_vpc.default.id + public = true + pod_cidr_name = "pod cidr" + service_cidr_name = "service cidr" + internal_lb_cidr_names = ["internal lb cidr"] +} + +resource "coreweave_object_storage_bucket" "models" { + name = "acme-models" + zone = "US-EAST-04A" +} +``` + +The Runpod configuration declares the volume and the endpoint, and nothing else: + +```hcl runpod.tf +terraform { + required_version = ">= 1.6" + + required_providers { + runpod = { + source = "runpod/runpod" + version = "~> 2.0" + } + } +} + +provider "runpod" { + # Reads the key from the RUNPOD_API_KEY environment variable. +} + +resource "runpod_network_volume" "models" { + name = "models" + size = 200 + data_center = "US-KS-2" +} + +resource "runpod_template" "llama" { + name = "llama-3-1-8b" + image = "registry.example.com/acme/llm-server:1.4.2" + serverless = true + disk = 40 + docker_entrypoint = ["vllm", "serve"] + args = "meta-llama/Llama-3.1-8B-Instruct --host 0.0.0.0 --port 8000 --download-dir /runpod-volume/models" + ports = ["8000/http"] + + env = { + PORT = "8000" + PORT_HEALTH = "8000" + HEALTH_CHECK_PATH = "/health" + HF_TOKEN = "{{ RUNPOD_SECRET_hf_token }}" + } +} + +resource "runpod_serverless_endpoint" "llama" { + name = "llama-3-1-8b" + template_id = runpod_template.llama.id + endpoint_type = "LOAD_BALANCER" + gpu_pools = ["ADA_80_PRO"] + gpu_count = 1 + min_cuda_version = "12.8" + ports = ["8000/http"] + network_volumes = [runpod_network_volume.models.id] + data_center_ids = ["US-KS-2"] + scaling_type = "REQUEST_COUNT" + scaling_value = 4 + workers_min = 0 + workers_max = 4 + timeout = 300000 +} + +output "llama_base_url" { + value = runpod_serverless_endpoint.llama.request_urls.base +} +``` + +Terraform creates the volume empty, so load the model data (Step 5) before you send traffic. Don't try to move state between the two providers. Apply the Runpod configuration alongside the CoreWeave one, cut over, then destroy the CoreWeave configuration. For every argument, see the [provider reference on the Terraform Registry](https://registry.terraform.io/providers/runpod/runpod/latest/docs). + +## Reference + +### CLI command map + +| CoreWeave | Runpod | +| --- | --- | +| `cwic auth login`, `export KUBECONFIG=...` | `export RUNPOD_API_KEY=...`, or `rp login` | +| `cwic auth whoami` | `runpodctl user` | +| `kubectl get nodepool`, `cwic node get` | `runpodctl gpu list`, `runpodctl datacenter list` | +| `kubectl apply -f deployment.yaml` | `runpodctl serverless create`, or `runpodctl pod create` | +| `kubectl get pods`, `kubectl get deploy` | `runpodctl pod list`, `runpodctl serverless list` | +| `kubectl describe pod NAME` | `runpodctl pod get POD_ID`, `runpodctl serverless get ENDPOINT_ID --include-workers` | +| `kubectl logs NAME` | `runpodctl pod logs POD_ID` | +| `kubectl exec -it NAME -- bash` | SSH. See [`runpodctl ssh`](/runpodctl/reference/runpodctl-ssh). | +| `kubectl scale deploy NAME --replicas N` | `runpodctl serverless update ENDPOINT_ID --workers-min N --workers-max N` | +| `kubectl scale nodepool NAME --replicas N` | Not needed. Capacity follows the workers. | +| `kubectl rollout restart deploy NAME` | `runpodctl pod restart POD_ID` | +| `kubectl delete deploy NAME` | `runpodctl serverless delete ENDPOINT_ID`, `runpodctl pod delete POD_ID` | +| `kubectl create secret generic` | `POST /v2/account/secrets` | +| `kubectl apply -f pvc.yaml` | `runpodctl network-volume create` | +| `kubectl create secret docker-registry` | `runpodctl registry create` | +| `kubectl cp` | `runpodctl send` and `runpodctl receive`. See [Transfer files](/pods/storage/transfer-files). | +| `cwic cwobject list`, creating a bucket | `runpodctl network-volume list`, `runpodctl network-volume create` | +| `sbatch`, `squeue`, `sinfo` on a SUNK login node | The same commands on the Slurm controller node | + +### API map + +| CoreWeave | Runpod REST API v2 (`https://api.runpod.io`) | +| --- | --- | +| Kubernetes API: create a Deployment and Service | `POST /v2/serverless`, or `POST /v2/pods` | +| Kubernetes API: patch replicas or an HPA | `PATCH /v2/serverless/ENDPOINT_ID` | +| Kubernetes API: create a Job | `POST https://api.runpod.ai/v2/ENDPOINT_ID/run` | +| Kubernetes API: read Pod logs | `GET /v2/pods/POD_ID/logs`, `GET /v2/serverless/ENDPOINT_ID/workers/WORKER_ID/logs` | +| Kubernetes API: create a PVC | `POST /v2/network-volumes`, `POST /v2/global-volumes` | +| Kubernetes API: create a Secret | `POST /v2/account/secrets` | +| `POST /v1alpha1/inference/deployments` | `POST /v2/serverless` | +| `DELETE /v1alpha1/inference/deployments/ID` | `DELETE /v2/serverless/ENDPOINT_ID` | +| CKS API: create a cluster | Not needed. For multi-node work, `POST /v2/clusters`. | +| Capacity Finder | `GET /v2/catalog/gpus?include=AVAILABILITY&product=POD` | +| PromQL and LogQL at `observe.coreweave.com` | `GET /v2/serverless/ENDPOINT_ID/metrics/requests`, `/metrics/cold-starts`, `/metrics/workers` | +| FOCUS billing export | `GET /v2/billing` | + +Both APIs take `Authorization: Bearer TOKEN`. + +### Environment variable map + +| On CKS | On Runpod | Notes | +| --- | --- | --- | +| `KUBERNETES_SERVICE_HOST` | None | No Kubernetes API in the container. | +| Pod name from the Downward API | `RUNPOD_POD_ID` | Set on Pods. | +| Zone or node labels from the Downward API | `RUNPOD_DC_ID` | Data center ID. | +| `nvidia.com/gpu` count you passed yourself | `RUNPOD_GPU_COUNT` | Set by Runpod. | +| Service port from the manifest | `PORT`, `PORT_HEALTH` | Load-balancing workers. You set both. | +| `AWS_ENDPOINT_URL_S3=http://cwlota.com` | Remove, or the Runpod S3-compatible endpoint | Prefer reading the mounted path. | +| Secret mounted as a variable | Variable set to `{{ RUNPOD_SECRET_name }}` | Resolved when the Pod or worker starts. | +| SUNK and Slurm variables | Unchanged, plus `PRIMARY_ADDR`, `NODE_RANK`, `NUM_NODES` | See [cluster configuration](/instant-clusters/configuration). | + +### Status map + +| Kubernetes | Runpod Pod | Runpod Serverless | +| --- | --- | --- | +| Pod `Pending` (unschedulable, pulling image) | `PROVISIONING`, `STARTING` | Worker `INITIALIZING`, or `THROTTLED` while the host's GPUs are busy | +| Pod `Running` and ready | `RUNNING` with `runtimeStatus: running` | Worker `RUNNING` or `IDLE` | +| Pod `Running` but not ready | `RUNNING` with `runtimeStatus: initializing` | Worker `UNHEALTHY`, or health check returning `204` | +| Pod `Succeeded` | `EXITED` | Job `COMPLETED` | +| Pod `Failed`, `CrashLoopBackOff` | `ERROR` | Job `FAILED` or `TIMED_OUT` | +| Deleted | `TERMINATED` | Endpoint deleted | +| Job active | Not applicable | Job `IN_QUEUE`, `IN_PROGRESS` | + +## Migrate with a coding agent + +A coding agent with the Runpod skills and MCP server can run this migration with you: it reads your manifests, proposes a mapping, and creates the Runpod resources after you approve. + +Install the tools for your agent: + + + + ```bash + # Claude Code (the plugin also connects the hosted MCP server) + claude plugin marketplace add runpod/runpod-plugins-official + claude plugin install runpod@runpod + ``` + + + ```bash + # Any other agent (Codex, Cursor, and others): skills, then the hosted MCP server + npx skills add runpod/runpod-plugins-official --skill '*' --agent AGENT_ID --global + npx @runpod/mcp-server@latest add --client CLIENT_ID + ``` + + + +The MCP server signs in with OAuth ("Sign in with Runpod"), so you don't need an API key to start. See [Agent skills](/get-started/agent-skills) and [MCP servers](/get-started/mcp-servers) for supported agents and options. + +Then give the agent this prompt from the root of the repository that holds your manifests: + +```text +Use the runpod-cloud-migrate skill to migrate this project from CoreWeave to Runpod. + +1. Inventory first, and change nothing. Read every Kubernetes manifest, Helm values file, + Terraform file, and CI script in this repo. If kubectl is configured for my CKS cluster, + also run read-only commands (kubectl get nodepool, deploy, statefulset, job, cronjob, hpa, + svc, ingress, pvc -A) to see what is running. List each workload with its image, + command, env, ports, GPU count, instance type, volumes, and replica range. +2. For each workload, propose a Runpod target (Serverless load-balancing endpoint, Serverless + queue endpoint, Pod, Instant Cluster, or SDK task) and say why. Flag anything with no + direct equivalent: DaemonSets, operators, CRDs, sidecars, VPC rules, bucket policies. +3. Show me the plan as a table, with a GPU type per workload, the data to copy and where it + lands, and a cost estimate from live Runpod pricing. Check GPU stock before proposing a + type. Wait for my approval before creating anything billable. +4. Migrate one workload end to end: create the secret references, volume, and endpoint or + Pod, copy only the data that workload needs, and verify with a real request from outside + Runpod. Show me the response and the latency. +5. Update the client code and CI config for that workload, on a branch. +6. Never delete, scale down, or modify CoreWeave resources, and never print secret values. + When I ask for the shutdown, list the exact commands and wait for my approval. +``` + +## Troubleshooting + +**A load-balancing worker shows as running but requests time out.** The health check never returned `200`. Expose the server's port in `ports`, set both `PORT` and `PORT_HEALTH` to that port, and set `HEALTH_CHECK_PATH` if your server's health route isn't `/ping`. A readiness probe that pointed at a different port than the Service is the usual cause after a manifest translation. + +**The model downloads again on every cold start.** The server is still writing to the PVC's old `mountPath`. On Serverless the volume is at `/runpod-volume`, so update the cache or download directory, or use [cached models](/serverless/endpoints/model-caching) for Hugging Face models. + +**The image fails to pull.** Check two things: the image must be `linux/amd64`, and a private image needs a registry credential (`registry` or `--registry-auth-id`). `imagePullSecrets` and registries that were only reachable inside the CKS cluster don't carry over. + +**Requests return `{"error":"not allowed for QB API"}`.** The URL style doesn't match the endpoint type. Load-balancing endpoints are called at `https://ENDPOINT_ID.api.runpod.ai/PATH`, and queue-based endpoints at `https://api.runpod.ai/v2/ENDPOINT_ID/run`. The type is fixed at creation, so recreate the endpoint if it's the wrong one. + +**Requests through a Pod's proxy URL fail after about 100 seconds.** That's the proxy timeout. Stream the response, move the workload to a Serverless endpoint, or expose a TCP port and connect to the Pod's public IP and mapped port. + +**The app can't reach another service by its Kubernetes name.** Names like `redis.default.svc.cluster.local` don't exist on Runpod. Use the other service's endpoint URL, or enable global networking on both Pods and use `POD_ID.runpod.internal`. Code that loads in-cluster Kubernetes credentials needs to be rewritten against the Runpod API. + +**The Pod or endpoint can't find capacity.** A network volume pins everything that mounts it to one data center, and a CUDA floor narrows the machines further. List more than one pool in `gpu.pools`, attach a network volume per data center, or pick another GPU type that fits the workload. + +**`aws s3` commands fail against the Runpod S3-compatible API.** Pass `--region` with the data center ID and `--endpoint-url` with that data center's URL, and use the Runpod S3 API key, not your Runpod API key. Keep the Runpod settings in a separate AWS CLI profile from the `cw` profile, because the `cw` profile sets CoreWeave's endpoint and addressing style. + +**`/status` returns 404 for a job that finished.** Results of `/run` jobs are kept for 30 minutes, and jobs are removed when their TTL expires. Have the handler write outputs to a volume or object store, or pass a `webhook`. + +**Slurm doesn't start on a new cluster.** Managed Slurm Clusters support only official Runpod PyTorch images. Recreate the cluster with one, or follow the [unmanaged Slurm guide](/instant-clusters/slurm) for a custom image. + +If something in this guide is wrong or missing, open an issue on the [Runpod docs repository](https://github.com/runpod/docs/issues). + +## Next steps + + + + Health checks, limits, and when to use a queue instead. + + + Tune workers, scaling, timeouts, and FlashBoot. + + + Storage tiers, multi-data-center setups, and the S3-compatible API. + + + Deploy and configure managed Slurm on Instant Clusters. + + diff --git a/migrate/fal.mdx b/migrate/fal.mdx new file mode 100644 index 000000000..fd3642353 --- /dev/null +++ b/migrate/fal.mdx @@ -0,0 +1,1392 @@ +--- +title: "Migrate from fal to Runpod" +sidebarTitle: "fal" +description: "Move fal Model API calls, fal Serverless apps, and fal Compute instances to Runpod Public Endpoints, Serverless, Pods, and Instant Clusters." +--- + +This guide is for teams that run production workloads on [fal](https://fal.ai/docs) and want to move some or all of them to Runpod. Calls to fal's hosted [Model APIs](https://fal.ai/docs/documentation/model-apis/overview) map to Runpod [Public Endpoints](/public-endpoints/overview), `fal.App` deployments on [fal Serverless](https://fal.ai/docs/documentation/serverless) map to Runpod [Serverless](/serverless/overview) endpoints, and [fal Compute](https://fal.ai/docs/documentation/compute) instances map to [Pods](/pods/overview) and [Instant Clusters](/instant-clusters). By the end, each workload runs on Runpod, your client code calls Runpod, and the fal resources that were billing are shut down. + + + A coding agent can do most of this migration for you. See [Migrate with a coding agent](#migrate-with-a-coding-agent). + + +## How fal maps to Runpod + +| fal concept | Runpod equivalent | Notes | +| --- | --- | --- | +| Model APIs (hosted model gallery, model IDs such as `fal-ai/flux/dev`) | [Public Endpoints](/public-endpoints/overview) | Hosted models you call with an API key. The catalogs overlap but aren't identical. See the [model map](#model-map). | +| Queue API (`https://queue.fal.run/MODEL_ID`) | `/run` and `/status/JOB_ID` on `https://api.runpod.ai/v2/ENDPOINT_ID` | Same submit, poll, and fetch pattern. On Runpod the result comes back in the status response. | +| Sync endpoint (`https://fal.run/MODEL_ID`), `subscribe` | `/runsync` | Waits 90 seconds by default. Raise it with `?wait=` (up to 300000 ms). | +| Streaming (`/stream`, SSE) | `/stream/JOB_ID` on queue-based endpoints, or your own SSE route on a load-balancing endpoint | Text Public Endpoints also expose an OpenAI-compatible route. | +| Realtime (`@fal.realtime`, WebSocket) | [Load-balancing endpoint](/serverless/load-balancing/overview) with your own WebSocket route | See [Realtime and WebSocket inference](#realtime-and-websocket-inference). | +| Workflow endpoints (`workflows/OWNER/NAME`) | Your own orchestration code | Chain Public Endpoint calls in your application. | +| fal Serverless (`fal.App`, `@fal.endpoint`) | [Serverless](/serverless/overview): the [Runpod SDK](/sdk/overview) (`runpod.App`) or a custom image with a handler | Queue-based endpoints for jobs, load-balancing endpoints for your own HTTP routes. | +| Runner | Worker | One container instance that serves requests. | +| `machine_type`, `num_gpus` | GPU pools and GPU types, `gpu_count` | See the [GPU map](#gpu-map). | +| `min_concurrency`, `max_concurrency`, `keep_alive` | Active (minimum) workers, max workers, idle timeout | See the [configuration field map](#configuration-field-map). | +| `requirements`, `ContainerImage` | SDK `dependencies` and `system_dependencies`, or your own Docker image and [template](/pods/templates/overview) | Images must be `linux/amd64`. | +| `/data` persistent storage | [Network volume](/storage/network-volumes), [global volume](/storage/globalvolume/overview), or [cached models](/serverless/endpoints/model-caching) | Network volumes mount at `/runpod-volume` on Serverless and `/workspace` on Pods, and live in one data center. | +| fal CDN (`fal.toolkit.Image`, `fal_client.upload_file`) | Your own S3-compatible bucket with the `rp_upload` helper, or base64 in the response | Public Endpoints return hosted output URLs that expire after 7 days. | +| `fal secrets set` | [Runpod secrets](/pods/templates/secrets) | You reference each secret explicitly in the environment. | +| `regions` | `datacenter` (SDK), `dataCenterIds` (REST), `--data-center-ids` (runpodctl) | GPU stock varies by data center. | +| fal Compute (`1xH100-SXM`, `8xH100-SXM`) | [Pods](/pods/overview), [Instant Clusters](/instant-clusters) | Pods are containers, not VMs. | +| Billing: per output (Model APIs), per second of runner lifetime (Serverless), per hour (Compute) | Per output (Public Endpoints), per second while workers run (Serverless), per second (Pods) | See [Serverless pricing](/serverless/pricing) and [Pod pricing](/pods/pricing). | +| `fal` CLI | [`runpodctl`](/runpodctl/overview) and the [`rp` CLI](/sdk/cli) | `rp flash` commands develop and deploy SDK apps. | +| `fal-client`, `@fal-ai/client` | `runpod` Python package, plain HTTP, [`@runpod/ai-sdk-provider`](/public-endpoints/ai-sdk) | The Runpod SDKs are built on REST API v2. | +| Platform APIs (`https://api.fal.ai/v1`) | [REST API v2](/api-reference-v2/overview) (`https://api.runpod.io/v2`) | Manage endpoints, templates, volumes, secrets, and billing. | +| `Authorization: Key FAL_KEY` | `Authorization: Bearer RUNPOD_API_KEY` | One API key for inference and management. | +| Teams and organizations | [Organizations](/organizations/orgs-overview) | Roles, groups, and billing per organization. | +| Dashboard logs and analytics, `fal runners logs` | Endpoint **Logs** and **Metrics** tabs, [worker logs](/serverless/development/logs) | [Job states and metrics](/serverless/endpoints/job-states) lists what's tracked. | +| fal MCP server | [Runpod MCP server](/get-started/mcp-servers) | OAuth sign-in. | + +## What's different + +### What works differently + +**The request body is wrapped.** fal takes model parameters at the top level of the JSON body. Runpod takes them inside an `input` object, and options such as `webhook` and `policy` sit beside `input`. + +**The result arrives with the status.** fal has separate status and result URLs. Runpod's `/status/JOB_ID` returns the status and, once the job completes, the `output`. + +**Failures have their own statuses.** fal reports `IN_QUEUE`, `IN_PROGRESS`, and `COMPLETED`, and a failed request is `COMPLETED` with an `error` field. Runpod adds `FAILED`, `CANCELLED`, and `TIMED_OUT`. Update any code that treats `COMPLETED` as the only terminal state. + +**Results don't stay around as long.** fal stores request payloads for 90 days by default. Runpod keeps the result of a `/run` job for 30 minutes after completion and a `/runsync` result for 1 minute. Fetch results promptly or use a webhook, and store what you need. + +**Output URLs expire.** Public Endpoint output URLs (`image_url`, `video_url`, `audio_url`, `result`) expire after 7 days. If you relied on fal CDN URLs as long-lived links, copy outputs to your own storage. + +**Secrets are opt-in per endpoint.** By default fal injects every secret in the environment into a runner. On Runpod you map each secret into an environment variable yourself, with `runpod.Secret("NAME")` in the SDK or `{{ RUNPOD_SECRET_NAME }}` in a template or endpoint. + +**Model weights aren't cached for you.** fal sets `HF_HOME` to `/data/.cache/huggingface` on every runner. On Runpod, weights download to the worker's container disk on each cold start unless you use [cached models](/serverless/endpoints/model-caching), a volume, or bake them into the image. + +**Storage is attached, not ambient.** `/data` is mounted on every fal runner in your account. A Runpod network volume is created with a size, lives in one data center, and is attached to specific endpoints or Pods. Attaching one limits those workers to that data center. + +**Auth is always a Runpod API key.** fal apps can be `private`, `public`, or `shared`. Runpod Serverless endpoints check a Runpod API key on every request. For unauthenticated access, put your own backend in front of the endpoint. + +**Idle timeout defaults are shorter.** fal's `keep_alive` defaults to 60 seconds. The Runpod SDK's `idle_timeout` also defaults to 60, but the console default is 5 seconds and the REST v2 default is 10. Set it explicitly. + +**Execution timeouts are shorter by default.** fal's `request_timeout` defaults to 3600 seconds. Runpod's execution timeout defaults to 600 seconds in the console and 300000 ms in REST v2. The SDK's `execution_timeout_ms` defaults to no limit. + +**You build images yourself for the custom-image path.** `fal deploy` builds containers remotely. With the Runpod SDK, `rp flash deploy` also packages your code without Docker. With a custom worker, you build and push a `linux/amd64` image, or let the [GitHub integration](/serverless/workers/github-integration) build it. + +### What has no direct equivalent + +| On fal | On Runpod | Workaround | +| --- | --- | --- | +| Workflow endpoints that chain models in one call | Not a hosted feature | Chain calls in your own code. See the [text-to-video pipeline tutorial](/tutorials/public-endpoints/text-to-video-pipeline). | +| `fal_client.upload_file` and `fal.storage.upload` for input files | Public Endpoint model pages take input media as URLs | Host inputs in your own bucket and pass a URL the endpoint can fetch. | +| Automatic CDN upload of `Image` and `File` outputs from your own app | Your handler uploads to your bucket with `rp_upload`, or returns base64 | See [Step 4](#step-4-move-your-images-and-code). | +| Signed webhooks (`X-Fal-Webhook-Signature`) | The Runpod webhook docs describe retries, not a signature | Put an unguessable token in the webhook URL and confirm the job with `/status/JOB_ID`. | +| `fal.realtime` client and msgpack protocol | Raw WebSockets on a load-balancing endpoint | Define your own message format. | +| `shared` auth (callers pay with their own key) | Each account calls its own endpoints | Publish the worker to the [Hub](/hub/overview) so others deploy it in their account. | +| `concurrency_buffer`, `concurrency_buffer_perc` | Autoscaling by queue delay or request count | Keep active workers for headroom and tune the scaler threshold. | +| 90-day request history and per-request payload APIs | Results expire in minutes | Persist inputs and outputs from your webhook receiver. | +| Automatic retries of failed runner attempts (up to 10) | `/retry/JOB_ID` requeues a failed or timed-out job | Call `/retry` from your client or webhook receiver. | + +Going the other way, some things exist on Runpod that you may not have used on fal: text models with an OpenAI-compatible route on Public Endpoints, `@app.task` for run-to-completion jobs on a dedicated Pod, per-second Pod billing, and Community Cloud or Secure Cloud placement for Pods. + +## Requirements + +- A [Runpod account](/accounts-billing/manage-accounts) with credits. +- A [Runpod API key](/get-started/credentials). +- Access to your fal account or team, with the `fal` CLI installed and logged in (`fal auth login`). +- Python 3.10 to 3.14 for the Runpod SDK. +- Docker, if you'll deploy a custom image. +- An S3-compatible bucket, if your app returns files and you don't want to return base64. + +## Step 1: Take inventory of what you run on fal + +List everything in each fal account and team before you move anything. If you use teams, switch with `fal account set TEAM_NAME` and repeat. + +```bash +# Who you're logged in as, and which accounts you can switch to +fal auth whoami +fal account list + +# Environments (main, staging, and so on) +fal environments list + +# Deployed apps, per environment +fal apps list --json +fal apps list --env staging --json + +# Runners for one app, to see real concurrency +fal apps runners APP_NAME + +# Secret names (values are never shown) +fal secrets list +fal secrets list --env staging + +# Files on /data +fal files list +fal files list models/ + +# API keys +fal keys list +``` + +fal Compute instances don't appear in the app list. List them in the [Compute dashboard](https://fal.ai/dashboard/compute) or through the Platform API. + +```bash +curl -s "https://api.fal.ai/v1/compute/instances" \ + -H "Authorization: Key $FAL_KEY" +``` + +For each `fal.App`, record these values from the source file or `pyproject.toml`: `machine_type`, `num_gpus`, `min_concurrency`, `max_concurrency`, `keep_alive`, `max_multiplexing`, `request_timeout`, `startup_timeout`, `regions`, `requirements` or the `ContainerImage` Dockerfile, the secrets it reads, every `/data` path it touches, and every `@fal.endpoint` and `@fal.realtime` path. Scaling values changed with `fal apps scale` override the code, so trust `fal apps list --json` over the source for those. + +Then find the code that calls fal. This command finds SDK imports, CLI usage, hostnames, and environment variables. + +```bash +grep -rnE "fal_client|@fal-ai/|@ai-sdk/fal|import fal\b|from fal[. ]|fal\.run|queue\.fal\.run|fal\.media|api\.fal\.ai|FAL_KEY|FAL_API_KEY|fal (deploy|run|apps|secrets|files) |\[tool\.fal" \ + --include="*.py" --include="*.ts" --include="*.tsx" --include="*.js" --include="*.mjs" \ + --include="*.toml" --include="*.yml" --include="*.yaml" --include="*.sh" --include="*.env*" \ + --exclude-dir=node_modules --exclude-dir=.venv . +``` + +Write down every model ID you call (strings such as `fal-ai/flux/dev` and `workflows/OWNER/NAME`), whether each call uses `subscribe`, `submit`, `run`, `stream`, or `realtime`, and whether it uses webhooks, `fal_client.upload_file`, or stored fal CDN URLs. + +## Step 2: Set up Runpod + +Create an [API key](/get-started/credentials) in the console, then install the tools you need. + +```bash +# runpodctl: manage endpoints, templates, Pods, and volumes +curl -fsSL https://cli.runpod.net | sudo bash + +# Runpod SDK and the rp CLI (Python 3.10 to 3.14) +pip install runpod + +# Log in once. Credentials are saved to ~/.runpod/config.toml +rp login + +# For scripts and CI +export RUNPOD_API_KEY="YOUR_API_KEY" +``` + +On macOS you can also install `runpodctl` with `brew install runpod/runpodctl/runpodctl`. For agent-driven work, connect the [Runpod MCP server](/get-started/mcp-servers). + +## Step 3: Choose where each workload lands + +| What you run on fal | Where it lands on Runpod | +| --- | --- | +| Calls to a hosted model that Runpod also hosts | [Public Endpoint](/public-endpoints/overview) | +| Calls to a hosted model that isn't a Public Endpoint | Deploy the open model yourself on Serverless, or check the [Hub](/hub/overview) for a ready-made worker. Closed models stay with a provider that hosts them. | +| `fal.App` called through the queue (`subscribe`, `submit`, webhooks) | Serverless queue-based endpoint: `@app.queue` in the Runpod SDK, or a custom image with a handler | +| `fal.App` with several HTTP routes, SSE streaming, or an existing HTTP server (`exposed_port`) | Serverless load-balancing endpoint: `@app.api` in the Runpod SDK, or a custom image that serves HTTP | +| `@fal.realtime` or `is_websocket=True` endpoints | Serverless load-balancing endpoint with a WebSocket route | +| ComfyUI workflow deployed as a `fal.App` | The [ComfyUI worker](/tutorials/serverless/comfyui) from the Hub, or a customized `runpod/worker-comfyui` image | +| `@fal.function` one-off jobs, training runs | `@app.task` in the Runpod SDK (one Pod per call), or a Pod | +| fal Compute `1xH100-SXM` | Pod | +| fal Compute `8xH100-SXM`, single node | Pod with multiple GPUs | +| fal Compute multi-node in one sector (InfiniBand) | [Instant Cluster](/instant-clusters) | + +### Model map + +These fal model IDs have a counterpart in the Runpod [model reference](/public-endpoints/reference). The Runpod endpoint ID is the path segment in `https://api.runpod.ai/v2/ENDPOINT_ID/runsync`. Both catalogs change, so check the model reference and the [Hub playground](https://console.runpod.io/hub?tabSelected=public_endpoints) before you commit to a mapping. + +| fal model ID | Runpod Public Endpoint ID | Runpod model page | +| --- | --- | --- | +| `fal-ai/flux/dev` | `black-forest-labs-flux-1-dev` | [Flux Dev](/public-endpoints/models/flux-dev) | +| `fal-ai/flux/schnell` | `black-forest-labs-flux-1-schnell` | [Flux Schnell](/public-endpoints/models/flux-schnell) | +| `fal-ai/flux-kontext/dev` | `black-forest-labs-flux-1-kontext-dev` | [Flux Kontext Dev](/public-endpoints/models/flux-kontext-dev) | +| `fal-ai/qwen-image` | `qwen-image-t2i` (with LoRAs: `qwen-image-t2i-lora`) | [Qwen Image](/public-endpoints/models/qwen-image), [Qwen Image LoRA](/public-endpoints/models/qwen-image-lora) | +| `fal-ai/qwen-image-edit` | `qwen-image-edit` | [Qwen Image Edit](/public-endpoints/models/qwen-image-edit) | +| `fal-ai/qwen-image-edit-2511` | `qwen-image-edit-2511` | [Qwen Image Edit 2511](/public-endpoints/models/qwen-image-edit-2511) | +| `fal-ai/qwen-image-edit-2511/lora` | `qwen-image-edit-2511-lora` | [Qwen Image Edit 2511 LoRA](/public-endpoints/models/qwen-image-edit-2511-lora) | +| `fal-ai/bytedance/seedream/v4/text-to-image` | `seedream-v4-t2i` | [Seedream 4.0 T2I](/public-endpoints/models/seedream-4-t2i) | +| `fal-ai/bytedance/seedream/v4/edit` | `seedream-v4-edit` | [Seedream 4.0 Edit](/public-endpoints/models/seedream-4-edit) | +| `fal-ai/z-image/turbo` | `z-image-turbo` | [Z-Image Turbo](/public-endpoints/models/z-image-turbo) | +| `fal-ai/nano-banana-2/edit` | `google-nano-banana-2-edit` | [Nano Banana 2 Edit](/public-endpoints/models/nano-banana-2-edit) | +| `fal-ai/nano-banana-pro/edit` | `nano-banana-pro-edit` | [Nano Banana Pro Edit](/public-endpoints/models/nano-banana-pro-edit) | +| `wan/v2.6/text-to-image` | `wan-2-6-t2i` | [WAN 2.6 T2I](/public-endpoints/models/wan-2-6-t2i) | +| `wan/v2.6/text-to-video` | `wan-2-6-t2v` | [WAN 2.6 T2V](/public-endpoints/models/wan-2-6-t2v) | +| `wan/v2.6/image-to-video` | `wan-2-6-i2v` | [WAN 2.6 I2V](/public-endpoints/models/wan-2-6-i2v) | +| `fal-ai/wan-25-preview/image-to-video` | `wan-2-5` | [WAN 2.5](/public-endpoints/models/wan-2-5) | +| `fal-ai/wan/v2.2-a14b/image-to-video` | `wan-2-2-i2v-720` | [WAN 2.2 I2V](/public-endpoints/models/wan-2-2-i2v) | +| `fal-ai/wan/v2.2-a14b/image-to-video/lora` | `wan-2-2-t2v-720-lora` | [WAN 2.2 I2V LoRA](/public-endpoints/models/wan-2-2-i2v-lora) | +| `fal-ai/kling-video/v2.6/standard/motion-control` | `kling-v2-6-std-motion-control` | [Kling v2.6 Motion Control](/public-endpoints/models/kling-v2-6-motion-control) | +| `bytedance/seedance-2.5/text-to-video` | `tdq07exynr7zux` | [Seedance 2.5 T2V](/public-endpoints/models/seedance-2-5-t2v) | +| `bytedance/seedance-2.5/image-to-video` | `16jsvx5ncmm24x` | [Seedance 2.5 I2V](/public-endpoints/models/seedance-2-5-i2v) | +| `fal-ai/vidu/q3/text-to-video` | `vidu-q3-t2v` | [Vidu Q3 T2V](/public-endpoints/models/vidu-q3-t2v) | +| `fal-ai/vidu/q3/image-to-video` | `vidu-q3-i2v` | [Vidu Q3 I2V](/public-endpoints/models/vidu-q3-i2v) | +| `fal-ai/infinitalk` | `infinitetalk` | [InfiniteTalk](/public-endpoints/models/infinitetalk) | +| `fal-ai/minimax/speech-02-hd` | `minimax-speech-02-hd` | [Minimax Speech 02 HD](/public-endpoints/models/minimax-speech) | +| `fal-ai/chatterbox/text-to-speech/turbo` | `chatterbox-turbo` | [Chatterbox Turbo](/public-endpoints/models/chatterbox-turbo) | + + + A matching name doesn't mean matching output. The two platforms can run different versions, tiers, default steps, and guidance values of the same model family, and fal lists both pro and standard tiers for some Kling endpoints. Run your own prompts through both before you switch traffic, and compare parameters on each Runpod model page. + + +Runpod also hosts text models (for example `qwen3-32b-awq` and `gpt-oss-120b`) with an OpenAI-compatible route at `https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1`. For fal models that aren't in the table, such as models from the FLUX Pro, Veo, Ideogram, and ElevenLabs families, or other Kling and Sora variants, check the model reference for a current listing, then plan to keep the call on a provider that hosts the model or self-host an open alternative on Serverless. + +### GPU map + +fal machine types name a GPU class. Runpod Serverless selects hardware by GPU pool, and Pods select by GPU type. Pick by the VRAM your model needs, then check live stock with `runpodctl gpu list` or `GET /v2/catalog/gpus`, because availability varies by data center. + +| fal `machine_type` | VRAM on fal | Runpod Serverless pool | Runpod GPU types in the pool | +| --- | --- | --- | --- | +| `GPU-A100` | 40 GB | `AMPERE_48` (48 GB) or `AMPERE_80` (80 GB) | A6000, A40, or A100 80GB | +| `GPU-L40` | 48 GB | `ADA_48_PRO` | L40, L40S, RTX 6000 Ada | +| `GPU-H100` | 80 GB | `ADA_80_PRO` | H100 PCIe, H100 80GB HBM3, H100 NVL | +| `GPU-RTXPRO6000` | 96 GB | `BLACKWELL_96` | RTX PRO 6000 Blackwell | +| `GPU-H200` | 141 GB | `HOPPER_141` | H200 | +| `GPU-B200` | 192 GB | `BLACKWELL_180` | B200 (listed at 180 GB) | +| `XS`, `S`, `M`, `L`, `XL` (CPU) | n/a | CPU instance IDs such as `cpu3c-2-4` | See [CPU types](/references/cpu-types) | + +fal's `GPU-A100` is the 40 GB card. Runpod's A100 pool is the 80 GB card, so a workload sized for 40 GB also fits the 48 GB pool. A list such as `machine_type = ["GPU-H100", "GPU-A100"]` maps to a list of pools, for example `gpu=["H100", "A100"]` in the SDK. See [GPU types](/references/gpu-types) for every type and pool. + +## Step 4: Move your images and code + +If you only call hosted models, skip to [Step 7](#step-7-update-the-code-that-calls-fal). + +### The fal app you're starting from + +This is the shape of a typical `fal.App`: `setup()` loads the model once per runner, and one endpoint takes a pydantic input and returns an `Image` that fal uploads to its CDN. + +```python fal_app.py +import fal +from pydantic import BaseModel, Field +from fal.toolkit import Image + + +class Input(BaseModel): + prompt: str = Field(description="The prompt to generate an image from") + + +class Output(BaseModel): + image: Image + + +class MyApp(fal.App): + app_name = "my-demo-app" + machine_type = "GPU-H100" + keep_alive = 300 + min_concurrency = 0 + max_concurrency = 5 + requirements = [ + "diffusers[torch]==0.32.2", + "transformers[sentencepiece]==4.51.0", + "accelerate==1.6.0", + ] + + def setup(self): + import torch + from diffusers import StableDiffusionXLPipeline + + self.pipe = StableDiffusionXLPipeline.from_pretrained( + "stabilityai/stable-diffusion-xl-base-1.0", + torch_dtype=torch.float16, + variant="fp16", + use_safetensors=True, + ).to("cuda") + + @fal.endpoint("/") + def run(self, request: Input) -> Output: + result = self.pipe(request.prompt) + return Output(image=Image.from_pil(result.images[0])) +``` + +Callers get `{"image": {"url": "https://...fal.media/..."}}`. On Runpod you choose between two paths: the Runpod SDK, which is closest to the fal workflow, or a custom image with a handler, which gives you full control of the container. + +### Option A: The Runpod SDK + + + + Use `@app.queue` when callers use `subscribe`, `submit`, or webhooks today. It creates a queue-based endpoint with `/run`, `/runsync`, and `/status`. + + ```python main.py + import runpod + + app = runpod.App("my-demo-app") + + sdxl = runpod.Model("stabilityai/stable-diffusion-xl-base-1.0") + + _PIPE = None + + + @app.queue( + gpu="H100", # machine_type = "GPU-H100" + workers=(0, 5), # (min_concurrency, max_concurrency) + idle_timeout=300, # keep_alive + model=sdxl, # staged on the host before the worker starts + dependencies=[ + "diffusers==0.32.2", + "transformers[sentencepiece]==4.51.0", + "accelerate==1.6.0", + ], + ) + def generate(prompt: str): + global _PIPE + import base64 + import io + + import torch + from diffusers import StableDiffusionXLPipeline + + # Deployed workers import this module once, so the load runs once per worker. + if _PIPE is None: + _PIPE = StableDiffusionXLPipeline.from_pretrained( + "stabilityai/stable-diffusion-xl-base-1.0", + torch_dtype=torch.float16, + variant="fp16", + use_safetensors=True, + ).to("cuda") + + image = _PIPE(prompt).images[0] + buffer = io.BytesIO() + image.save(buffer, format="JPEG", quality=95) + return { + "image_b64": base64.b64encode(buffer.getvalue()).decode("utf-8"), + "content_type": "image/jpeg", + "width": image.width, + "height": image.height, + } + + + @runpod.local_entrypoint + def main(): + result = generate.remote("A cyberpunk cityscape at night with neon lights") + print(result["width"], result["height"], len(result["image_b64"])) + ``` + + The function's keyword arguments become the job `input`, so a caller sends `{"input": {"prompt": "..."}}`. The pydantic `Input` model becomes a typed function signature. `torch` is already installed on the built-in GPU images, so it isn't listed in `dependencies`. Packages are imported inside the function because top-level imports run on your machine. + + + + Use `@app.api` when you want the same class shape as `fal.App`: a setup hook and one method per route. It creates a load-balancing endpoint, so requests go straight to a worker with no queue. + + ```python main.py + import runpod + + app = runpod.App("my-demo-app") + + sdxl = runpod.Model("stabilityai/stable-diffusion-xl-base-1.0") + + + @app.api( + gpu="H100", + workers=(0, 5), + idle_timeout=300, + model=sdxl, + dependencies=[ + "diffusers==0.32.2", + "transformers[sentencepiece]==4.51.0", + "accelerate==1.6.0", + ], + ) + class MyApp: + @runpod.init + def setup(self): + # Runs once when a worker starts, like fal's setup() + import torch + from diffusers import StableDiffusionXLPipeline + + self.pipe = StableDiffusionXLPipeline.from_pretrained( + "stabilityai/stable-diffusion-xl-base-1.0", + torch_dtype=torch.float16, + variant="fp16", + use_safetensors=True, + ).to("cuda") + + @runpod.post("/generate") + async def generate(self, body: dict): + import base64 + import io + + image = self.pipe(body["prompt"]).images[0] + buffer = io.BytesIO() + image.save(buffer, format="JPEG", quality=95) + return { + "image_b64": base64.b64encode(buffer.getvalue()).decode("utf-8"), + "content_type": "image/jpeg", + } + + @runpod.get("/health") + async def health(self): + return {"status": "ok"} + + + @runpod.local_entrypoint + def main(): + result = MyApp.post("/generate", {"prompt": "A cyberpunk cityscape at night"}) + print(len(result["image_b64"])) + ``` + + Callers post the JSON body itself to `https://ENDPOINT_ID.api.runpod.ai/generate`, with no `input` wrapper. Load-balancing endpoints have a 5.5 minute processing timeout per request and a 30 MB payload limit, and they don't queue or retry requests. + + + +Both versions return the image as base64 because nothing on Runpod uploads your outputs for you. Job requests and results are JSON with a size limit (10 MB for `/run`, 20 MB for `/runsync`), so return base64 only for small outputs. For video, audio, or large images, upload to your own bucket and return the URL, as Option B does. + +`model=sdxl` replaces fal's automatic Hugging Face cache on `/data`. Runpod stages the weights on the host before the worker starts, and libraries that use the standard Hugging Face cache find them. Gated models need an `HF_TOKEN` on the same resource, passed as `env={"HF_TOKEN": runpod.Secret("hf-token")}`. + +### Option B: A custom image with a handler + +Use this path when you already have a Dockerfile (a `ContainerImage` on fal), need system packages or a pinned CUDA stack, or want the response to carry a file URL. + +The handler loads the model at module level, which runs once per worker, and uses the `rp_upload` helper to put the image in your S3-compatible bucket. + +```python handler.py +import runpod +import torch +from diffusers import StableDiffusionXLPipeline +from runpod.serverless.utils import rp_upload + +# Module level: runs once when the worker starts, like fal's setup() +pipe = StableDiffusionXLPipeline.from_pretrained( + "stabilityai/stable-diffusion-xl-base-1.0", + torch_dtype=torch.float16, + variant="fp16", + use_safetensors=True, +).to("cuda") + + +def handler(job): + job_input = job["input"] + prompt = job_input.get("prompt") + if not prompt: + return {"error": "Input is missing the 'prompt' key."} + + image = pipe(prompt).images[0] + image_path = f"/tmp/{job['id']}.png" + image.save(image_path) + + # Reads BUCKET_ENDPOINT_URL, BUCKET_ACCESS_KEY_ID, and BUCKET_SECRET_ACCESS_KEY + image_url = rp_upload.upload_image(job["id"], image_path) + + return { + "image": { + "url": image_url, + "content_type": "image/png", + "width": image.width, + "height": image.height, + } + } + + +runpod.serverless.start({"handler": handler}) +``` + +The return value keeps fal's `image.url` shape, so callers change `result["image"]["url"]` to `result["output"]["image"]["url"]`. Input validation that pydantic did on fal is now your code. Returning a dictionary with an `error` key marks the job as failed. + +The Dockerfile installs the handler's dependencies on a CUDA base image and starts the handler. + +```dockerfile Dockerfile +FROM runpod/pytorch:1.0.2-cu1281-torch280-ubuntu2404 + +WORKDIR / +RUN pip install --no-cache-dir runpod "diffusers==0.32.2" \ + "transformers[sentencepiece]==4.51.0" "accelerate==1.6.0" + +COPY handler.py /handler.py + +CMD ["python", "-u", "/handler.py"] +``` + +If you used `ContainerImage.from_dockerfile_str` on fal, start from that Dockerfile and add three things back: `pip install runpod`, a `COPY` of your handler, and a `CMD` that runs it. Build for `linux/amd64` and push to a registry Runpod can pull from. + +```bash +docker build --platform linux/amd64 -t DOCKER_USERNAME/my-demo-app:v1 . +docker push DOCKER_USERNAME/my-demo-app:v1 +``` + +For a private registry, save credentials with `runpodctl registry create --name "docker-hub" --username "DOCKER_USERNAME"` and pass the returned ID as `--registry-auth-id` when you create the template. This replaces the `registries` parameter on fal's `ContainerImage`. + + + This handler downloads the SDXL weights to the container disk on every cold start. Add a cached model in [Step 6](#step-6-deploy-on-runpod), bake the weights into the image, or use a network volume. See [Step 5](#step-5-move-your-data-and-models). + + +### Existing HTTP servers + +If your fal app runs in direct server mode (`exposed_port` in `pyproject.toml`), the same image runs on a Runpod [load-balancing endpoint](/serverless/load-balancing/overview). Your server must answer `GET /ping` with `200` when it's ready (`204` while initializing) and listen on the port in the `PORT` environment variable, which defaults to `80`. If you set `PORT`, set `PORT_HEALTH` to the same value and expose that port on the endpoint. To use a different health path, set `HEALTH_CHECK_PATH`. + +### Streaming endpoints + +A fal endpoint at `/stream` that returns a `StreamingResponse` has two Runpod forms. On a queue-based endpoint, write a generator handler (or a generator function under `@app.queue`) that yields chunks, and have clients read them from `/stream/JOB_ID`. On a load-balancing endpoint, keep your FastAPI `StreamingResponse` as it is. See [handler functions](/serverless/workers/handler-functions) for streaming handlers and progress updates. + +### Realtime and WebSocket inference + +Runpod supports WebSocket connections on load-balancing endpoints. Queue-based endpoints don't accept custom routes, so a WebSocket route there returns `not allowed for QB API`. To move a `@fal.realtime` or `is_websocket=True` endpoint: + +1. Serve the WebSocket route from your own HTTP server (for example FastAPI's `@app.websocket("/ws")`) alongside `GET /ping`, in a custom image deployed as a load-balancing endpoint. +2. Replace fal's realtime protocol with your own messages. `@fal.realtime` uses fal's binary msgpack protocol and the `fal_client.realtime()` and `fal.realtime.connect()` clients. On Runpod the client is any WebSocket library, and the frames are whatever your server sends. +3. Connect to `wss://ENDPOINT_ID.api.runpod.ai/ws` and send `Authorization: Bearer YOUR_API_KEY` as a header on the connection request. +4. Raise the client's connection timeout. The default of about 5 seconds isn't enough when a worker has to start. The Runpod docs recommend `open_timeout=60`. + +```python client.py +import asyncio +import json + +import websockets + + +async def main(): + url = "wss://ENDPOINT_ID.api.runpod.ai/ws" + headers = [("Authorization", "Bearer YOUR_API_KEY")] + + async with websockets.connect( + url, additional_headers=headers, open_timeout=60.0 + ) as ws: + await ws.send(json.dumps({"prompt": "a sunset over mountains"})) + while True: + message = json.loads(await ws.recv()) + if message.get("done"): + break + print(message) + + +asyncio.run(main()) +``` + +The load-balancing limits apply: a request waits up to 2 minutes for a worker, processing is limited to 5.5 minutes per request, and payloads are limited to 30 MB. Test how long your sessions stay open before you rely on them, and keep at least one active worker for interactive use so the first connection doesn't wait on a cold start. A browser can't hold your API key safely, so keep the proxy pattern you used with fal (`proxyUrl`) and connect from your backend. See [Build a load-balancing worker](/serverless/load-balancing/build-a-worker#optional-websocket-support) and the [worker-lb-websocket example](https://github.com/runpod-workers/worker-lb-websocket). + +fal's hosted realtime models (`fal-ai/fast-lcm-diffusion`, `fal-ai/fast-turbo-diffusion`) aren't in the Runpod model reference. To keep that feature, self-host the model behind a WebSocket route. + +### ComfyUI workflows + +fal's ComfyUI pattern runs the ComfyUI server in a custom container inside a `fal.App` and posts an API-format workflow to it. On Runpod, the [ComfyUI worker](/tutorials/serverless/comfyui) does the same job as a ready-made queue-based endpoint. Deploy it from the [Hub listing](https://console.runpod.io/hub/runpod-workers/worker-comfyui), or use one of the `runpod/worker-comfyui` images (`-base`, `-flux1-schnell`, `-flux1-dev`, `-sdxl`, `-sd3`). + +Your exported `workflow_api.json` goes in the request as `{"input": {"workflow": { ... }}}`, and the result returns images as base64 in `output.images[0].data`. The Hub listing ships with FLUX.1-dev-fp8 and works only with that model. For other checkpoints, LoRAs, or custom nodes, build a customized image by following the worker's [customization guide](https://github.com/runpod-workers/worker-comfyui/blob/main/docs/customization.md), the same way you packaged nodes and models into the fal container. + +### Runtime environment variables + +Code that reads fal's injected variables needs new names. See the [environment variable map](#environment-variable-map). `FAL_KEY` deserves attention: fal injects it so an app can call other fal models, and those calls are billed to you. If your worker calls Runpod Public Endpoints or other endpoints, pass a Runpod API key to it as a secret. + +## Step 5: Move your data and models + +Decide what each `/data` path was for, because the Runpod target differs. + +| What's on `/data` | Runpod target | How | +| --- | --- | --- | +| Hugging Face cache (`/data/.cache/huggingface`) | [Cached models](/serverless/endpoints/model-caching) | Don't copy it. Set `model=runpod.Model("ORG/NAME")` in the SDK or `--model-reference` on the endpoint. Weights appear under `/runpod-volume/huggingface-cache/hub/`. | +| Your own weights, LoRAs, and checkpoints | [Network volume](/storage/network-volumes) | Copy once, attach to the endpoint, read from `/runpod-volume`. | +| Files shared across regions | [Global volume](/storage/globalvolume/overview) (beta) | Not tied to a data center. GPU compute only. | +| Scratch files and per-request temp output | Container disk | Ephemeral. Size it with `container_disk_gb` or the template's container disk. | +| Datasets and checkpoints from Compute instances | Network volume, or a Pod's volume disk at `/workspace` | Volume disk is deleted with the Pod. A network volume outlives it. | + +Create the network volume in the data center where you want workers to run. Check GPU stock there first, because an attached volume keeps the endpoint in that data center. + +```bash +runpodctl network-volume create --name "models" --size 100 --data-center-id "US-KS-2" +``` + +Download your files from fal to a machine with enough disk, then copy them to the volume through the [S3-compatible API](/storage/s3-api). Create an S3 API key in the console first and configure it with `aws configure`. The bucket name is the network volume ID. + +```bash +# From fal /data to local disk +fal files list models/ +fal files download models/my-lora.safetensors ./export/my-lora.safetensors + +# From local disk to the Runpod network volume +aws s3 cp --region US-KS-2 \ + --endpoint-url https://s3api-us-ks-2.runpod.io/ \ + ./export/my-lora.safetensors \ + s3://NETWORK_VOLUME_ID/models/my-lora.safetensors + +# Check it arrived +aws s3 ls --region US-KS-2 \ + --endpoint-url https://s3api-us-ks-2.runpod.io/ \ + s3://NETWORK_VOLUME_ID/models/ +``` + +The S3 API is available in the data centers listed on the [S3 API page](/storage/s3-api). For large transfers, or a data center that isn't listed, start a Pod with the volume attached and pull the files from their original source there (for example with `hf download` or `wget`), which avoids routing everything through your machine. `aws s3 sync` works for basic cases but can have trouble with very large file counts. + + + Loading weights directly from the network mount on every cold start is slower than loading from local disk. The [volume cache](/serverless/development/volume-cache) pattern copies them to the worker's disk first. + + +Files on the fal CDN (generated outputs and `fal_client.upload_file` uploads) are public URLs with a retention you may have configured. If your database stores `fal.media` URLs, download those files to your own storage before you close the account. + +Recreate secrets by name. Values can't be read back from fal, so take them from your secret manager. + +```bash +rp secret add hf-token +rp secret list +``` + +`rp secret add` prompts for the value, which keeps it out of your shell history. Secret names that start with `RUNPOD` are rejected. + +## Step 6: Deploy on Runpod + +### Public Endpoints + +There's nothing to deploy. Public Endpoints are live for every account, and your API key works on all of them. Go to [Step 7](#step-7-update-the-code-that-calls-fal). + +### Serverless with the Runpod SDK + +`rp flash dev` plays the role of `fal run`: it creates temporary endpoints, runs your `@runpod.local_entrypoint`, streams worker logs, and deletes the endpoints when you exit. `rp flash deploy` plays the role of `fal deploy`. + + + + ```bash + fal run fal_app.py::MyApp + fal deploy fal_app.py::MyApp --env staging + fal apps scale my-demo-app --min-concurrency 1 --max-concurrency 10 --keep-alive 300 + ``` + + + + ```bash + rp flash dev main.py + rp flash deploy --env staging + rp flash env get staging + ``` + + + +Scaling lives in code on Runpod (`workers=(1, 10)`, `idle_timeout=300`), so change the decorator and deploy again. `rp flash deploy` prints a link to each endpoint. To change scaling on a live endpoint without a deploy, use `runpodctl serverless update ENDPOINT_ID --workers-min 1 --workers-max 10 --idle-timeout 300`. + +### Serverless with a custom image + +Create a Serverless template for the image, then an endpoint from the template. This is the queue-based deployment of the handler from Option B. + +```bash +runpodctl template create --name "my-demo-app" --serverless \ + --image "DOCKER_USERNAME/my-demo-app:v1" \ + --container-disk-in-gb 30 \ + --env '{"BUCKET_ENDPOINT_URL":"https://BUCKET_NAME.s3.us-west-2.amazonaws.com","BUCKET_ACCESS_KEY_ID":"{{ RUNPOD_SECRET_bucket_access_key_id }}","BUCKET_SECRET_ACCESS_KEY":"{{ RUNPOD_SECRET_bucket_secret_access_key }}"}' + +runpodctl serverless create --name "my-demo-app" \ + --template-id TEMPLATE_ID \ + --gpu-id ADA_80_PRO \ + --workers-min 0 --workers-max 5 \ + --idle-timeout 300 \ + --execution-timeout 600 \ + --min-cuda-version 12.8 \ + --model-reference https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0:main +``` + +`--gpu-id` takes a pool ID or a GPU type ID. `--model-reference` attaches a cached model so workers don't download the weights. To attach the network volume from Step 5, add `--network-volume-id NETWORK_VOLUME_ID`. Each Serverless template binds to one endpoint, so create a template per endpoint. Check `runpodctl serverless create --help` for the full flag list. + +The same endpoint through REST API v2, without a separate template: + +```bash +curl -X POST "https://api.runpod.io/v2/serverless" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "my-demo-app", + "image": "DOCKER_USERNAME/my-demo-app:v1", + "type": "QUEUE", + "gpu": {"pools": ["ADA_80_PRO"], "count": 1, "minCudaVersion": "12.8"}, + "workers": {"min": 0, "max": 5, "idleTimeout": 300}, + "scaling": {"type": "QUEUE_DELAY", "queueDelay": 4}, + "timeout": 600000, + "flashboot": "FLASHBOOT", + "disk": 30, + "env": { + "BUCKET_ENDPOINT_URL": "https://BUCKET_NAME.s3.us-west-2.amazonaws.com", + "BUCKET_ACCESS_KEY_ID": "{{ RUNPOD_SECRET_bucket_access_key_id }}", + "BUCKET_SECRET_ACCESS_KEY": "{{ RUNPOD_SECRET_bucket_secret_access_key }}" + } + }' +``` + +`type` and `scaling` are required in REST v2, and its defaults differ from the console: FlashBoot is `OFF` and the idle timeout is 10 seconds unless you set them. In the console, the same flow is **Serverless**, then **New Endpoint**. + +### Load-balancing endpoint (HTTP servers and WebSockets) + +Create a load-balancing endpoint through REST API v2 with `"type": "LOAD_BALANCER"` and a request-count scaler. Expose the port your server binds and set `PORT` and `PORT_HEALTH` to match it. + +```bash +curl -X POST "https://api.runpod.io/v2/serverless" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "my-realtime-app", + "image": "DOCKER_USERNAME/my-realtime-app:v1", + "type": "LOAD_BALANCER", + "gpu": {"pools": ["ADA_48_PRO"], "count": 1}, + "workers": {"min": 1, "max": 3, "idleTimeout": 60}, + "scaling": {"type": "REQUEST_COUNT", "requestCount": 4}, + "ports": ["8000/http"], + "env": {"PORT": "8000", "PORT_HEALTH": "8000"} + }' +``` + +With `workers.min` at 1, one worker bills continuously. Wait for `GET https://ENDPOINT_ID.api.runpod.ai/ping` to return `200` before you send traffic. If the ports don't match, the worker shows as running but never becomes healthy. + +### Pods and Instant Clusters (fal Compute) + +A fal Compute instance is an Ubuntu machine you SSH into. A Pod is a container with SSH, so install your stack in the image or on a volume rather than with a VM-level setup, and note that Pods don't run Docker Compose. + +```bash +runpodctl pod create --name "train-h100" \ + --image "runpod/pytorch:1.0.3-cu1281-torch291-ubuntu2404" \ + --gpu-id "NVIDIA H100 80GB HBM3" \ + --gpu-count 1 \ + --container-disk-in-gb 50 \ + --network-volume-id NETWORK_VOLUME_ID \ + --ports "22/tcp,8888/http" +``` + +For the `8xH100-SXM` shape, set `--gpu-count 8` if a host with eight GPUs is in stock in your data center. For multi-node training that used InfiniBand between 8-GPU instances in one fal sector, deploy an [Instant Cluster](/instant-clusters), which provides high-speed networking between nodes with Slurm, PyTorch, or Ray setups. GPU and cluster availability varies by data center and over time, so check stock before you plan a cutover date, and add your SSH key with `rp ssh add` or in the console. Check `runpodctl pod create --help` for the full flag list. + +## Step 7: Update the code that calls fal + +### Hosted model calls + +The examples below call FLUX.1 dev. Four things change: the URL, the auth header, the `input` wrapper, and the parameter and output names. + +| | fal | Runpod | +| --- | --- | --- | +| Submit | `POST https://queue.fal.run/fal-ai/flux/dev` | `POST https://api.runpod.ai/v2/black-forest-labs-flux-1-dev/run` | +| Sync | `POST https://fal.run/fal-ai/flux/dev` | `POST https://api.runpod.ai/v2/black-forest-labs-flux-1-dev/runsync` | +| Status | `GET .../requests/REQUEST_ID/status` | `GET .../status/JOB_ID` | +| Result | `GET .../requests/REQUEST_ID` | Included in the status response as `output` | +| Cancel | `PUT .../requests/REQUEST_ID/cancel` | `POST .../cancel/JOB_ID` | +| Auth header | `Authorization: Key FAL_KEY` | `Authorization: Bearer RUNPOD_API_KEY` | +| Body | `{"prompt": "..."}` | `{"input": {"prompt": "..."}}` | +| Job ID field | `request_id` | `id` | +| Image URL | `images[0].url` | `output.image_url` | + +For FLUX.1 dev, the parameters map as follows. Other models differ, so read the Runpod model page for each one. + +| fal parameter | Runpod parameter | Notes | +| --- | --- | --- | +| `prompt` | `prompt` | Same. | +| `image_size` (preset such as `landscape_4_3`, or width and height) | `width`, `height` | Integers divisible by 64, from 256 to 1536. Default 1024. | +| `guidance_scale` (default 3.5) | `guidance` (default 7.5) | Set it explicitly to keep your current look. | +| `num_inference_steps` (default 28) | `num_inference_steps` (default 28) | Same name. Range 1 to 50. | +| `seed` | `seed` | `-1` means random on Runpod. | +| `output_format` (`jpeg`, `png`) | `image_format` (`jpeg`, `png`) | Renamed. | +| `num_images` | One image per request | Send several requests with `/run`. | +| not listed | `negative_prompt` | Available on Runpod. | +| `sync_mode`, `acceleration`, `enable_safety_checker` | Not listed on the Runpod Flux Dev page | Remove them. | + +Across models, the common renames are `image_url` to `image`, `image_urls` to `images`, and `audio_url` to `audio`. Sizes are often a `size` string such as `1024*1024` or `1280*720`. Outputs are `output.image_url`, `output.video_url`, `output.audio_url`, or `output.result`, depending on the model, plus `output.cost`. + + + + Before, with `fal-client`: + + ```python fal_call.py + import fal_client + + # Blocking call through the queue + result = fal_client.subscribe( + "fal-ai/flux/dev", + arguments={ + "prompt": "A serene mountain landscape at sunset", + "image_size": {"width": 1024, "height": 1024}, + "num_inference_steps": 28, + "guidance_scale": 3.5, + "seed": 42, + "output_format": "png", + }, + ) + print(result["images"][0]["url"]) + + # Submit, poll, and fetch + handle = fal_client.submit( + "fal-ai/flux/dev", + arguments={"prompt": "A serene mountain landscape at sunset"}, + ) + for event in handle.iter_events(): + if isinstance(event, fal_client.InProgress): + print("Processing...") + result = handle.get() + print(result["images"][0]["url"]) + ``` + + After, with plain HTTP: + + ```python runpod_call.py + import os + import time + + import requests + + BASE_URL = "https://api.runpod.ai/v2/black-forest-labs-flux-1-dev" + HEADERS = { + "Authorization": f"Bearer {os.environ['RUNPOD_API_KEY']}", + "Content-Type": "application/json", + } + TERMINAL = {"COMPLETED", "FAILED", "CANCELLED", "TIMED_OUT"} + + + def wait_for(job: dict) -> dict: + """Poll /status until the job reaches a terminal state.""" + delay = 1.0 + while job["status"] not in TERMINAL: + time.sleep(delay) + delay = min(delay * 1.5, 10.0) + response = requests.get( + f"{BASE_URL}/status/{job['id']}", headers=HEADERS, timeout=30 + ) + response.raise_for_status() + job = response.json() + if job["status"] != "COMPLETED": + raise RuntimeError(f"Job {job['id']} ended as {job['status']}: {job}") + return job + + + payload = { + "input": { + "prompt": "A serene mountain landscape at sunset", + "width": 1024, + "height": 1024, + "num_inference_steps": 28, + "guidance": 3.5, + "seed": 42, + "image_format": "png", + } + } + + # Blocking call (replaces fal_client.subscribe and fal_client.run) + response = requests.post( + f"{BASE_URL}/runsync", headers=HEADERS, json=payload, timeout=120 + ) + response.raise_for_status() + job = wait_for(response.json()) # Keeps polling if /runsync returned early + print(job["output"]["image_url"]) + + # Submit, poll, and fetch (replaces fal_client.submit) + response = requests.post(f"{BASE_URL}/run", headers=HEADERS, json=payload, timeout=30) + response.raise_for_status() + job = wait_for(response.json()) + print(job["output"]["image_url"]) + ``` + + `/runsync` returns the finished job when it completes within the wait window. If it doesn't, the response carries the job `id` and a non-terminal status, and `wait_for` picks it up from `/status`. The polling interval grows instead of running in a tight loop, which keeps you within the endpoint's rate limits. + + + + Before, with `@fal-ai/client`: + + ```javascript fal-call.mjs + import { fal } from "@fal-ai/client"; + + // Blocking call through the queue + const result = await fal.subscribe("fal-ai/flux/dev", { + input: { + prompt: "A serene mountain landscape at sunset", + image_size: { width: 1024, height: 1024 }, + num_inference_steps: 28, + guidance_scale: 3.5, + seed: 42, + output_format: "png", + }, + }); + console.log(result.data.images[0].url); + + // Submit, poll, and fetch + const { request_id } = await fal.queue.submit("fal-ai/flux/dev", { + input: { prompt: "A serene mountain landscape at sunset" }, + }); + const status = await fal.queue.status("fal-ai/flux/dev", { requestId: request_id }); + console.log(status.status); + const queued = await fal.queue.result("fal-ai/flux/dev", { requestId: request_id }); + console.log(queued.data.images[0].url); + ``` + + After, with `fetch`: + + ```javascript runpod-call.mjs + const BASE_URL = "https://api.runpod.ai/v2/black-forest-labs-flux-1-dev"; + const HEADERS = { + Authorization: `Bearer ${process.env.RUNPOD_API_KEY}`, + "Content-Type": "application/json", + }; + const TERMINAL = new Set(["COMPLETED", "FAILED", "CANCELLED", "TIMED_OUT"]); + + async function waitFor(job) { + let delay = 1000; + while (!TERMINAL.has(job.status)) { + await new Promise((resolve) => setTimeout(resolve, delay)); + delay = Math.min(delay * 1.5, 10000); + const response = await fetch(`${BASE_URL}/status/${job.id}`, { headers: HEADERS }); + if (!response.ok) throw new Error(`Status check failed: ${response.status}`); + job = await response.json(); + } + if (job.status !== "COMPLETED") { + throw new Error(`Job ${job.id} ended as ${job.status}: ${JSON.stringify(job)}`); + } + return job; + } + + const payload = { + input: { + prompt: "A serene mountain landscape at sunset", + width: 1024, + height: 1024, + num_inference_steps: 28, + guidance: 3.5, + seed: 42, + image_format: "png", + }, + }; + + // Blocking call (replaces fal.subscribe and fal.run) + let response = await fetch(`${BASE_URL}/runsync`, { + method: "POST", + headers: HEADERS, + body: JSON.stringify(payload), + }); + if (!response.ok) throw new Error(`Request failed: ${response.status}`); + let job = await waitFor(await response.json()); + console.log(job.output.image_url); + + // Submit, poll, and fetch (replaces fal.queue.submit, status, and result) + response = await fetch(`${BASE_URL}/run`, { + method: "POST", + headers: HEADERS, + body: JSON.stringify(payload), + }); + if (!response.ok) throw new Error(`Request failed: ${response.status}`); + job = await waitFor(await response.json()); + console.log(job.output.image_url); + ``` + + As with fal, don't ship the API key to a browser. Keep these calls on your server, where your `@fal-ai/server-proxy` route used to be. + + + + Before, with the fal queue API: + + ```bash + # Submit + curl -X POST "https://queue.fal.run/fal-ai/flux/dev" \ + -H "Authorization: Key $FAL_KEY" \ + -H "Content-Type: application/json" \ + -d '{"prompt": "A serene mountain landscape at sunset", "num_inference_steps": 28}' + + # Status (use the status_url from the submit response) + curl "https://queue.fal.run/fal-ai/flux/requests/REQUEST_ID/status" \ + -H "Authorization: Key $FAL_KEY" + + # Result + curl "https://queue.fal.run/fal-ai/flux/requests/REQUEST_ID" \ + -H "Authorization: Key $FAL_KEY" + + # Cancel + curl -X PUT "https://queue.fal.run/fal-ai/flux/requests/REQUEST_ID/cancel" \ + -H "Authorization: Key $FAL_KEY" + ``` + + After, with Runpod: + + ```bash + # Submit + curl -X POST "https://api.runpod.ai/v2/black-forest-labs-flux-1-dev/run" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"input": {"prompt": "A serene mountain landscape at sunset", "num_inference_steps": 28}}' + + # Status and result in one call + curl "https://api.runpod.ai/v2/black-forest-labs-flux-1-dev/status/JOB_ID" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" + + # Cancel + curl -X POST "https://api.runpod.ai/v2/black-forest-labs-flux-1-dev/cancel/JOB_ID" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" + + # Blocking call (replaces https://fal.run/fal-ai/flux/dev) + curl -X POST "https://api.runpod.ai/v2/black-forest-labs-flux-1-dev/runsync" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"input": {"prompt": "A serene mountain landscape at sunset"}}' + ``` + + The completed response looks like this: + + ```json + { + "id": "sync-a1b2c3d4-e5f6-7890-abcd-ef1234567890-u1", + "status": "COMPLETED", + "delayTime": 17, + "executionTime": 3986, + "workerId": "oqk7ao1uomckye", + "output": { + "image_url": "https://image.runpod.ai/abc123/output.png", + "cost": 0.02097152 + } + } + ``` + + + +Use `/run` with polling or a webhook for video models, which can run longer than the `/runsync` wait window. + +### Input files + +`fal_client.upload_file("image.png")` and `fal.storage.upload(file)` return a fal CDN URL that you pass to a model. Runpod Public Endpoint model pages take input media as URLs (for example `input.image` on WAN 2.5 and `input.images` on Nano Banana 2 Edit). Upload inputs to your own bucket and pass a URL the endpoint can fetch, such as a presigned URL. + +### Webhooks + +fal takes the webhook as a `fal_webhook` query parameter (or `webhook_url` in the clients). Runpod takes a top-level `webhook` field in the body. + + + + ```bash + curl -X POST "https://queue.fal.run/fal-ai/flux/dev?fal_webhook=https://example.com/api/webhook" \ + -H "Authorization: Key $FAL_KEY" \ + -H "Content-Type: application/json" \ + -d '{"prompt": "A serene mountain landscape at sunset"}' + ``` + + + + ```bash + curl -X POST "https://api.runpod.ai/v2/black-forest-labs-flux-1-dev/run" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "input": {"prompt": "A serene mountain landscape at sunset"}, + "webhook": "https://example.com/api/webhook" + }' + ``` + + + +Update your receiver for three differences: + +- **Payload.** fal posts `{"request_id", "status": "OK" or "ERROR", "payload"}`. Runpod posts the job object, the same shape `/status` returns: `id`, `status` (`COMPLETED`, `FAILED`, and so on), and `output`. +- **Retries.** fal retries with backoff for up to about an hour. Runpod expects a `200` and retries up to 2 more times with a 10-second delay, so make the receiver fast and reliable, and fall back to `/status/JOB_ID` for jobs you haven't heard about. Results are only retained for 30 minutes after completion. +- **Verification.** If you verify fal's `X-Fal-Webhook-Signature` today, replace it with a secret token in the webhook URL path or query, and confirm the job by calling `/status/JOB_ID` with your API key. + +### Vercel AI SDK + +If you call fal through `@ai-sdk/fal`, switch to [`@runpod/ai-sdk-provider`](/public-endpoints/ai-sdk). The provider reads `RUNPOD_API_KEY` in place of `FAL_API_KEY` or `FAL_KEY`. + + + + ```typescript + import { fal } from "@ai-sdk/fal"; + import { generateImage } from "ai"; + + const { image } = await generateImage({ + model: fal.image("fal-ai/flux/dev"), + prompt: "A serene mountain landscape at sunset", + aspectRatio: "16:9", + providerOptions: { + fal: { guidance_scale: 3.5, num_inference_steps: 28 }, + }, + }); + ``` + + + + ```typescript + import { runpod } from "@runpod/ai-sdk-provider"; + import { experimental_generateImage as generateImage } from "ai"; + + const { image } = await generateImage({ + model: runpod.image("black-forest-labs-flux-1-dev"), + prompt: "A serene mountain landscape at sunset", + aspectRatio: "16:9", + providerOptions: { + runpod: { guidance: 3.5, num_inference_steps: 28 }, + }, + }); + ``` + + + +Install it with `npm install @runpod/ai-sdk-provider ai`. Model-specific options move from the `fal` key to the `runpod` key in `providerOptions`, with Runpod's parameter names. The provider's model IDs for some image and video models differ from the REST endpoint IDs (for example `alibaba/wan-2.6-t2v`), so copy them from the [supported models tables](/public-endpoints/ai-sdk#supported-models). The Runpod page imports `experimental_generateImage`. Use whichever export your installed `ai` version provides. The provider also accepts your own Serverless endpoint ID as a model identifier. + +### Calls to your own app + +A deployed `fal.App` is called by its `OWNER/APP_NAME` ID. A Runpod endpoint is called by its endpoint ID. + + + + ```python + import fal_client + + result = fal_client.subscribe( + "your-username/my-demo-app", + arguments={"prompt": "A cyberpunk cityscape at night"}, + ) + print(result["image"]["url"]) + + handle = fal_client.submit( + "your-username/my-demo-app", + arguments={"prompt": "A cyberpunk cityscape at night"}, + webhook_url="https://example.com/api/webhook", + ) + print(handle.request_id) + ``` + + + + ```python + import runpod + + endpoint = runpod.Endpoint("ENDPOINT_ID") + + # Blocking call + result = endpoint.run_sync({"prompt": "A cyberpunk cityscape at night"}, timeout=120) + print(result["image"]["url"]) + + # Submit and check later + run_request = endpoint.run({"prompt": "A cyberpunk cityscape at night"}) + print(run_request.status()) + print(run_request.output()) + ``` + + + +`runpod.Endpoint` wraps the input in `{"input": ...}` for you and reads the API key from `rp login`, `RUNPOD_API_KEY`, or `runpod.api_key`. To send a webhook, post to `/run` directly with a top-level `webhook` field as shown above. From another SDK codebase, you can also call a deployed app by name with `runpod.Queue(app="my-demo-app", name="generate")` or `runpod.Api(app="my-demo-app", name="MyApp")`. + +### Control-plane automation + +Scripts and CI jobs that run the `fal` CLI or call `https://api.fal.ai/v1` move to `runpodctl`, the `rp` CLI, or REST API v2. See the [CLI command map](#cli-command-map) and [API map](#api-map). In CI, replace the fal key with a `RUNPOD_API_KEY` secret and run `rp flash deploy --env production` in place of `fal deploy --env production`. + +## Step 8: Verify, cut over, and shut down + + + + Call each Runpod endpoint from the environment your production clients run in, not from the machine you deployed from. For a queue-based endpoint: + + ```bash + curl -s -X POST "https://api.runpod.ai/v2/ENDPOINT_ID/runsync" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"input": {"prompt": "A cyberpunk cityscape at night"}}' + + curl -s "https://api.runpod.ai/v2/ENDPOINT_ID/health" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" + ``` + + Open the returned URL or decode the base64 and check the output is what you expect. + + + + Run the same prompts and seeds through fal and Runpod. Compare output quality, then the timing fields: `delayTime` (queue wait, including any cold start) and `executionTime` on Runpod against `metrics.inference_time` and your measured wall-clock time on fal. Measure a cold request and a warm one separately. + + + + Put the provider behind a flag in your client so a request can go to either. Send a small share of traffic to Runpod, watch the endpoint's **Metrics** and **Logs** tabs, and check that webhooks arrive and that your code handles `FAILED` and `TIMED_OUT`. + + + + Raise the share in stages. Before each stage, confirm your max workers and scaler settings can absorb the new volume. Keep fal deployed until you've gone a full traffic cycle on Runpod. + + + + Scale warm runners to zero first, then delete what you no longer need. Deleting is permanent, so export `/data` files and any CDN outputs you need before you run these. + + ```bash + # Stop paying for warm runners while you finish verifying + fal apps scale APP_NAME --min-concurrency 0 --concurrency-buffer 0 + + # Delete apps, per environment + fal apps delete APP_NAME + fal apps delete APP_NAME --env staging + + # Confirm nothing is still running + fal apps list + fal runners list + + # Remove stored files and secrets + fal files rm models/my-lora.safetensors + fal secrets unset HF_TOKEN + + # Revoke keys your services no longer use + fal keys list + fal keys revoke KEY_ID + ``` + + Delete fal Compute instances in the [Compute dashboard](https://fal.ai/dashboard/compute) or with `DELETE https://api.fal.ai/v1/compute/instances/INSTANCE_ID`. They bill per hour while they exist. Model API usage stops when your code stops calling it. + + + + Keep `workers` minimums at 0 for endpoints that don't need warm capacity, set a max worker count you're comfortable paying for, and set an execution timeout so a stuck job can't run indefinitely. Delete network volumes and stopped Pods you no longer need, because storage bills until it's deleted. Review spend on the [billing page](/accounts-billing/billing). + + + +## Reference + +### CLI command map + +| fal CLI | Runpod | +| --- | --- | +| `pip install fal` | `pip install runpod` (includes `rp`), plus `runpodctl` | +| `fal auth login` | `rp login` | +| `fal auth whoami` | `runpodctl user` | +| `fal run app.py::MyApp` | `rp flash dev main.py` | +| `fal deploy app.py::MyApp` | `rp flash deploy`, or `runpodctl template create --serverless` then `runpodctl serverless create` | +| `fal deploy ... --env staging` | `rp flash deploy --env staging` | +| `fal environments list` | `rp flash env list --app APP_NAME` | +| `fal apps list` | `rp flash app list`, `runpodctl serverless list` | +| `fal apps scale APP --min-concurrency N --max-concurrency M --keep-alive S` | `runpodctl serverless update ENDPOINT_ID --workers-min N --workers-max M --idle-timeout S` | +| `fal apps runners APP` | `runpodctl serverless get ENDPOINT_ID --include-workers` | +| `fal apps delete APP` | `rp flash undeploy --app APP_NAME`, or `runpodctl serverless delete ENDPOINT_ID` | +| `fal runners logs` | Endpoint **Logs** tab in the console. For Pods, `rp logs POD_ID --follow` | +| `fal runners shell RUNNER_ID` | [SSH into a worker](/serverless/development/ssh-into-workers). For Pods, `rp ssh POD_ID` | +| `fal secrets set NAME=VALUE` | `rp secret add NAME` | +| `fal secrets list` | `rp secret list` | +| `fal secrets unset NAME` | `rp secret delete NAME` | +| `fal files list`, `fal files upload`, `fal files download` | `aws s3 ls` and `aws s3 cp` against the [network volume S3 API](/storage/s3-api) | +| `fal queue flush APP` | `POST https://api.runpod.ai/v2/ENDPOINT_ID/purge-queue` | +| `fal keys create` | Create a key in the console. See [credentials](/get-started/credentials) | + +### API map + +| fal | Runpod | +| --- | --- | +| `POST https://queue.fal.run/MODEL_ID` | `POST https://api.runpod.ai/v2/ENDPOINT_ID/run` | +| `POST https://fal.run/MODEL_ID` | `POST https://api.runpod.ai/v2/ENDPOINT_ID/runsync` | +| `GET https://queue.fal.run/OWNER/APP/requests/ID/status` | `GET https://api.runpod.ai/v2/ENDPOINT_ID/status/JOB_ID` | +| `GET https://queue.fal.run/OWNER/APP/requests/ID` | Same `/status/JOB_ID` call, `output` field | +| `GET .../requests/ID/status/stream` (SSE status updates) | Poll `/status/JOB_ID`, or use a webhook | +| `PUT .../requests/ID/cancel` | `POST https://api.runpod.ai/v2/ENDPOINT_ID/cancel/JOB_ID` | +| `POST https://fal.run/MODEL_ID/stream` | `GET https://api.runpod.ai/v2/ENDPOINT_ID/stream/JOB_ID` after `/run` | +| `?fal_webhook=URL` | `"webhook": "URL"` in the request body | +| `X-Fal-Request-Start-Timeout` header | Closest: `"policy": {"ttl": MILLISECONDS}` in the request body. TTL covers queue time and execution time together. | +| App `request_timeout` | `"policy": {"executionTimeout": MILLISECONDS}`, or the endpoint's execution timeout | +| `fal_client.subscribe`, `fal.subscribe` | `/runsync`, or `runpod.Endpoint(ID).run_sync(...)` | +| `fal_client.submit`, `fal.queue.submit` | `/run`, or `runpod.Endpoint(ID).run(...)` | +| `fal_client.realtime`, `fal.realtime.connect` | Any WebSocket client to `wss://ENDPOINT_ID.api.runpod.ai/PATH` | +| `GET https://api.fal.ai/v1/serverless/...` (apps, metrics, logs) | `GET https://api.runpod.io/v2/serverless`, `/v2/serverless/ID/metrics/requests`, `/v2/serverless/ID/workers/WORKER_ID/logs` | +| `GET https://api.fal.ai/v1/compute/instances` | `GET https://api.runpod.io/v2/pods` | +| `GET https://api.fal.ai/v1/models/usage`, `/v1/serverless/usage` | `GET https://api.runpod.io/v2/billing`, `/v2/billing/serverless` | + +### Configuration field map + +| `fal.App` attribute | Runpod SDK argument | `runpodctl serverless create` flag | REST v2 field | +| --- | --- | --- | --- | +| `app_name` | `runpod.App("NAME")`, `name=` | `--name` | `name` | +| `machine_type` | `gpu=` or `cpu=` | `--gpu-id`, or `--compute-type CPU --instance-id` | `gpu.pools`, or `cpu` | +| `num_gpus` | `gpu_count=` | `--gpu-count` | `gpu.count` | +| `min_concurrency` | `workers=(MIN, MAX)` | `--workers-min` | `workers.min` | +| `max_concurrency` (fal default 2) | `workers=(MIN, MAX)` (SDK default `(0, 3)`) | `--workers-max` | `workers.max` | +| `keep_alive` (fal default 60 s) | `idle_timeout=` (seconds, default 60) | `--idle-timeout` (1 to 3600) | `workers.idleTimeout` | +| `max_multiplexing` | `max_concurrency=` on `@app.queue` | A [concurrent handler](/serverless/workers/concurrent-handler) in your image | A concurrent handler in your image | +| `scaling_delay` | `scaler_type="QUEUE_DELAY"`, `scaler_value=` | `--scale-by delay --scale-threshold` | `scaling.type`, `scaling.queueDelay` | +| `request_timeout` | `execution_timeout_ms=` | `--execution-timeout` (seconds) | `timeout` (ms) | +| `startup_timeout` | `env={"RUNPOD_INIT_TIMEOUT": "SECONDS"}` | `RUNPOD_INIT_TIMEOUT` in the template env | `env.RUNPOD_INIT_TIMEOUT` | +| `regions` | `datacenter=` | `--data-center-ids` | `dataCenterIds` | +| `requirements` | `dependencies=`, `system_dependencies=` | Installed in your image | Installed in your image | +| `image = ContainerImage(...)` | `image=`, `registry_auth=` | Template `--image`, `--registry-auth-id` | `image`, `registry` | +| `secrets = [...]` | `env={"KEY": runpod.Secret("NAME")}` | Template `--env` with `{{ RUNPOD_SECRET_NAME }}` | `env` with `{{ RUNPOD_SECRET_NAME }}` | +| `/data` | `mounts={"/runpod-volume": runpod.NetworkVolume("NAME")}`, or `model=` | `--network-volume-id`, `--model-reference` | `mounts.network`, `mounts.global` | +| `exposed_port` (direct server mode) | `@app.api` | Template `--ports` plus `PORT` env | `type: "LOAD_BALANCER"`, `ports`, `env.PORT` | +| `app_auth` | Runpod API key on every request | Runpod API key on every request | Runpod API key on every request | + +### Environment variable map + +| fal runner variable | Runpod | Notes | +| --- | --- | --- | +| `FAL_KEY` | A Runpod API key you pass as a secret | Serverless workers don't get an account-wide key unless you provide one. Pods receive a Pod-scoped `RUNPOD_API_KEY`. | +| `FAL_APP_ID`, `FAL_APP_NAME` | `RUNPOD_ENDPOINT_ID` | Set automatically on Serverless workers. | +| `FAL_REGION` | `RUNPOD_DC_ID` | Listed for Pods. | +| `FAL_JOB_ID` (runner ID) | `RUNPOD_POD_ID` | Listed for Pods. On Serverless, the job ID is `job["id"]` in the handler. | +| `HF_HOME=/data/.cache/huggingface` | Not set for you | Use cached models, or set the cache path yourself. | +| Your secrets (injected by name) | Variables you map with `runpod.Secret` or `{{ RUNPOD_SECRET_NAME }}` | Nothing is injected unless you reference it. | +| n/a | `PORT`, `PORT_HEALTH`, `HEALTH_CHECK_PATH` | Load-balancing endpoints. | +| n/a | `BUCKET_ENDPOINT_URL`, `BUCKET_ACCESS_KEY_ID`, `BUCKET_SECRET_ACCESS_KEY` | Read by the `rp_upload` helper. | + +### Status map + +| fal queue status | Runpod job status | Notes | +| --- | --- | --- | +| `IN_QUEUE` | `IN_QUEUE` | Waiting for a worker. fal includes `queue_position`. | +| `IN_PROGRESS` | `IN_PROGRESS` | A worker picked up the job. | +| `COMPLETED` (no `error`) | `COMPLETED` | Result is in `output`. | +| `COMPLETED` with `error` and `error_type` | `FAILED` | The job hit an error. | +| Cancel response `OK` or `CANCELLATION_REQUESTED` | `CANCELLED` | Cancelled with `/cancel/JOB_ID`. | +| `504` with `X-Fal-Request-Timeout-Type: user`, or `request_timeout` exceeded | `TIMED_OUT` | The job expired before a worker picked it up, or the worker didn't report back in time. | +| Webhook `status: "OK"` or `"ERROR"` | Webhook body `status: "COMPLETED"`, `"FAILED"`, and so on | The Runpod webhook body is the job object. | + +## Migrate with a coding agent + +Install the Runpod plugin or skills and connect the MCP server. MCP sign-in is OAuth ("Sign in with Runpod"), so you don't need an API key to start. See [agent skills](/get-started/agent-skills) and [MCP servers](/get-started/mcp-servers). + + + + ```bash + # The plugin also connects the hosted MCP server + claude plugin marketplace add runpod/runpod-plugins-official + claude plugin install runpod@runpod + ``` + + + + ```bash + # Codex, Cursor, and others: skills, then the hosted MCP server + npx skills add runpod/runpod-plugins-official --skill '*' --agent AGENT_ID --global + npx @runpod/mcp-server@latest add --client CLIENT_ID + ``` + + + +Then give the agent this prompt from the root of your repository: + +```text +Migrate this project from fal to Runpod using the runpod-cloud-migrate skill. + +1. Inventory first, and change nothing yet. Find every use of fal in this repo: + fal_client, @fal-ai/client, @ai-sdk/fal, fal.App classes, [tool.fal] in + pyproject.toml, fal CLI calls in scripts and CI, queue.fal.run and fal.run URLs, + FAL_KEY and FAL_API_KEY, webhook receivers that read fal payloads, and stored + fal.media URLs. If the fal CLI is logged in, also run the read-only commands: + fal apps list --json, fal secrets list, fal files list, fal environments list. +2. Classify each workload: hosted model call (Runpod Public Endpoint), fal.App + (Runpod SDK @app.queue or @app.api, or a custom image with a handler), + realtime or WebSocket endpoint (load-balancing endpoint), ComfyUI app + (ComfyUI worker), or fal Compute instance (Pod or Instant Cluster). +3. For hosted model calls, map each fal model ID to a Runpod Public Endpoint ID + only if the Runpod model reference lists it. List the models with no match + and stop to ask me what to do with them. +4. Show me a written plan before creating anything: the target for each workload, + the GPU pool for each endpoint, where model weights and /data files will live, + which secrets I need to recreate, every code change, and a cost estimate from + Runpod's published pricing with the assumptions you used. +5. Wait for my approval before you create anything billable on Runpod (endpoints, + Pods, volumes, clusters). +6. Migrate one workload end to end first. Verify it with a real request from + outside the worker, and show me the response and the delayTime and + executionTime values. Then continue with the rest. +7. Update client code behind a flag so I can run fal and Runpod in parallel. + Handle the Runpod statuses FAILED, CANCELLED, and TIMED_OUT, the "input" + wrapper, the Bearer auth header, and the 30-minute result retention. +8. Never delete, scale down, or modify fal resources, and never revoke fal keys, + without my explicit approval. When I approve, list the exact fal commands + first. +``` + +## Troubleshooting + +**`401` from every Runpod request.** The client still sends `Authorization: Key ...`. Runpod expects `Authorization: Bearer YOUR_API_KEY`. On WebSockets, send the header on the connection request. + +**The job fails validation or ignores your parameters.** The body isn't wrapped in `input`, or it uses fal parameter names. Send `{"input": {...}}` and rename parameters per the Runpod model page (`image_size` to `width` and `height`, `guidance_scale` to `guidance`, `image_url` to `image`). + +**`KeyError: 'images'` or `undefined` when reading the result.** The output shape changed. Read `output.image_url`, `output.video_url`, `output.audio_url`, or `output.result`, depending on the model. + +**`/status` returns `404` or no output for a job that finished.** Results are kept for 30 minutes after a `/run` job completes and 1 minute for `/runsync`, and a job's TTL (24 hours by default) deletes it regardless of state. Fetch results sooner or use a webhook. + +**Code waits forever on a job that failed.** A polling loop that only exits on `COMPLETED` never sees `FAILED`, `CANCELLED`, or `TIMED_OUT`. Treat all four as terminal. + +**Jobs end as `TIMED_OUT` that finished on fal.** fal's `request_timeout` defaults to 3600 seconds, and Runpod's execution timeout defaults to 600 seconds in the console and 300000 ms in REST v2. Raise the endpoint's execution timeout or set `policy.executionTimeout` on the request. + +**Every cold start downloads the model again.** fal cached Hugging Face downloads on `/data` without configuration. On Runpod, add a cached model (`model=` or `--model-reference`), bake weights into the image, or use a network volume with the volume cache pattern. + +**`os.getenv("HF_TOKEN")` is `None` on the worker.** Secrets aren't injected automatically. Create the secret, then reference it with `runpod.Secret("hf-token")` or `{{ RUNPOD_SECRET_hf-token }}` in the endpoint's environment. Values in a local `.env` file aren't sent to workers. + +**A load-balancing worker shows as running but requests hang or return `502`.** The server isn't listening where the health check looks. Bind `0.0.0.0` on `PORT`, set `PORT_HEALTH` to the same port, expose that port, and answer `GET /ping` with `200`. An error of `not allowed for QB API` means the endpoint was created as queue-based. + +**WebSocket connections time out on connect.** The client gave up while a worker was starting. Set `open_timeout=60` or the equivalent in your client, and keep one active worker for interactive apps. + +**The job fails when the handler returns a large image or video.** Job payloads are limited to 10 MB for `/run` and 20 MB for `/runsync`. Upload the file to your bucket with `rp_upload` and return the URL. + +**The image won't start, or the worker exits immediately.** Images built on Apple silicon default to `arm64`. Build with `docker build --platform linux/amd64`. If CUDA initialization fails, set a minimum CUDA version on the endpoint that matches your image. + +## Next steps + + + + Check current models, parameters, and pricing. + + + + Define queue, API, and task resources in Python. + + + + Tune workers, idle timeout, and autoscaling. + + + + Serve your own HTTP and WebSocket routes. + + diff --git a/migrate/fireworks-ai.mdx b/migrate/fireworks-ai.mdx new file mode 100644 index 000000000..c8f5b2bb6 --- /dev/null +++ b/migrate/fireworks-ai.mdx @@ -0,0 +1,929 @@ +--- +title: "Migrate from Fireworks AI to Runpod" +sidebarTitle: "Fireworks AI" +description: "Move Fireworks AI serverless inference, on-demand deployments, custom models, and fine-tuning to Runpod Serverless, Public Endpoints, and Pods." +--- + +This guide is for teams that call [Fireworks AI](https://docs.fireworks.ai) from production code and want to run the same workloads on Runpod. Fireworks [Serverless inference](https://docs.fireworks.ai/serverless/overview) maps to either a Runpod [Public Endpoint](/public-endpoints/overview) (when Runpod hosts the model) or your own OpenAI-compatible endpoint on [Runpod Serverless](/serverless/vllm/overview). Fireworks [on-demand deployments](https://docs.fireworks.ai/guides/ondemand-deployments), [custom models](https://docs.fireworks.ai/models/uploading-custom-models), and LoRA adapters map to a Serverless endpoint running the vLLM worker, and [training](https://docs.fireworks.ai/fine-tuning/finetuning-intro) maps to a [Pod](/pods/overview). When you finish, your application sends the same OpenAI-format requests to a Runpod base URL, and nothing is left billing on Fireworks. + + +You can hand most of this migration to a coding agent. See [Migrate with a coding agent](#migrate-with-a-coding-agent). + + +## How Fireworks AI maps to Runpod + +Fireworks is a model platform: you pick a model ID and Fireworks runs the inference stack. Runpod is a compute platform with a hosted model catalog on top. On Runpod, you either call a model Runpod already hosts, or you run an open-source inference server (vLLM) on GPUs you configure. + +| Fireworks AI concept | Runpod equivalent | Notes | +| --- | --- | --- | +| Serverless inference (shared, per-token) | [Public Endpoints](/public-endpoints/overview) | Only for models in the [Public Endpoints catalog](/public-endpoints/reference). Fireworks' serverless catalog is larger. | +| Serverless inference for a model Runpod doesn't host | Serverless endpoint running the [vLLM worker](/serverless/vllm/overview) | Serves a Hugging Face model you choose. You size the GPU and pay for worker time. | +| On-demand deployment (dedicated GPUs) | Serverless endpoint running the vLLM worker | The closest match: dedicated workers, autoscaling, scale to zero. | +| Deployment shape | GPU pool, GPUs per worker, and vLLM environment variables | Runpod has no pre-validated shapes. You choose hardware and serving flags. | +| Replicas (`--min-replica-count`, `--max-replica-count`) | Active workers and max workers | See [endpoint settings](/serverless/endpoints/endpoint-configurations). | +| `--scale-to-zero-window` | Idle timeout | Runpod's range is 1 to 3,600 seconds. | +| `--load-targets` | Autoscaling type (queue delay or request count) | Different signals. See [What works differently](#what-works-differently). | +| Model ID (`accounts/fireworks/models/NAME`) | Hugging Face repo ID, or a Public Endpoint slug | You can alias the served name with `OPENAI_SERVED_MODEL_NAME_OVERRIDE`. | +| Custom base model (`firectl model create`) | Hugging Face repo with [cached models](/serverless/endpoints/model-caching), or a [network volume](/storage/network-volumes) | The vLLM worker loads a repo ID or a local path. | +| LoRA addon (live merge or multi-LoRA) | vLLM `ENABLE_LORA` and `LORA_MODULES`, or a merged model | Adapters load from Hugging Face. | +| Training API (SFT, DPO, RL) | A [Pod](/pods/overview) running your own training code | Runpod provides GPUs and storage, not a managed training service. | +| Batch API (paused, per Fireworks docs) | [Batch jobs](/serverless/batch-jobs), or `/run` with webhooks | Queue-based endpoints only. | +| Datasets | Network volume, or your own object storage | | +| Secrets (`firectl secret create`) | [Runpod secrets](/pods/templates/secrets) | Reference as `{{ RUNPOD_SECRET_name }}` in an environment variable. | +| Regions (`--region GLOBAL`, `US`, single regions) | Data centers on the endpoint | Leaving data centers unrestricted gives the scheduler the most hardware to choose from. | +| Billing: per token (serverless), per GPU-second (deployments) | Per token or per output (Public Endpoints), per second of worker time (Serverless), per second (Pods) | See [Serverless pricing](/serverless/pricing). | +| Reserved capacity (enterprise) | Active workers, with discounts through sales | Runpod doesn't guarantee GPU capacity. | +| `firectl` | [`runpodctl`](/runpodctl/overview) | See the [CLI command map](#cli-command-map). | +| Fireworks Python SDK (`fireworks-ai`) | OpenAI SDK for inference, [Runpod SDK](/sdk/overview) (`pip install runpod`) for building apps | | +| REST control plane (`api.fireworks.ai/v1/accounts/...`) | [REST API v2](/api-reference-v2/overview) (`api.runpod.io/v2`) | | +| API key (`FIREWORKS_API_KEY`) | API key (`RUNPOD_API_KEY`) | One key for inference and management. | +| Users, roles, service accounts | [Organizations](/organizations/orgs-overview) and [roles](/organizations/org-roles) | | +| Prometheus metrics endpoint for deployments | Endpoint metrics in the console and REST API v2 (`/v2/serverless/ENDPOINT_ID/metrics/...`) | See [What has no direct equivalent](#what-has-no-direct-equivalent). | +| Infrastructure as code | [Terraform provider](/integrations/terraform/overview) (`runpod/runpod`, built on REST API v2) | Fireworks publishes a Terraform module for bring-your-own-cluster setups, not a provider for deployments. | + +## What's different + +### What works differently + +**You choose the model weights and the serving flags.** A Fireworks model ID hides the checkpoint, precision, chat template, and tool-call parser. On a vLLM endpoint, you pick the Hugging Face repo and set those yourself with [environment variables](/serverless/vllm/environment-variables). Output can differ from Fireworks for the same model family if the quantization or chat template differs, so compare outputs before you cut over. + +**Billing moves from tokens to time for your own endpoints.** Fireworks Serverless bills input, cached input, and generated tokens. A Runpod Serverless endpoint bills per second while a worker runs, which includes the cold start, request processing, and the idle timeout. Public Endpoints bill per use (per token for text models). A workload with steady traffic and high GPU utilization behaves differently from a spiky, low-volume one, so estimate from your own request logs. + +**A cold endpoint queues requests instead of rejecting them.** A Fireworks deployment scaled to zero returns `503` with `DEPLOYMENT_SCALING_UP` and doesn't queue the request, so clients retry. A Runpod queue-based endpoint holds the request in its queue while a worker starts. The trap is the synchronous path: `/runsync` and the `/openai/v1` route sit behind an edge timeout of about 100 seconds, so a first request that waits on a large model load can return a `524`. Use `/run` plus `/status` for the first request, raise the idle timeout, or keep an active worker. + +**Scale-down happens much sooner by default.** Fireworks' default scale-to-zero window is 1 hour (minimum 5 minutes). Runpod's console default idle timeout is 5 seconds, and the maximum is 3,600 seconds. Set it deliberately. Workers are billed while they wait. + +**Autoscaling signals are different.** Fireworks scales on load targets such as `concurrent_requests` or `tokens_generated_per_second`. Runpod scales on queue delay (seconds a request waits) or request count (queued plus in-progress requests divided by a target). The vLLM worker's `MAX_CONCURRENCY` (default 30) sets how many requests one worker takes at once. + +**Idle resources are cleaned up differently.** Fireworks deletes a deployment with min replicas of 0 after 7 days without traffic. Runpod reduces an idle endpoint's max workers to 2 after 3 days and to 0 after 7 days, and keeps the endpoint. Raise max workers to use it again. + +**Model names change.** `accounts/fireworks/models/llama-v3p1-8b-instruct` becomes the Hugging Face repo the worker serves, for example `meta-llama/Llama-3.1-8B-Instruct`. Set `OPENAI_SERVED_MODEL_NAME_OVERRIDE` to serve it under a name of your choice. + +**Reasoning output uses a different field.** Fireworks returns reasoning in `reasoning_content`. vLLM v0.32.0 (the version the current vLLM worker release is built on) returns it in `reasoning`. Read both fields in your client. Reasoning also needs `REASONING_PARSER` set on the worker. + +**Fireworks request extensions don't carry over.** Remove `service_tier`, `context_length_exceeded_behavior`, `perf_metrics_in_response`, and the `x-session-affinity` header unless you've confirmed the target accepts them. Fireworks lowers `max_tokens` when a request would exceed the context window. On Runpod, set `max_tokens` explicitly. The Runpod vLLM docs list a default of 16. + +**Token usage in streams.** Fireworks includes `usage` in the last streamed chunk by default. If your code reads it, test that path against your Runpod endpoint. + +**Rate limits are per endpoint, not per model.** Fireworks Serverless enforces adaptive tokens-per-minute limits per model. Runpod enforces request-rate limits per endpoint and operation (see [rate limits](/serverless/endpoints/send-requests#rate-limits)). Throughput on your own endpoint is bounded by your workers. + +### What has no direct equivalent + +| Fireworks AI feature | Status on Runpod | Workaround | +| --- | --- | --- | +| Large shared serverless catalog, including Priority, Fast, and Spot variants | Public Endpoints host a smaller catalog. Service tiers aren't part of it. | Deploy the model on your own vLLM endpoint. Use active workers for consistent latency. | +| Embeddings (`/v1/embeddings`) and reranking (`/v1/rerank`) | The vLLM worker docs list `/chat/completions`, `/completions`, and `/models`. | Run an embedding server on a [load balancing endpoint](/serverless/load-balancing/overview) or a custom [queue worker](/serverless/workers/handler-functions). Budget time for this. | +| Responses API (`/v1/responses`) with stored state and server-run MCP tools | Not documented for the vLLM worker or Public Endpoints. | Use chat completions and keep conversation state in your application. | +| Anthropic-compatible Messages API | Not documented on Runpod. | Use the OpenAI-compatible route, or put a translation gateway in front. | +| Deployment shapes (validated hardware and precision per model) | You size the GPU yourself. | Use the [VRAM estimates](/serverless/vllm/configuration#gpu-selection), then test. | +| Managed Training API | Runpod doesn't run training jobs for you. | Train on a Pod with your own code. See [Step 5](#step-5-move-your-data-and-models). | +| Serverless LoRA inference billed per token | LoRA adapters run on your own endpoint. | Serve adapters with `ENABLE_LORA`. | +| Prompt caching discount | Your own endpoint bills by time, so there's no cached-token rate. | `ENABLE_PREFIX_CACHING=true` enables vLLM prefix caching for speed. | +| Multi-region placement with automatic cross-region moves (`GLOBAL`) | An endpoint can run in several data centers, with no cross-region replica migration feature. | Leave data centers unrestricted, and list several GPU pools. | +| Reserved capacity with a capacity guarantee | No capacity guarantee. | Active workers keep workers running. Talk to sales about committed pricing. | +| Direct routes over AWS PrivateLink or GCP Private Service Connect | Not documented on Runpod. | Check [security and compliance](/references/security-and-compliance) and ask sales. | +| AMD accelerators (`AMD_MI325X_256GB`, `AMD_MI350X_288GB`) | The vLLM worker is a CUDA image. | Use an NVIDIA GPU with enough VRAM. | + +Going the other way, Runpod gives you things Fireworks deployments don't: any container image and any inference engine, SSH access to Pods, network volumes shared between training and serving, a job queue with webhooks and `/status` polling, and a [batch jobs](/serverless/batch-jobs) API while Fireworks' Batch API is paused. + +## Requirements + +- A [Runpod account](/accounts-billing/manage-accounts) with credits and a [Runpod API key](/get-started/credentials). +- `firectl` signed in to the Fireworks account you're migrating (`firectl whoami` shows the account). +- A [Hugging Face access token](https://huggingface.co/docs/hub/en/security-tokens) if any model is gated or private. +- Your Fireworks request logs or billing export, to size GPUs and estimate cost. + +## Step 1: Take inventory of what you run on Fireworks AI + +List every resource in the account. Add `--output json` (or `-o json`) to capture the full configuration. + +```bash +firectl whoami # the account you're signed in to +firectl deployment list --output json # on-demand deployments +firectl deployed-model list # LoRA adapters loaded on deployments +firectl model list # custom base models and adapters +firectl model list --kind HF_PEFT_ADDON # LoRA adapters only +firectl dataset list # training datasets +firectl supervised-fine-tuning-job list # training jobs +firectl router list # routers +firectl secret list # secret names +firectl reservation list # reserved capacity (enterprise) +firectl quota list # GPU quotas and rate limits +``` + +For each deployment, export the settings you'll translate in Step 3 and Step 6. + +```bash +firectl deployment get DEPLOYMENT_ID --output json +``` + +Record the base model, deployment shape, accelerator type and count, precision, min and max replica counts, the three scaling windows, load targets, region, max context length, and whether addons are enabled. `replicaStats.readyReplicaCount` shows how many replicas are serving. + +Serverless usage has no resource to list, so find it in code. This command finds SDK imports, base URLs, model IDs, and environment variables. + +```bash +grep -rnE "api\.fireworks\.ai|accounts/[a-z0-9-]+/(models|deployments|routers)/|FIREWORKS_API_KEY|fireworks-ai|from fireworks|import fireworks|@ai-sdk/fireworks|fireworks_ai/|firectl" \ + --include="*.py" --include="*.ts" --include="*.tsx" --include="*.js" --include="*.go" \ + --include="*.json" --include="*.yaml" --include="*.yml" --include="*.toml" --include="*.env*" \ + --include="*.sh" --include="Dockerfile*" . +``` + +For each call site, write down the model ID, the route (`/chat/completions`, `/completions`, `/embeddings`, `/rerank`, `/responses`, or the Anthropic-compatible `/v1/messages`), and which features it uses: streaming, tools, `response_format`, vision input, `reasoning_effort`, or `service_tier`. Gateways such as LiteLLM reference Fireworks as `fireworks_ai/MODEL`, so check their config files too. + + +Fireworks' docs say the Batch API is paused and that stored Batch input datasets and job outputs will be permanently deleted. If you have Batch data there, download it now with `firectl dataset download DATASET_ID --output-dir ./data`. + + +## Step 2: Set up Runpod + +Create an API key on the [API keys page](https://console.runpod.io/user/settings), then install the CLI and export the key. + +```bash +curl -fsSL https://cli.runpod.net | sudo bash # or: brew install runpod/runpodctl/runpodctl +export RUNPOD_API_KEY="YOUR_API_KEY" +runpodctl doctor +``` + +`runpodctl doctor` stores the key and verifies the setup. You need `runpodctl` 2.4.0 or later for `--model-reference`. Check with `runpodctl version`. + +Inference clients keep using the OpenAI SDK, so there's nothing else to install for them. If you also want to build custom workers in Python, install the Runpod SDK with `pip install runpod` (see the [SDK overview](/sdk/overview)). To let a coding agent manage resources, connect the [Runpod MCP server](/get-started/mcp-servers). + +## Step 3: Choose where each workload lands + +Use this table to place each item from your inventory. + +| Fireworks AI workload | Runpod target | Why | +| --- | --- | --- | +| Serverless call to a model that Runpod hosts | Public Endpoint | No deployment. OpenAI-compatible for the text models listed below. | +| Serverless call to any other open model | Serverless queue endpoint with the vLLM worker and a cached model | You run the model. Scale to zero when idle. | +| On-demand deployment of a base model | Serverless queue endpoint with the vLLM worker | Same shape: dedicated workers, min and max, scale to zero. | +| On-demand deployment that needs a newer vLLM than the Hub worker ships, or direct HTTP without a queue | Serverless load balancing endpoint running `vllm/vllm-openai` | Your requests go straight to the vLLM server. | +| Custom base model or merged fine-tune | Same as above, loading from Hugging Face or a network volume | | +| LoRA adapter (live merge or multi-LoRA) | vLLM worker with `ENABLE_LORA` and `LORA_MODULES`, or merge and serve as a base model | | +| Embedding or reranking model | Load balancing endpoint or custom queue worker | No prebuilt path in the Runpod docs. | +| Training job | Pod with a network volume | Run your own trainer. | +| Batch or offline inference | Queue endpoint with [batch jobs](/serverless/batch-jobs), or `/run` with a webhook | | +| Image generation (for example FLUX.1 [dev] on a Fireworks deployment) | Public Endpoint where listed, otherwise a [Hub](/hub/overview) worker | Request format changes. | +| Speech-to-text | Hub worker (`runpodctl hub search whisper --type SERVERLESS`) | Public Endpoints list text-to-speech models, not transcription. | + +### Models that Runpod hosts + +Build your model map from the [Public Endpoints model reference](/public-endpoints/reference), which lists image, video, audio (text-to-speech), and text models with their prices. These are the text models with pages in the Runpod docs today. + +| Runpod model | Endpoint slug | `model` value on the OpenAI route | Notes | +| --- | --- | --- | --- | +| [Qwen3 32B AWQ](/public-endpoints/models/qwen3-32b) | `qwen3-32b-awq` | `Qwen/Qwen3-32B-AWQ` | Documented as OpenAI-compatible. | +| [Moonshot Kimi](/public-endpoints/models/moonshot-kimi) | `moonshot-kimi` | `kimi-k2.6` (default), `kimi-k2.7-code`, `kimi-k3`, `kimi-k3.1-pro`, `kimi-k3.1-pro-highspeed` | Documented as OpenAI-compatible. One endpoint serves all five variants. | +| [GPT-OSS 120B](/public-endpoints/models/gpt-oss-120b) | `gpt-oss-120b` | `gpt-oss-120b` on the model page, `openai/gpt-oss-120b` on the [AI coding tools page](/public-endpoints/ai-coding-tools) | Documented as OpenAI-compatible, with tool calling in the [AI SDK guide](/public-endpoints/ai-sdk). Call `/models` to confirm the ID. Not in the model reference table. | +| [Cogito 671B v2.1](/public-endpoints/models/cogito-671b) | `cogito-671b-v2-1-fp8-dynamic` | `cogito-671b-v2-1-fp8-dynamic` | Documented as OpenAI-compatible. Not in the model reference table, so confirm it in the playground. | +| [IBM Granite 4.0](/public-endpoints/models/granite-4) | `granite-4-0-h-small` | Check the model | The model page documents the native `/runsync` route with `input.messages`. | + +The OpenAI-compatible base URL for a Public Endpoint is `https://api.runpod.ai/v2/SLUG/openai/v1`. Every Public Endpoint also accepts the native routes: `https://api.runpod.ai/v2/SLUG/runsync` and `/run`. + +For Fireworks model IDs that match these, such as `accounts/fireworks/models/gpt-oss-120b` or `accounts/fireworks/models/kimi-k3`, compare the model card on each side before you assume they're the same checkpoint and precision. For everything else, the general answer is your own vLLM endpoint: Fireworks' serverless catalog is larger than Runpod's Public Endpoints catalog. + +### GPU mapping + +Fireworks accelerator types translate to Runpod GPU pools (for Serverless) or GPU type IDs (for Pods) as follows. IDs come from the [GPU types reference](/references/gpu-types). + +| Fireworks accelerator | Runpod Serverless pool | Runpod GPU type IDs | VRAM | +| --- | --- | --- | --- | +| `NVIDIA_A100_80GB` | `AMPERE_80` | `NVIDIA A100 80GB PCIe`, `NVIDIA A100-SXM4-80GB` | 80 GB | +| `NVIDIA_H100_80GB` | `ADA_80_PRO` | `NVIDIA H100 80GB HBM3`, `NVIDIA H100 PCIe` | 80 GB | +| `NVIDIA_H200_141GB` | `HOPPER_141` | `NVIDIA H200` | 141 GB | +| `NVIDIA_B200_180GB` | Read the pool from `GET /v2/catalog/gpus` | `NVIDIA B200` | 180 GB | +| `NVIDIA_B300_288GB` | Read the pool from `GET /v2/catalog/gpus` | `NVIDIA B300 SXM6 AC` | 288 GB | +| `AMD_MI325X_256GB`, `AMD_MI350X_288GB` | None for the vLLM worker | Use an NVIDIA type with enough VRAM | | + +You aren't limited to the hardware Fireworks used. A model that Fireworks served on an 80 GB card at FP8 may fit a 48 GB pool (`AMPERE_48`, `ADA_48_PRO`) or a 24 GB pool (`AMPERE_24`, `ADA_24`) depending on its size and quantization. Pick the GPU that fits the model and your latency target, and list more than one pool so workers have a fallback. Stock varies by data center, so check it before you commit to a pool. + +```bash +runpodctl gpu list +``` + +## Step 4: Move your images and code + +Most Fireworks users have no container images to move, because Fireworks runs the inference server. On Runpod, the prebuilt [vLLM worker](/serverless/vllm/overview) fills that role, so this step is about translating serving configuration into environment variables. + +The vLLM worker turns any `vllm serve` flag into an environment variable in upper snake case (`--max-model-len` becomes `MAX_MODEL_LEN`). `VLLM_EXTRA_ARGS` passes flags that have no variable. + +| What Fireworks decided for you | Set on the Runpod vLLM worker | +| --- | --- | +| Model checkpoint | `MODEL_NAME` (Hugging Face repo ID or local path), `MODEL_REVISION` | +| Context length (`--max-context-length`) | `MAX_MODEL_LEN` | +| Precision (`--precision FP8`, `FP4`, `BF16`) | Choose a repo with the quantization you want. Set `QUANTIZATION` or `DTYPE` when the model's docs call for it. | +| Tool calling | `ENABLE_AUTO_TOOL_CHOICE=true` and `TOOL_CALL_PARSER` (for example `llama3_json`, `hermes`, `mistral`) | +| Reasoning | `REASONING_PARSER` (for example `deepseek_r1`, `qwen3`) | +| Chat template | The model's own template, or `CUSTOM_CHAT_TEMPLATE` | +| GPUs per replica (`--accelerator-count`) | GPUs per worker. The worker sets `TENSOR_PARALLEL_SIZE` to the number of visible GPUs. | +| Speculative decoding (`--draft-model`) | `SPECULATIVE_MODEL`, `NUM_SPECULATIVE_TOKENS` | +| LoRA addons (`--enable-addons`) | `ENABLE_LORA=true`, `LORA_MODULES`, `MAX_LORAS`, `MAX_LORA_RANK` | +| Gated or private weights | `HF_TOKEN`, from a Runpod secret | +| Served model name | `OPENAI_SERVED_MODEL_NAME_OVERRIDE` | + +Check the model's Hugging Face README and the [model-specific settings](/serverless/vllm/configuration#model-specific-configurations) for the parser and tokenizer values your model family needs. + +If the model needs a vLLM version newer than the Hub worker ships, use the upstream `vllm/vllm-openai` image on a load balancing endpoint instead (see [Step 6](#step-6-deploy-on-runpod)). Any image you build yourself must be `linux/amd64`. + +If you run workloads vLLM doesn't serve (embeddings with a different server, rerankers, image models), you write a worker. A queue worker is a Python handler started with `runpod.serverless.start({"handler": handler})`. See [handler functions](/serverless/workers/handler-functions) and [deploy a worker](/serverless/workers/deploy). + +## Step 5: Move your data and models + +### Base models from the Fireworks library + +If you deploy a Fireworks-hosted model (`accounts/fireworks/models/NAME`), don't move weights. Find the matching Hugging Face repo and reference it. Runpod [caches the model](/serverless/endpoints/model-caching) on worker hosts. + +### Custom models and fine-tuned weights + +Download each custom model and adapter from Fireworks. + +```bash +firectl model download MODEL_ID ./weights/MODEL_ID +``` + +Fireworks' docs note that an adapter is tied to the exact base checkpoint it was trained against, which can differ from the public Hugging Face weights. Download the base with `firectl model download` too if you aren't sure, or ask Fireworks which base was used. + +To serve a fine-tune as a single model, merge the adapter into the base. This follows the PEFT recipe in Fireworks' own docs. + +```python merge.py +from peft import PeftModel +from transformers import AutoModelForCausalLM, AutoTokenizer + +base = AutoModelForCausalLM.from_pretrained("./weights/base", torch_dtype="bfloat16") +merged = PeftModel.from_pretrained(base, "./weights/adapter").merge_and_unload() + +merged.save_pretrained("./weights/merged") +AutoTokenizer.from_pretrained("./weights/base").save_pretrained("./weights/merged") +``` + +The merged directory is a standard Hugging Face checkpoint in BF16. Then put the weights where a Runpod worker can load them. You have three options. + + + + Push the merged model (or the adapter) to a private Hugging Face repo. The endpoint then uses cached models, which gives the fastest cold starts and no data-center pinning. + + ```bash + hf auth login + hf repos create YOUR_ORG/my-model --private + hf upload YOUR_ORG/my-model ./weights/merged + ``` + + Create the repository first. `hf upload` creates a missing repository for you, and a repository created that way isn't private unless you ask for it. + + Reference it in Step 6 with `--model-reference https://huggingface.co/YOUR_ORG/my-model:main` and an `HF_TOKEN` that can read the repo. + + + Use a network volume when the weights can't go to Hugging Face. Create the volume in a data center that supports the [S3-compatible API](/storage/s3-api), then upload with the AWS CLI and an S3 API key from the [Credentials page](https://console.runpod.io/user/credentials). + + ```bash + runpodctl network-volume create --name fw-models --size 100 --data-center-id US-KS-2 + + aws s3 sync ./weights/merged s3://NETWORK_VOLUME_ID/models/my-model/ \ + --region US-KS-2 \ + --endpoint-url https://s3api-us-ks-2.runpod.io/ + ``` + + The volume mounts at `/runpod-volume` on Serverless workers, so set `MODEL_NAME=/runpod-volume/models/my-model`. A network volume pins the endpoint to the volume's data center, which narrows the GPUs available to it. + + + For a fully reproducible image, build a custom vLLM worker image with the model inside. See [deployment options](/serverless/vllm/overview#deployment-options). The image must be `linux/amd64`. + + + +### LoRA adapters without merging + +To keep adapters separate (the equivalent of Fireworks multi-LoRA), push each adapter to Hugging Face and load it by name. `LORA_MODULES` takes a JSON list. + +```text +ENABLE_LORA=true +MAX_LORA_RANK=64 +LORA_MODULES=[{"name": "support-adapter", "path": "YOUR_ORG/support-adapter"}] +``` + +Clients then set `model` to the adapter name (`support-adapter`) instead of Fireworks' `MODEL#DEPLOYMENT` form. Set `MAX_LORA_RANK` to at least your adapter's rank (the default is 16), and `MAX_LORAS` to the number of adapters used in one batch. + +### Datasets and training + +Fireworks training datasets use the OpenAI chat format (`messages` with `role` and `content`), which most open-source trainers read directly. Download them, then train on a Pod that writes to a network volume. + +```bash +firectl dataset download DATASET_ID --output-dir ./data + +runpodctl network-volume create --name ft-train --size 100 --data-center-id US-KS-2 +runpodctl pod create --name ft-train \ + --image "runpod/pytorch:1.0.3-cu1281-torch291-ubuntu2404" \ + --gpu-id "NVIDIA A100 80GB PCIe" \ + --data-center-ids US-KS-2 \ + --network-volume-id NETWORK_VOLUME_ID \ + --volume-mount-path /workspace +runpodctl ssh info POD_ID +``` + +The Pod and the volume must be in the same data center, so confirm the GPU has stock there before you create the volume. Run your trainer over SSH (for example PEFT with the Transformers `Trainer`, or Axolotl), write checkpoints under `/workspace`, then delete the Pod with `runpodctl pod delete POD_ID`. The volume keeps the adapter. See [connect to a Pod](/pods/connect-to-a-pod) and the [fine-tuning overview](/fine-tune). + +Runpod has no managed counterpart to Fireworks' Serverless Training, reinforcement fine-tuning, or DPO jobs. You bring the training loop, and a Pod bills per second until you stop or delete it. + +## Step 6: Deploy on Runpod + +### Public Endpoint + +There's nothing to deploy. Confirm the model works for your prompts in the [Hub playground](https://console.runpod.io/hub?tabSelected=public_endpoints), then go to [Step 7](#step-7-update-the-code-that-calls-fireworks-ai). + +### vLLM queue endpoint from the Hub + +This is the default target for on-demand deployments and for serverless models Runpod doesn't host. The before and after below deploy Llama 3.1 8B Instruct with scale to zero and up to three replicas. + + + + ```bash + firectl deployment create accounts/fireworks/models/llama-v3p1-8b-instruct \ + --deployment-shape default \ + --min-replica-count 0 \ + --max-replica-count 3 \ + --scale-to-zero-window 10m \ + --load-targets concurrent_requests=4 \ + --wait + ``` + + + ```bash + runpodctl hub search vllm # prints the listing ID for runpod-workers/worker-vllm + + runpodctl serverless create \ + --name llama-31-8b \ + --hub-id HUB_ID \ + --gpu-id AMPERE_24 \ + --workers-min 0 \ + --workers-max 3 \ + --idle-timeout 600 \ + --scale-by delay \ + --scale-threshold 4 \ + --model-reference https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct:main \ + --env MODEL_NAME=meta-llama/Llama-3.1-8B-Instruct \ + --env MAX_MODEL_LEN=8192 \ + --env ENABLE_AUTO_TOOL_CHOICE=true \ + --env TOOL_CALL_PARSER=llama3_json \ + --env HF_TOKEN='{{ RUNPOD_SECRET_hf_token }}' + ``` + + + +What matters in the Runpod command: + +- `--hub-id` takes the listing ID that `runpodctl hub search vllm` returns. Replace `HUB_ID` with the ID from your own search output. +- `--gpu-id` accepts a pool ID or a GPU type ID. Always pass it. Without it, the endpoint uses the Hub release's default pools, which may be larger GPUs than you need. +- `--model-reference` attaches the cached model. It takes the full Hugging Face URL with a ref (`:main`, a tag, or a commit), and it's GPU only. Each endpoint supports one cached model. Keep `MODEL_NAME` pointed at the same repo. +- `--idle-timeout 600` mirrors the 10-minute scale-to-zero window. Lower it to pay for less idle time, or raise it (up to 3,600) to avoid cold starts. +- `--scale-by delay --scale-threshold 4` adds workers when requests wait longer than 4 seconds. Request count scaling (`--scale-by requests`) is the closer match to a `concurrent_requests` load target and is the recommended scaler for LLM workloads, but REST API v2 rejects an idle timeout on a queue-based endpoint that scales on request count. This example keeps the long idle timeout, so it scales on queue delay. If you prefer request count scaling, drop the idle timeout and use active workers to stay warm. +- `--workers-min` is the number of active workers. Set it to 1 or more to mirror `--min-replica-count` above 0. Active workers bill continuously. +- `HF_TOKEN` references a [Runpod secret](/pods/templates/secrets) named `hf_token`. Llama models are gated, so create the secret first. Public models don't need a token. +- `--env` applies with `--hub-id`. With `--template-id`, environment variables come from the template. + +The command prints the endpoint ID. In the console, the same deployment is **Hub** > **vLLM** > **Deploy**, with the model in the **Model** field. See the [vLLM quickstart](/serverless/vllm/get-started). + +A new endpoint isn't ready when the create call returns. Poll its health until a worker is ready. + +```bash +runpodctl serverless get ENDPOINT_ID +curl -s "https://api.runpod.ai/v2/ENDPOINT_ID/health" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" +``` + +The first deployment can take several minutes while hosts pull the worker image and the model. You aren't billed for a cached model download. + +### vLLM load balancing endpoint (alternative) + +Use this when you need a specific `vllm/vllm-openai` tag or want requests to reach vLLM directly with no queue. Create it with REST API v2. The `args` string holds the `vllm serve` arguments, and the three environment variables tell Runpod where the health check lives. + +```bash +curl -X POST "https://api.runpod.io/v2/serverless" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "qwen3-8b-lb", + "type": "LOAD_BALANCER", + "image": "vllm/vllm-openai:v0.32.0", + "args": "Qwen/Qwen3-8B --max-model-len 8192 --gpu-memory-utilization 0.90 --enable-auto-tool-choice --tool-call-parser hermes --reasoning-parser qwen3 --port 8000", + "env": { "PORT": "8000", "PORT_HEALTH": "8000", "HEALTH_CHECK_PATH": "/health" }, + "ports": ["8000/http"], + "disk": 50, + "gpu": { "pools": ["AMPERE_24"], "count": 1, "minCudaVersion": "13.0" }, + "workers": { "min": 0, "max": 3, "idleTimeout": 600 }, + "scaling": { "type": "REQUEST_COUNT", "requestCount": 4 }, + "flashboot": "FLASHBOOT", + "timeout": 600000 + }' +``` + +The OpenAI base URL for this endpoint type is `https://ENDPOINT_ID.api.runpod.ai/v1`, and the gateway checks your Runpod API key, so don't also set a vLLM API key. This image is a CUDA 13 build, which is why the request sets `minCudaVersion`. Weights download to the container disk, so size `disk` for the model plus headroom. Load balancing endpoints don't have a queue, `/run`, webhooks, or batch jobs. + +### Translate the rest of your deployment settings + +| `firectl deployment create` flag | `runpodctl serverless create` flag | REST API v2 field | +| --- | --- | --- | +| Model argument (`accounts/.../models/NAME`) | `--model-reference`, `--env MODEL_NAME=` | `env.MODEL_NAME` | +| `--deployment-shape`, `--accelerator-type` | `--gpu-id` | `gpu.pools` | +| `--accelerator-count` | `--gpu-count` | `gpu.count` | +| `--min-replica-count` | `--workers-min` | `workers.min` | +| `--max-replica-count` | `--workers-max` | `workers.max` | +| `--scale-to-zero-window`, `--scale-down-window` | `--idle-timeout` (seconds) | `workers.idleTimeout` | +| `--scale-up-window`, `--load-targets` | `--scale-by`, `--scale-threshold` | `scaling.type`, `scaling.queueDelay` or `scaling.requestCount`. On a queue-based endpoint, REST API v2 rejects `workers.idleTimeout` together with `REQUEST_COUNT` scaling. | +| `--region` | `--data-center-ids` | `dataCenterIds` | +| `--max-context-length` | `--env MAX_MODEL_LEN=` | `env.MAX_MODEL_LEN` | +| `--display-name` | `--name` | `name` | +| `--wait` | Poll `/health` | Poll `/health` | + +## Step 7: Update the code that calls Fireworks AI + +For chat and text completions, three values change: the base URL, the API key, and the model name. + +| | Fireworks AI | Runpod vLLM queue endpoint | Runpod Public Endpoint | +| --- | --- | --- | --- | +| Base URL | `https://api.fireworks.ai/inference/v1` | `https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1` | `https://api.runpod.ai/v2/SLUG/openai/v1` | +| API key | `FIREWORKS_API_KEY` | `RUNPOD_API_KEY` | `RUNPOD_API_KEY` | +| Model | `accounts/fireworks/models/NAME` or `accounts/ACCOUNT_ID/deployments/DEPLOYMENT_ID` | Hugging Face repo ID, or your `OPENAI_SERVED_MODEL_NAME_OVERRIDE` | The value on the model's page | + +Put the base URL and model name behind configuration so you can switch back during cutover. + +### Python (OpenAI SDK) + + + + ```python + import os + from openai import OpenAI + + client = OpenAI( + base_url="https://api.fireworks.ai/inference/v1", + api_key=os.environ["FIREWORKS_API_KEY"], + ) + + response = client.chat.completions.create( + model="accounts/fireworks/models/llama-v3p1-8b-instruct", + messages=[{"role": "user", "content": "Say hello in one word."}], + max_tokens=64, + ) + print(response.choices[0].message.content) + ``` + + + ```python + import os + from openai import OpenAI + + client = OpenAI( + base_url=f"https://api.runpod.ai/v2/{os.environ['RUNPOD_ENDPOINT_ID']}/openai/v1", + api_key=os.environ["RUNPOD_API_KEY"], + ) + + response = client.chat.completions.create( + model="meta-llama/Llama-3.1-8B-Instruct", + messages=[{"role": "user", "content": "Say hello in one word."}], + max_tokens=64, + ) + print(response.choices[0].message.content) + ``` + + + ```python + import os + from openai import OpenAI + + client = OpenAI( + base_url="https://api.runpod.ai/v2/qwen3-32b-awq/openai/v1", + api_key=os.environ["RUNPOD_API_KEY"], + ) + + response = client.chat.completions.create( + model="Qwen/Qwen3-32B-AWQ", + messages=[{"role": "user", "content": "Say hello in one word."}], + max_tokens=512, + ) + print(response.choices[0].message.content) + ``` + + + +Always set `max_tokens`. Qwen3 is a reasoning model, so a small `max_tokens` can end the response inside its thinking block. + +### Python (Fireworks SDK) + +The Fireworks SDK (`from fireworks import Fireworks`) exposes the same `chat.completions.create` call shape, so inference code moves to the OpenAI SDK with a changed import and constructor. + + + + ```python + import os + from fireworks import Fireworks + + client = Fireworks(api_key=os.environ["FIREWORKS_API_KEY"]) + + stream = client.chat.completions.create( + model="accounts/fireworks/models/llama-v3p1-8b-instruct", + messages=[{"role": "user", "content": "Write a haiku about queues."}], + max_tokens=128, + stream=True, + ) + for chunk in stream: + if chunk.choices and chunk.choices[0].delta.content: + print(chunk.choices[0].delta.content, end="", flush=True) + ``` + + + ```python + import os + from openai import OpenAI + + client = OpenAI( + base_url=f"https://api.runpod.ai/v2/{os.environ['RUNPOD_ENDPOINT_ID']}/openai/v1", + api_key=os.environ["RUNPOD_API_KEY"], + ) + + stream = client.chat.completions.create( + model="meta-llama/Llama-3.1-8B-Instruct", + messages=[{"role": "user", "content": "Write a haiku about queues."}], + max_tokens=128, + stream=True, + ) + for chunk in stream: + if chunk.choices and chunk.choices[0].delta.content: + print(chunk.choices[0].delta.content, end="", flush=True) + ``` + + + +Replace `fireworks-ai` with `openai` in your dependencies once no code imports it. Code that uses the Fireworks SDK for platform automation (`client.deployments.create(...)`, datasets, training jobs) moves to `runpodctl` or REST API v2. See the [API map](#api-map). + +### TypeScript + + + + ```typescript + import OpenAI from "openai"; + + const client = new OpenAI({ + apiKey: process.env.FIREWORKS_API_KEY, + baseURL: "https://api.fireworks.ai/inference/v1", + }); + + const response = await client.chat.completions.create({ + model: "accounts/fireworks/models/llama-v3p1-8b-instruct", + messages: [{ role: "user", content: "Say hello in one word." }], + max_tokens: 64, + }); + console.log(response.choices[0].message.content); + ``` + + + ```typescript + import OpenAI from "openai"; + + const client = new OpenAI({ + apiKey: process.env.RUNPOD_API_KEY, + baseURL: `https://api.runpod.ai/v2/${process.env.RUNPOD_ENDPOINT_ID}/openai/v1`, + }); + + const response = await client.chat.completions.create({ + model: "meta-llama/Llama-3.1-8B-Instruct", + messages: [{ role: "user", content: "Say hello in one word." }], + max_tokens: 64, + }); + console.log(response.choices[0].message.content); + ``` + + + +If you use the Vercel AI SDK, the [`@runpod/ai-sdk-provider`](/public-endpoints/ai-sdk) package works with Public Endpoints and with your own endpoint IDs. + +### curl + + + + ```bash + curl https://api.fireworks.ai/inference/v1/chat/completions \ + -H "Authorization: Bearer $FIREWORKS_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "model": "accounts/fireworks/models/llama-v3p1-8b-instruct", + "messages": [{"role": "user", "content": "Say hello in one word."}], + "max_tokens": 64 + }' + ``` + + + ```bash + curl https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1/chat/completions \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "model": "meta-llama/Llama-3.1-8B-Instruct", + "messages": [{"role": "user", "content": "Say hello in one word."}], + "max_tokens": 64 + }' + ``` + + + +### Tool calling + +The request and response format stays the OpenAI `tools` format. On your own endpoint, it works only when the worker has `ENABLE_AUTO_TOOL_CHOICE=true` and a `TOOL_CALL_PARSER` that matches the model, as set in Step 6. + +```python +tools = [{ + "type": "function", + "function": { + "name": "get_weather", + "description": "Get the current weather for a location", + "parameters": { + "type": "object", + "properties": {"location": {"type": "string", "description": "City name"}}, + "required": ["location"], + }, + }, +}] + +response = client.chat.completions.create( + model="meta-llama/Llama-3.1-8B-Instruct", + messages=[{"role": "user", "content": "What's the weather in San Francisco?"}], + tools=tools, + max_tokens=256, +) +print(response.choices[0].message.tool_calls) +``` + +If `tool_calls` comes back empty and the call appears as text in `content`, the parser doesn't match the model's tool format. + +### Feature parity for the OpenAI-compatible surface + +Each cell reflects what the platform's docs state. "Check the model" means the docs don't settle it, so test before you rely on it. + +| Feature | Fireworks AI | Runpod vLLM endpoint | Runpod Public Endpoints | +| --- | --- | --- | --- | +| `/chat/completions` | Yes | Yes | Yes, for the text models documented as OpenAI-compatible | +| `/completions` | Yes | Yes | Check the model | +| `/models` | Check the Fireworks API reference | Yes | Check the model | +| Streaming (`stream: true`) | Yes | Yes | Yes (documented for Qwen3, Kimi, GPT-OSS, Cogito) | +| `usage` in the final stream chunk | Yes, by default | Check the model | Check the model | +| Tool calling | Yes, on supported models | Yes, with `ENABLE_AUTO_TOOL_CHOICE` and `TOOL_CALL_PARSER`. Depends on the model. | Documented for GPT-OSS 120B. Check other models. | +| JSON mode and JSON schema (`response_format`) | Yes (`json_object`, `json_schema`) | Not covered in the Runpod docs. The worker exposes `GUIDED_DECODING_BACKEND`. Check the model. | Check the model | +| Grammar mode (BNF) | Yes | Check the model | Check the model | +| Reasoning | Yes (`reasoning_effort`, output in `reasoning_content`) | Yes, with `REASONING_PARSER`. Output in `reasoning` on vLLM v0.32.0. | Kimi documents `reasoning_effort`. Check other models. | +| Vision input (`image_url`) | Yes, on vision models | Depends on the model you deploy | Kimi K2.6 is described as multimodal. Check the model. | +| Embeddings | Yes (`/v1/embeddings`) | Not listed for the vLLM worker | Not listed | +| Reranking | Yes (`/v1/rerank`) | Not listed | Not listed | +| Responses API | Yes (`/v1/responses`) | Not listed | Not listed | +| Anthropic Messages API | Yes | Not listed | Not listed | +| LoRA adapters | Yes (deployments, and serverless for select bases) | Yes (`ENABLE_LORA`, `LORA_MODULES`) | Not applicable for text models | +| Async jobs | The Responses API can run jobs in the background. Chat completions are synchronous. | Yes (`/run`, `/status`, webhooks) | Yes (`/run`, `/status`) | + +### Batch and offline inference + +Fireworks' Batch API used JSONL files with `custom_id` and `body` per line. On a Runpod queue endpoint, each request is a job. The vLLM worker's native input takes `messages` and a `sampling_params` object. This script submits a Fireworks-format batch file as Runpod jobs and records which job belongs to which `custom_id`. + +```python submit_batch.py +import json +import os +import sys + +import requests + +ENDPOINT_ID = os.environ["RUNPOD_ENDPOINT_ID"] +HEADERS = { + "Authorization": f"Bearer {os.environ['RUNPOD_API_KEY']}", + "Content-Type": "application/json", +} +SAMPLING_KEYS = ("max_tokens", "temperature", "top_p", "top_k", "stop", "presence_penalty", "frequency_penalty") + +with open(sys.argv[1]) as batch_file, open("jobs.jsonl", "a") as jobs_file: + for line in batch_file: + item = json.loads(line) + body = item["body"] + payload = { + "input": { + "messages": body["messages"], + "sampling_params": {k: body[k] for k in SAMPLING_KEYS if k in body}, + }, + "webhook": os.environ.get("RESULT_WEBHOOK_URL"), + } + if payload["webhook"] is None: + del payload["webhook"] + response = requests.post( + f"https://api.runpod.ai/v2/{ENDPOINT_ID}/run", headers=HEADERS, json=payload, timeout=30 + ) + response.raise_for_status() + jobs_file.write(json.dumps({"custom_id": item["custom_id"], "job_id": response.json()["id"]}) + "\n") +``` + +Fetch each result with `GET https://api.runpod.ai/v2/ENDPOINT_ID/status/JOB_ID`, or receive it at the webhook URL. Async results are kept for 30 minutes after completion, so collect them promptly or use webhooks. `/run` accepts 1,000 requests per 10 seconds per endpoint, so pace large files. For very large offline workloads, [batch jobs](/serverless/batch-jobs) run on dedicated batch workers and take a JSON array of the same `{"input": ...}` objects. + +### Image and audio models + +Public Endpoints for image, video, and text-to-speech models use the native job format, not the OpenAI format. A request is `POST https://api.runpod.ai/v2/SLUG/runsync` (or `/run`) with a body of `{"input": {...}}`, and the result is under `output` (for example `output.image_url`). + +```bash +curl -X POST "https://api.runpod.ai/v2/black-forest-labs-flux-1-dev/runsync" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"input": {"prompt": "A serene mountain landscape at sunset", "width": 1024, "height": 1024}}' +``` + +Output URLs expire after 7 days, so download results you need to keep. Each model's page lists its parameters. + +### Control-plane automation + +Scripts and CI jobs that call `firectl` or the Fireworks REST API move to `runpodctl` or REST API v2. See the [CLI command map](#cli-command-map) and the [API map](#api-map). + +## Step 8: Verify, cut over, and shut down + + + + Use the same prompts your application sends, from the same network. For a cold endpoint, submit the first request asynchronously so the model load doesn't hit the synchronous timeout. + + ```bash + JOB=$(curl -s -X POST "https://api.runpod.ai/v2/ENDPOINT_ID/run" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" \ + -d '{"input": {"messages": [{"role": "user", "content": "Say hello in one word."}], "sampling_params": {"max_tokens": 16}}}' | jq -r .id) + curl -s "https://api.runpod.ai/v2/ENDPOINT_ID/status/$JOB" -H "Authorization: Bearer $RUNPOD_API_KEY" + ``` + + When the job is `COMPLETED`, a worker is warm and the OpenAI route responds normally. + + + Replay a sample of production prompts against both platforms. Compare output quality, tool-call correctness, JSON validity, time to first token, and tokens per second. Differences usually trace to quantization, chat template, or parser settings. + + + Run representative load, then read worker time from the endpoint's **Metrics** tab or `runpodctl billing serverless`. Tune the idle timeout, active workers, and GPU pool from that data. + + + Switch the base URL and model name through configuration for a small share of traffic, watch error rates and latency, then increase the share. Keep the Fireworks path callable until you're done. + + + After traffic has moved and you've downloaded any weights and datasets you need, stop the resources that bill. + + ```bash + firectl deployment list + firectl deployment scale DEPLOYMENT_ID --replica-count=0 # reversible: stop GPU billing first + firectl deployment delete DEPLOYMENT_ID # then delete + firectl model list + firectl model delete MODEL_ID # custom models and adapters + firectl dataset delete DATASET_ID + firectl reservation list # reservations bill until they end + firectl api-key list + firectl api-key delete --help # revoke keys the old path used + ``` + + Fireworks bills reserved capacity until the reservation ends whether or not you use it, so talk to your Fireworks account manager about any reservation. Serverless usage stops billing when you stop sending requests. + + + +On the Runpod side, guard cost with three settings: max workers caps concurrency and spend, active workers of 0 lets the endpoint scale to zero, and the execution timeout stops runaway jobs. Delete test endpoints with `runpodctl serverless delete ENDPOINT_ID`, and check for leftover network volumes with `runpodctl network-volume list`. + +## Reference + +### CLI command map + +| `firectl` | `runpodctl` | +| --- | --- | +| `firectl signin`, `firectl set-api-key API_KEY` | `runpodctl doctor`, or `runpodctl config --apiKey YOUR_API_KEY` | +| `firectl whoami` | `runpodctl user` | +| `firectl deployment create MODEL --deployment-shape default` | `runpodctl serverless create --hub-id HUB_ID --gpu-id POOL --model-reference URL` | +| `firectl deployment list` | `runpodctl serverless list` | +| `firectl deployment get DEPLOYMENT_ID` | `runpodctl serverless get ENDPOINT_ID --include-template --include-workers` | +| `firectl deployment update DEPLOYMENT_ID --max-replica-count 5` | `runpodctl serverless update ENDPOINT_ID --workers-max 5` | +| `firectl deployment scale DEPLOYMENT_ID --replica-count=0` | `runpodctl serverless update ENDPOINT_ID --workers-min 0` (scales to zero when idle, and starts a worker on the next request) | +| `firectl deployment delete DEPLOYMENT_ID` | `runpodctl serverless delete ENDPOINT_ID` | +| `firectl deployment-shape-version match --model MODEL` | `runpodctl gpu list` (you choose the hardware) | +| `firectl model create MODEL_ID /path` | `hf upload`, or upload to a network volume | +| `firectl model load-lora ADAPTER --deployment DEPLOYMENT` | Set `LORA_MODULES` on the endpoint | +| `firectl dataset create`, `firectl dataset download` | `runpodctl network-volume create` plus the S3-compatible API | +| `firectl supervised-fine-tuning-job create` | `runpodctl pod create` and your own trainer | +| `firectl secret create --name NAME --value VALUE` | Create a secret in the console, reference it as `{{ RUNPOD_SECRET_NAME }}` | +| `firectl quota list` | `runpodctl gpu list`, `runpodctl datacenter list` | +| `firectl billing get-usage` | `runpodctl billing serverless`, `runpodctl billing pods` | +| `firectl version`, `firectl upgrade` | `runpodctl version`, `runpodctl update` | + +### API map + +| Fireworks AI | Runpod | +| --- | --- | +| `POST https://api.fireworks.ai/inference/v1/chat/completions` | `POST https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1/chat/completions` | +| `POST https://api.fireworks.ai/inference/v1/completions` | `POST https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1/completions` | +| `POST https://api.fireworks.ai/inference/v1/embeddings`, `/rerank`, `/responses` | No prebuilt route. See [What has no direct equivalent](#what-has-no-direct-equivalent). | +| `POST https://api.fireworks.ai/v1/accounts/ACCOUNT_ID/deployments` | `POST https://api.runpod.io/v2/serverless` | +| List, get, update, and delete deployment | `GET /v2/serverless`, `GET`, `PATCH`, and `DELETE /v2/serverless/ENDPOINT_ID` | +| Scale deployment | `PATCH /v2/serverless/ENDPOINT_ID` (`workers.min`, `workers.max`) | +| `GET https://api.fireworks.ai/v1/accounts/ACCOUNT_ID/metrics` (Prometheus) | `GET /v2/serverless/ENDPOINT_ID/metrics/requests`, `/metrics/cold-starts`, `/metrics/workers` | +| Billing usage | `GET /v2/billing/serverless`, `GET /v2/billing/endpoints` | +| Batch inference jobs | `POST https://api.runpod.ai/v2/ENDPOINT_ID/batch`, or `/run` | +| `client.deployments.create(...)` (Fireworks Python SDK) | `POST /v2/serverless` or `runpodctl serverless create` | + +Note the two Runpod hosts: `api.runpod.ai` serves requests to endpoints, and `api.runpod.io` serves the management API. Both take `Authorization: Bearer RUNPOD_API_KEY`. + +### Environment variable map + +These are client-side variables, since Fireworks doesn't run your containers. + +| Fireworks AI | Runpod | +| --- | --- | +| `FIREWORKS_API_KEY` | `RUNPOD_API_KEY` | +| `OPENAI_BASE_URL` or `OPENAI_API_BASE` set to `https://api.fireworks.ai/inference/v1` | `https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1` | +| `OPENAI_API_KEY` set to a Fireworks key | Set to a Runpod API key | +| `ANTHROPIC_BASE_URL` set to `https://api.fireworks.ai/inference` | No equivalent route | +| `~/.fireworks/auth.ini` (`firectl` credentials) | `~/.runpod/config.toml` (written by `runpodctl doctor` or `rp login`) | + +### Status map + +| Fireworks deployment state | Runpod equivalent | +| --- | --- | +| `CREATING` | Workers **Initializing** (pulling the image, downloading cached models) | +| `READY` | `/health` reports a ready worker. Workers are **Idle** or **Running**. | +| `READY`, shown as "Scaled to 0" | All workers **Idle** (scaled down, not billed) | +| `UPDATING` | A new endpoint release rolling out. Old workers show **Outdated**. | +| `FAILED` | Workers **Unhealthy**. Read the worker logs. | +| `PARTIAL_READY` (some replicas waiting on capacity) | Some workers **Throttled** | +| `DELETING`, `DELETED` | Endpoint deleted | + +Request-level states on a Runpod queue endpoint are `IN_QUEUE`, `IN_PROGRESS`, `COMPLETED`, `FAILED`, `CANCELLED`, and `TIMED_OUT`. Fireworks chat completion requests are synchronous and have no job state. + +## Migrate with a coding agent + +Install the Runpod skills and MCP server in your agent. MCP sign-in uses OAuth ("Sign in with Runpod"), so you don't need an API key to start. + + + + ```bash + # The plugin also connects the hosted MCP server + claude plugin marketplace add runpod/runpod-plugins-official + claude plugin install runpod@runpod + ``` + + + ```bash + # Codex, Cursor, and others: skills, then the hosted MCP server + npx skills add runpod/runpod-plugins-official --skill '*' --agent AGENT_ID --global + npx @runpod/mcp-server@latest add --client CLIENT_ID + ``` + + + +See [agent skills](/get-started/agent-skills) and [MCP servers](/get-started/mcp-servers) for agent and client IDs. Then give the agent this prompt from the root of your repository. + +```text +Use the runpod-cloud-migrate skill to migrate this project from Fireworks AI to Runpod. + +1. Inventory first. Scan the repo for Fireworks usage (api.fireworks.ai, accounts/*/models/*, + accounts/*/deployments/*, FIREWORKS_API_KEY, the fireworks-ai SDK, firectl) and, if firectl + is signed in, list deployments, models, deployed LoRA adapters, datasets, and training jobs. + Treat Fireworks as read-only. +2. For each model, tell me whether Runpod hosts it as a Public Endpoint or whether it needs + its own vLLM Serverless endpoint, which Hugging Face repo you'd serve, and which features + each call site uses (streaming, tools, response_format, vision, reasoning, embeddings, + rerank, the Responses API). +3. Show me a plan with the GPU pool, worker counts, idle timeout, and an hourly cost bound + for each endpoint. List every gap that needs my decision. Wait for my approval before + creating anything billable. +4. Migrate one workload: deploy it, put the base URL and model name behind configuration, + and verify with a real request (chat, streaming, and a tool call if the code uses tools). +5. Report what you created, what it costs while running, and what is still billing. + Never scale down or delete Fireworks deployments, models, or datasets without my + explicit approval. +``` + +## Troubleshooting + +**The first request returns a 524 or times out.** The synchronous routes have an edge timeout of about 100 seconds, and the first request to a cold worker waits for the model to load. Send the first request with `/run` and poll `/status`, raise the idle timeout so workers stay warm between requests, or set one active worker. + +**"Invalid model" or a 404 on the model name.** You sent the Fireworks ID. The `model` value must be the Hugging Face repo the worker serves or your `OPENAI_SERVED_MODEL_NAME_OVERRIDE` value. List what the endpoint serves with `client.models.list()`. + +**Responses are cut off after a few tokens.** Fireworks adjusts `max_tokens` to the context window for you. The Runpod vLLM docs list a default `max_tokens` of 16. Set it on every request. + +**Tool calls come back as plain text.** The worker is missing `ENABLE_AUTO_TOOL_CHOICE=true`, or `TOOL_CALL_PARSER` doesn't match the model's tool format. See the [model-specific settings](/serverless/vllm/configuration#model-specific-configurations). + +**The reasoning trace disappeared.** Your client reads `reasoning_content`. vLLM v0.32.0 returns `reasoning`. Read both, and set `REASONING_PARSER` on the worker. Without a parser, thinking text arrives inside `content`. + +**Workers stay throttled or never start.** The selected pool has no free GPUs in the allowed data centers, or the endpoint's CUDA constraint excludes the available hosts. Add a second GPU pool, remove data center restrictions, and check stock with `runpodctl gpu list`. A network volume restricts the endpoint to one data center. + +**The worker crashes while loading the model.** Usually out of memory. vLLM reserves VRAM for the KV cache on top of the weights. Lower `MAX_MODEL_LEN`, use a quantized repo, or move to a larger pool. Gated models fail to download without a valid `HF_TOKEN`. + +**Output quality differs from Fireworks.** Check that the Hugging Face repo matches the checkpoint and precision Fireworks served, and that the chat template is the model's own. For fine-tunes, confirm you merged the adapter into the exact base it was trained on. + +**The bill is higher than expected at low traffic.** A long idle timeout or an active worker bills for time with no requests. Lower the idle timeout, set active workers to 0, or move low-volume models to a Public Endpoint if one is listed. + +**429 responses.** On Runpod, a 429 is the endpoint's request-rate limit, not a sign that workers are saturated. Queue-based endpoints hold excess requests in the queue, so watch queue delay and raise max workers. + +## Next steps + + + + Deploy and configure OpenAI-compatible LLM endpoints. + + + Browse the models Runpod hosts and their parameters. + + + Tune workers, idle timeout, and autoscaling. + + + Cut cold starts by caching Hugging Face models on hosts. + + diff --git a/migrate/google-cloud.mdx b/migrate/google-cloud.mdx new file mode 100644 index 000000000..fa9560f80 --- /dev/null +++ b/migrate/google-cloud.mdx @@ -0,0 +1,1446 @@ +--- +title: "Migrate from Google Cloud to Runpod" +sidebarTitle: "Google Cloud" +description: "Move GPU training, inference, and batch workloads from Compute Engine, Vertex AI, Cloud Run, and GKE to Runpod, and keep the rest on Google Cloud." +--- + +This guide is for teams that run GPU workloads on Google Cloud and want to move the GPU-heavy part (training, inference, batch) to Runpod. Most teams keep their data, identity, and the rest of the application on Google Cloud, so the guide covers that hybrid setup first and a full exit second. [Compute Engine](https://docs.cloud.google.com/compute/docs/gpus) GPU VMs map to [Pods](/pods/overview), [Vertex AI](https://docs.cloud.google.com/vertex-ai/docs/predictions/get-online-predictions) endpoints and [Cloud Run](https://docs.cloud.google.com/run/docs/configuring/services/gpu) GPU services map to [Serverless](/serverless/overview) endpoints, managed model APIs map to [Public Endpoints](/public-endpoints/overview), and Slurm clusters map to [Instant Clusters](/instant-clusters/slurm-clusters). When you finish, your workload runs on Runpod, your callers use a Runpod API key, and the Google Cloud resources you replaced are shut down. + + +You can hand this migration to a coding agent. See [Migrate with a coding agent](#migrate-with-a-coding-agent). + + + +Google's documentation now calls Vertex AI "Gemini Enterprise Agent Platform". The `gcloud ai` commands, the Vertex AI SDK for Python, the `aiplatform.googleapis.com` API, and the Terraform `google_vertex_ai_*` resources keep their names, so this guide says Vertex AI throughout. + + +## How Google Cloud maps to Runpod + +| Google Cloud concept | Runpod equivalent | Notes | +| --- | --- | --- | +| Compute Engine GPU VM (A2, A3, A4, G2, G4, or N1 with attached GPUs) | [Pod](/pods/overview) | A Pod is a container, not a VM. You bring a `linux/amd64` image and Runpod provides the GPU drivers. | +| Deep Learning VM images, Deep Learning Containers | Runpod PyTorch images and [templates](/pods/templates/overview) | Official Runpod PyTorch images start SSH and JupyterLab for you. | +| Spot VMs | On-demand Pods | The Runpod CLI and REST API v2 create on-demand Pods. Design for capacity checks instead of preemption. | +| Reservations, committed use discounts, flex-start (Dynamic Workload Scheduler) | On-demand Pods, queued Pod creates, and sales-assisted pricing | REST API v2 can queue a Pod create until capacity appears (`queue` on `POST /v2/pods`). | +| Managed instance groups with instance templates | Serverless endpoint (autoscaled workers) | For request-driven fleets. For fixed fleets, create Pods from one template. | +| Vertex AI online inference (Model, Endpoint, DeployedModel) | Serverless [load-balancing endpoint](/serverless/load-balancing/overview) or [queue endpoint](/serverless/endpoints/overview) | Load balancing keeps your HTTP server. Queue endpoints give you a job queue with retries. | +| Vertex AI batch inference | Queue endpoint with `/run` and webhooks | You submit one job per input (or per chunk) and collect results by webhook or `/status`. | +| Vertex AI custom training jobs | Pod, or `@app.task` in the [Runpod SDK](/sdk/overview) | `@app.task` provisions one Pod per call and terminates it when the function returns. | +| Vertex AI Workbench, Colab Enterprise | Pod with JupyterLab | See [Connect to a Pod](/pods/connect-to-a-pod). | +| Model Garden model APIs, the OpenAI-compatible Chat Completions endpoint | [Public Endpoints](/public-endpoints/overview) or a Serverless [vLLM endpoint](/serverless/vllm/overview) | Public Endpoints cover only the models listed in the [model reference](/public-endpoints/reference). | +| Cloud Run service with a GPU | Serverless load-balancing endpoint | Closest match. See [What works differently](#what-works-differently) for the contract differences. | +| Cloud Run job, Batch job | Queue endpoint or `@app.task` | Use a queue endpoint for many short tasks, `@app.task` for run-to-completion jobs. | +| GKE GPU node pools, Autopilot GPU Pods | Serverless endpoints or Pods, or [SkyPilot](/integrations/skypilot) and [dstack](/integrations/dstack) | Runpod runs containers directly. There's no Kubernetes API. | +| Cluster Toolkit Slurm clusters | [Instant Clusters with Slurm](/instant-clusters/slurm-clusters) | Stock depends on GPU type and region. Check before you plan around it. | +| Artifact Registry | Any registry, pulled with [registry credentials](/runpodctl/reference/runpodctl-registry) | You can keep images in Artifact Registry. | +| Persistent Disk, Hyperdisk boot and data disks | Container disk and volume disk | Container disk is ephemeral. Volume disk persists across stops and is deleted with the Pod. | +| Filestore, Managed Lustre | [Network volume](/storage/network-volumes), including [high-performance storage](/storage/high-performance-storage) | A network volume lives in one data center. | +| Cloud Storage, Cloud Storage FUSE mounts | Keep Cloud Storage, or use a network volume or [global volume](/storage/globalvolume/overview) | Cloud Storage stays reachable from Runpod with a service account key or HMAC key. | +| VPC, firewall rules, load balancers | [Exposed ports](/pods/configuration/expose-ports), [global networking](/pods/networking), Serverless endpoint URLs | See [What has no direct equivalent](#what-has-no-direct-equivalent). | +| Secret Manager | [Runpod secrets](/pods/templates/secrets) | Reference a secret in an environment variable as `{{ RUNPOD_SECRET_name }}`. | +| Regions and zones | [Data centers](/runpodctl/reference/runpodctl-datacenter) | IDs look like `US-GA-1` or `EU-RO-1`. GPU stock differs by data center. | +| Per-second VM billing, node-hour billing for deployed models, network data transfer charges | Per-second billing for compute and storage | Runpod doesn't charge for data ingress or egress. See [Pod pricing](/pods/pricing) and [Serverless pricing](/serverless/pricing). | +| Terraform `hashicorp/google` provider | Terraform [`runpod/runpod` provider](/integrations/terraform/overview) 2.x | Built on REST API v2. Both providers work in one configuration. | +| `gcloud`, Cloud client libraries, REST APIs | [`runpodctl`](/runpodctl/reference/runpodctl-pod), the [Runpod SDK](/sdk/overview), [REST API v2](/api-reference-v2/overview), the [MCP server](/get-started/mcp-servers) | | +| IAM principals, service accounts, access tokens | One [API key](/get-started/credentials) per user or integration | Keys can be restricted by permission. | +| Organizations, folders, projects, IAM roles | [Organizations](/organizations/orgs-overview) with roles and groups | Organizations are in beta and provisioned by Runpod. | +| Cloud Logging, Cloud Monitoring | Pod and worker logs, endpoint metrics | `runpodctl pod logs`, `runpodctl serverless logs`, and the metrics tab on each endpoint. | + +## What's different + +### What works differently + +**Pods are containers, not VMs.** A Compute Engine VM boots an OS image and runs a startup script as root on every boot. A Pod starts your container image and runs its start command. Move startup-script logic into the image or the Pod's start command. Pods don't run Docker Compose, Windows, or UDP services. + +**Health checks are required for load-balancing endpoints.** Vertex AI sends health checks to the route in `AIP_HEALTH_ROUTE` and Cloud Run only needs your server to listen on `PORT`. A Runpod load-balancing worker must answer `GET /ping` (or the path in `HEALTH_CHECK_PATH`) with `200` when ready and `204` while it's still loading. Any other code marks the worker unhealthy. + +**Ports default to 80, not 8080.** Vertex AI sets `AIP_HTTP_PORT` to `8080` by default and Cloud Run sets `PORT` to `8080` by default. Runpod load-balancing workers read `PORT` and `PORT_HEALTH`, which both default to `80`. Set both to your server's port and expose that port on the endpoint. + +**Request shape for queue endpoints.** Vertex AI sends `{"instances": [...], "parameters": {...}}` and expects `{"predictions": [...]}`. A Runpod queue endpoint takes `{"input": {...}}` and returns your handler's return value under `output`. On a load-balancing endpoint the request and response bodies are whatever your server defines, so you can keep the Vertex AI shape. + +**Limits on load-balancing endpoints.** Cloud Run requests time out after 5 minutes by default and can be extended to 60 minutes. Vertex AI shared public endpoints time out at 60 seconds and dedicated endpoints are configurable up to 1 hour. Runpod load-balancing endpoints allow 5.5 minutes of processing per request and 30 MB per request or response, and they don't queue: a request that finds no available worker within 2 minutes fails. For longer work, or work that must not be dropped, use a queue endpoint, where the execution timeout can be raised up to 7 days. + +**Scale to zero is the default.** A Runpod endpoint starts with zero active workers and scales up on demand. Vertex AI deployments default to a minimum of one replica, and setting the minimum to `0` opts in to scale to zero, which Google lists as a Preview feature. On Runpod, set active (minimum) workers above zero only when you need to remove cold starts, because active workers bill continuously. + +**Concurrency is expressed through the scaler.** Cloud Run has a per-instance concurrency setting (up to 1,000). On Runpod you set the `REQUEST_COUNT` scaler value, and Runpod sizes the worker count as in-flight requests divided by that value. Your server handles the concurrent requests it receives. + +**Authentication is an API key.** Google Cloud callers use OAuth access tokens (Vertex AI) or ID tokens (private Cloud Run services), usually through Application Default Credentials. Every Runpod Serverless request carries `Authorization: Bearer YOUR_API_KEY`, and Runpod checks the key before it routes the request. For traffic from browsers or anonymous users, put your own API or proxy in front so the key stays server-side. + +**GPU selection is by GPU type or pool, not machine type.** A Pod takes one GPU type ID and a GPU count. A Serverless endpoint takes one or more GPU pools, and workers land on whichever pool has capacity. vCPU and RAM come with the GPU instead of being a separate machine-type choice. On Pods you can set a floor with `min_ram_per_gpu` and `min_vcpu_count_per_gpu` placement filters. + +**Storage is tied to a data center.** A network volume lives in one data center, and anything that mounts it must run there. That narrows GPU choice the same way a zonal Persistent Disk does. A global volume removes the constraint for read-heavy data such as model weights. + +**Model artifacts.** Vertex AI copies `artifactUri` to a managed location and exposes it as `AIP_STORAGE_URI`. On Runpod, bake weights into the image, load them from a network or global volume, or use [cached models](/serverless/endpoints/model-caching) for Hugging Face models. + +### What has no direct equivalent + +**On Runpod:** + +- **VPC networks, firewall rules, and Private Service Connect.** Runpod has no customer-defined private network or firewall rules. Pods get [global networking](/pods/networking) (a private network between your own Pods, at `POD_ID.runpod.internal`) and explicitly [exposed ports](/pods/configuration/expose-ports). HTTP ports are served through a public HTTPS proxy URL that isn't authenticated by Runpod, so add authentication in your app. If you need a private path into your Google Cloud VPC, run a VPN or proxy that you control, or keep that workload on Google Cloud. +- **IAM service accounts, Workload Identity, and the metadata server.** A Pod or worker has no Google identity and no `metadata.google.internal`. To call Google Cloud APIs from Runpod, create a least-privilege service account, store its key as a Runpod secret, and load it at startup. Where a caller only needs one object, hand it a signed URL instead of a key. +- **Customer-managed encryption keys (CMEK).** You can encrypt a Pod's volume disk at creation, with a key Runpod manages. Bring-your-own-key isn't supported, and container disks and network volumes can't be encrypted this way. +- **Spot pricing through the CLI or REST API v2.** `runpodctl` and REST API v2 create on-demand Pods. Checkpoint to a volume and plan capacity with `runpodctl gpu list` rather than relying on a spot fallback. +- **Traffic splitting between model versions on one endpoint.** Vertex AI splits traffic across the DeployedModels on an endpoint. On Runpod, run two endpoints and split in your client or gateway. Rolling out a new image to one endpoint is covered in [rolling releases](/serverless/endpoints/rolling-releases). +- **Managed batch input and output.** Vertex AI batch inference reads JSON Lines, CSV, or BigQuery and writes results to Cloud Storage or BigQuery. On Runpod you write the loop: read inputs, submit `/run` jobs, and write results where you want them. +- **A Kubernetes API.** There's no control plane to `kubectl apply` against. Use Serverless endpoints, Pods, or an orchestrator such as SkyPilot or dstack. + +**On Google Cloud:** + +- **A built-in job queue for inference.** Runpod queue endpoints accept `/run`, hold jobs while workers scale, retry failures, and call a webhook on completion. Vertex AI online inference and Cloud Run services are request and response only. +- **No data transfer fees.** Runpod doesn't charge for ingress or egress. +- **GPU pools with fallback.** A Serverless endpoint can list several GPU pools and place workers wherever stock exists. + +Compliance posture also differs. Secure Cloud runs in T3/T4 data centers operated by vetted partners that meet standards including SOC 2, ISO 27001, and PCI DSS, and `runpodctl pod create --compliance` restricts placement to data centers holding the certifications you name. Review [Security and compliance](/references/security-and-compliance) against your own requirements before you move regulated data. + +## Requirements + +- A Google Cloud project with the [gcloud CLI](https://docs.cloud.google.com/sdk/docs/install) authenticated and permission to list and delete the resources you're migrating. +- A [Runpod account](https://www.console.runpod.io) with a payment method or credits. Deploying an on-demand Pod requires at least one hour of credits for that configuration. +- Docker, if you need to rebuild images. +- Terraform 1.6 or later, if you manage infrastructure as code. + +## Step 1: Take inventory of what you run on Google Cloud + +List every GPU resource, export its configuration, and find the code that calls Google Cloud. Start with a project-wide search so nothing is missed, then drill into each product. + +```bash +gcloud asset search-all-resources \ + --scope=projects/PROJECT_ID \ + --format="table(name, assetType, location)" +``` + +This lists every searchable resource in the project. Narrow it with `--asset-types`, for example `compute.googleapis.com/Instance`. + +Next, export the details for each product you use. Each command prints what you need to recreate the workload: machine and accelerator type, image, environment, ports, disks, region, and scaling limits. + +```bash +# Compute Engine: VMs, disks, templates, managed instance groups +gcloud compute instances list \ + --format="table(name, zone.basename(), machineType.basename(), scheduling.provisioningModel, status)" +gcloud compute instances describe VM_NAME --zone=ZONE --format=json > vm.json +gcloud compute disks list +gcloud compute instance-templates list +gcloud compute instance-groups managed list + +# Vertex AI: endpoints, deployed models, models, training jobs (repeat per region) +gcloud ai endpoints list --region=LOCATION_ID +gcloud ai endpoints describe ENDPOINT_ID --region=LOCATION_ID --format=json > endpoint.json +gcloud ai models list --region=LOCATION_ID +gcloud ai custom-jobs list --region=LOCATION_ID + +# Cloud Run: services and jobs +gcloud run services list +gcloud run services describe SERVICE --region=REGION --format=export > service.yaml +gcloud run jobs list + +# GKE, images, storage, secrets, notebooks +gcloud container clusters list +gcloud artifacts repositories list +gcloud storage ls +gcloud filestore instances list +gcloud secrets list +gcloud workbench instances list --location=ZONE +``` + +In `vm.json`, read `machineType`, `guestAccelerators`, `disks`, `scheduling`, `serviceAccounts`, and the `startup-script` key under `metadata`. In `endpoint.json`, each entry under `deployedModels` has `dedicatedResources` with the machine type, accelerator, and `minReplicaCount` and `maxReplicaCount`. In `service.yaml`, read the container image, `ports`, `env`, resource limits (including `nvidia.com/gpu`), the `run.googleapis.com/accelerator` node selector, concurrency, timeout, and scaling annotations. + +Then find the code and configuration that depends on Google Cloud. + +```bash +grep -rnE \ + "from google\.cloud import aiplatform|import vertexai|aiplatform\.googleapis\.com|\.run\.app|gcloud (ai|run|compute|storage|batch)|gs://|docker\.pkg\.dev|gcr\.io|AIP_[A-Z_]+|K_SERVICE|CLOUD_RUN_[A-Z_]+|metadata\.google\.internal|GOOGLE_APPLICATION_CREDENTIALS|/gcs/|google_(compute_instance|cloud_run_v2|vertex_ai)" \ + --include="*.py" --include="*.ts" --include="*.js" --include="*.go" --include="*.sh" \ + --include="*.yaml" --include="*.yml" --include="*.tf" --include="Dockerfile*" . +``` + +Each match is either a client call to update in [Step 7](#step-7-update-the-code-that-calls-google-cloud), a runtime assumption to change in [Step 4](#step-4-move-your-images-and-code), or infrastructure to translate in [Migrate your Terraform](#migrate-your-terraform). + +## Step 2: Set up Runpod + +Create an API key on the [Credentials page](/get-started/credentials), then install the tools. + +```bash +# runpodctl (or: brew install runpod/runpodctl/runpodctl) +curl -fsSL https://cli.runpod.net | sudo bash + +# Runpod SDK for Python 3.10 to 3.14, which includes the rp CLI +pip install runpod + +# Store your API key in ~/.runpod/config.toml +rp login +``` + +`rp login` opens a browser to approve access. Tools also read the `RUNPOD_API_KEY` environment variable, which is the simplest option in CI. + +If you'll connect to Pods over SSH, add your public key on the Credentials page first. If you manage infrastructure as code, set up the [Terraform provider](/integrations/terraform/overview). To let a coding agent manage Runpod resources, connect the [MCP server](/get-started/mcp-servers). + +## Step 3: Choose where each workload lands + +Pick a Runpod target for each workload from your inventory. + +| Google Cloud workload | Runpod target | Why | +| --- | --- | --- | +| Interactive GPU VM, Workbench instance, Colab Enterprise runtime | Pod | Long-lived, SSH and JupyterLab access, persistent `/workspace`. | +| Training on a GPU VM or a Vertex AI custom job | Pod, or `@app.task` | `@app.task` gives you run-to-completion with automatic teardown. | +| Vertex AI endpoint or Cloud Run GPU service with short, synchronous requests | Serverless load-balancing endpoint | Keeps your HTTP server and routes. | +| Vertex AI endpoint whose requests run for minutes, or that must never drop a request | Serverless queue endpoint | Queueing, retries, async `/run`, webhooks. | +| Vertex AI batch inference, Cloud Run jobs, Batch jobs | Serverless queue endpoint, or `@app.task` | Fan out with `/run`. Use `@app.task` for one large job. | +| Self-hosted LLM on Vertex AI, Cloud Run, or GKE | Serverless [vLLM endpoint](/serverless/vllm/get-started) | OpenAI-compatible route with no server code to write. | +| Calls to Model Garden open models | [Public Endpoint](/public-endpoints/overview), if the model is listed | No deployment. Otherwise deploy the model on a vLLM endpoint. | +| GKE GPU Deployment behind a Service | Serverless load-balancing endpoint | One container per worker. Sidecars need to be merged into the image. | +| GKE batch Jobs, multi-cloud schedulers | Queue endpoint, or SkyPilot or dstack on Pods | | +| Slurm cluster from Cluster Toolkit, multi-node training | Instant Cluster | Check stock for your GPU type and region first. | + +Check the [Public Endpoints reference](/public-endpoints/reference) for the hosted models Runpod offers. Keep calls to models that aren't listed there on Google Cloud, or move them to an open model you deploy yourself. + +### Map your GPUs + +Use this table to translate Compute Engine accelerator types to Runpod GPU type IDs (used by Pods) and Serverless GPU pools. Choose by fit for the workload: VRAM first, then GPU generation. + +| Google Cloud accelerator type | Machine family | VRAM per GPU | Runpod GPU type ID | Serverless pool | +| --- | --- | --- | --- | --- | +| `nvidia-tesla-t4` | N1 (attached) | 16 GB | `NVIDIA RTX A4000` or `NVIDIA RTX 2000 Ada Generation` (16 GB class) | `AMPERE_16` | +| `nvidia-tesla-v100` | N1 (attached) | 16 GB | `Tesla V100-PCIE-16GB`, `Tesla V100-SXM2-16GB` | Check the catalog | +| `nvidia-l4` | G2 | 24 GB | `NVIDIA L4` | `AMPERE_24` | +| `nvidia-tesla-a100` | A2 Standard | 40 GB | `NVIDIA A40` or `NVIDIA RTX A6000` (48 GB), or an 80 GB A100 | `AMPERE_48` or `AMPERE_80` | +| `nvidia-a100-80gb` | A2 Ultra | 80 GB | `NVIDIA A100 80GB PCIe`, `NVIDIA A100-SXM4-80GB` | `AMPERE_80` | +| `nvidia-h100-80gb`, `nvidia-h100-mega-80gb` | A3 High, A3 Edge, A3 Mega | 80 GB | `NVIDIA H100 80GB HBM3`, `NVIDIA H100 PCIe` | `ADA_80_PRO` | +| `nvidia-h200-141gb` | A3 Ultra | 141 GB | `NVIDIA H200` | `HOPPER_141` | +| `nvidia-b200` | A4 | 180 GB | `NVIDIA B200` | Check the catalog | +| `nvidia-gb200` | A4X | 186 GB | `NVIDIA B200` | Check the catalog | +| `nvidia-gb300` | A4X Max | 279 GB | `NVIDIA B300 SXM6 AC` | Check the catalog | +| `nvidia-rtx-pro-6000` | G4 (`g4-standard-48` and larger) | 96 GB | `NVIDIA RTX PRO 6000 Blackwell Server Edition` | Check the catalog | +| `nvidia-rtx-pro-6000` (fractional) | G4 (`g4-standard-12`, `g4-standard-24`) | 24 or 48 GB | `NVIDIA RTX PRO 6000 Blackwell Server Edition MIG 1g.24gb` or `MIG 2g.48gb` | Check the catalog | + +Machine types that carry several GPUs, such as `a3-highgpu-8g` or `a2-ultragpu-4g`, become one Pod with a GPU count (`--gpu-count 8`) or an Instant Cluster when you need several nodes. + +GPU stock varies by data center and changes through the day. Before you commit to a type, check live stock with `runpodctl gpu list` and `runpodctl datacenter list`, or call `GET /v2/catalog/gpus`, where each GPU's `pool` field gives its Serverless pool ID. The full list of IDs is in the [GPU types reference](/references/gpu-types). + +## Step 4: Move your images and code + +### Check image portability + +Runpod runs `linux/amd64` images. Cloud Run and Compute Engine GPU VMs are also x86-64, so most images run unchanged. If you build on an Arm machine, build for the right platform. + +```bash +docker build --platform linux/amd64 -t us-central1-docker.pkg.dev/PROJECT_ID/REPO/IMAGE:TAG . +``` + +Runpod provides the NVIDIA driver on the host. Your image supplies CUDA user-space libraries (for example through a PyTorch wheel or an `nvidia/cuda` base image). Set a CUDA floor that matches your image when you create Pods or endpoints (`--min-cuda-version 12.8`), so the host driver is new enough. + +If your VM used a Deep Learning VM image (for example the `pytorch-2-9-cu129-ubuntu-2204-nvidia-580` family in the `deeplearning-platform-release` project), the nearest starting point is an official Runpod PyTorch image such as `runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04`. Install anything else your startup script added. + +### Pull private images from Artifact Registry + +You can leave images in Artifact Registry and let Runpod pull them. Google documents two username and password methods for Docker: a service account key (username `_json_key`, or `_json_key_base64` for a base64-encoded key) and an access token (username `oauth2accesstoken`). Access tokens last 60 minutes, so they don't work for a stored credential. Use a service account key that can only read the repository. + + + + Create a service account and grant it the Artifact Registry Reader role on the repository. + + ```bash + gcloud iam service-accounts create runpod-puller --display-name="Runpod image pull" + + gcloud artifacts repositories add-iam-policy-binding REPO \ + --location=us-central1 \ + --member="serviceAccount:runpod-puller@PROJECT_ID.iam.gserviceaccount.com" \ + --role="roles/artifactregistry.reader" + ``` + + + Create a JSON key for the service account. + + ```bash + gcloud iam service-accounts keys create runpod-puller.json \ + --iam-account=runpod-puller@PROJECT_ID.iam.gserviceaccount.com + ``` + + Google describes service account keys as the least secure Docker authentication option because they're long-lived. Scope this account to image pulls only, and rotate the key on a schedule. + + + Pass the key file on standard input with the literal username `_json_key`. + + ```bash + runpodctl registry create --name "artifact-registry" \ + --username "_json_key" --password-stdin < runpod-puller.json + ``` + + The command prints the credential ID. The same credential can be created with `POST /v2/registries` or the `runpod_registry` Terraform resource. + + + Use the full image path and tag, and pass the credential ID when you create a Pod or template. + + ```bash + runpodctl pod create \ + --image "us-central1-docker.pkg.dev/PROJECT_ID/REPO/IMAGE:TAG" \ + --registry-auth-id REGISTRY_AUTH_ID \ + --gpu-id "NVIDIA L4" + ``` + + + +Delete `runpod-puller.json` from your machine after you store it. If you'd rather not keep a Google key on Runpod, push the image to another registry or [deploy from GitHub](/serverless/workers/github-integration). + +### Translate a Vertex AI custom container + +A Vertex AI custom container is an HTTP server that reads its port and routes from `AIP_*` environment variables. This is a typical FastAPI server for Vertex AI. + +```python main.py +import os + +from fastapi import FastAPI, Request +from sentence_transformers import SentenceTransformer + +app = FastAPI() +model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2", device="cuda") + +HEALTH_ROUTE = os.environ.get("AIP_HEALTH_ROUTE", "/health") +PREDICT_ROUTE = os.environ.get("AIP_PREDICT_ROUTE", "/predict") + + +@app.get(HEALTH_ROUTE) +def health(): + return {"status": "ok"} + + +@app.post(PREDICT_ROUTE) +async def predict(request: Request): + body = await request.json() + instances = body["instances"] + parameters = body.get("parameters", {}) + embeddings = model.encode( + [instance["text"] for instance in instances], + normalize_embeddings=parameters.get("normalize", True), + ) + return {"predictions": embeddings.tolist()} + + +if __name__ == "__main__": + import uvicorn + + uvicorn.run(app, host="0.0.0.0", port=int(os.environ.get("AIP_HTTP_PORT", "8080"))) +``` + +The server binds to `AIP_HTTP_PORT`, answers health checks on `AIP_HEALTH_ROUTE`, and serves the `instances` and `parameters` request shape on `AIP_PREDICT_ROUTE`. + +You have two ways to run this on Runpod. + + + + Keep the FastAPI server. Read the port from `PORT`, add a `/ping` route that reports readiness, and use fixed paths. + + ```python main.py + import os + import threading + + from fastapi import FastAPI, Request, Response + from sentence_transformers import SentenceTransformer + + app = FastAPI() + model = None + + + def load_model(): + global model + model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2", device="cuda") + + + threading.Thread(target=load_model, daemon=True).start() + + + @app.get("/ping") + def ping(): + # 204 tells the load balancer the worker is still initializing. + if model is None: + return Response(status_code=204) + return {"status": "healthy"} + + + @app.post("/predict") + async def predict(request: Request): + body = await request.json() + instances = body["instances"] + parameters = body.get("parameters", {}) + embeddings = model.encode( + [instance["text"] for instance in instances], + normalize_embeddings=parameters.get("normalize", True), + ) + return {"predictions": embeddings.tolist()} + + + if __name__ == "__main__": + import uvicorn + + uvicorn.run(app, host="0.0.0.0", port=int(os.environ.get("PORT", "80"))) + ``` + + The request and response bodies are unchanged, so callers only change the URL and the auth header. Loading the model in a background thread lets `/ping` return `204` during startup, which is how Runpod measures the cold start of a load-balancing worker. + + Package it with a Dockerfile. + + ```dockerfile Dockerfile + FROM runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04 + + WORKDIR /app + RUN pip install --no-cache-dir fastapi uvicorn sentence-transformers + COPY main.py . + + ENV PORT=8080 + ENV PORT_HEALTH=8080 + CMD ["python", "main.py"] + ``` + + The image sets `PORT` and `PORT_HEALTH` to the same value. Set both on the endpoint too and expose `8080/http`, as shown in [Step 6](#load-balancing-endpoint). + + + To run the original image with no code change, set `PORT` and `PORT_HEALTH` to the port the server listens on (`8080` when `AIP_HTTP_PORT` is unset) and set `HEALTH_CHECK_PATH` to your health route, for example `/health`. Requests to `https://ENDPOINT_ID.api.runpod.ai/predict` then reach the predict route unchanged. + + + + Replace the server with a handler function. Runpod delivers each job's `input` to the handler and returns what it returns. + + ```python handler.py + import runpod + from sentence_transformers import SentenceTransformer + + model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2", device="cuda") + + + def handler(job): + job_input = job["input"] + instances = job_input["instances"] + parameters = job_input.get("parameters", {}) + embeddings = model.encode( + [instance["text"] for instance in instances], + normalize_embeddings=parameters.get("normalize", True), + ) + return {"predictions": embeddings.tolist()} + + + runpod.serverless.start({"handler": handler}) + ``` + + The model loads once at import, when the worker starts. Callers wrap the old request body in `input` and read the result from `output`. + + ```dockerfile Dockerfile + FROM runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04 + + WORKDIR /app + RUN pip install --no-cache-dir runpod sentence-transformers + COPY handler.py . + + CMD ["python", "-u", "handler.py"] + ``` + + A queue worker needs no port or health route. See [Handler functions](/serverless/workers/handler-functions) for streaming, progress updates, and error handling. + + + +If your model loads from `AIP_STORAGE_URI`, change it to read from a volume path (see [Step 5](#step-5-move-your-data-and-models)) or download from Cloud Storage at startup with a service account key. + +### Translate a Cloud Run GPU service + +A Cloud Run service already follows most of the load-balancing contract: it's an HTTP server that listens on `0.0.0.0` at the port in `PORT`. Make these changes: + +- **Add a health route.** Serve `GET /ping`, or set `HEALTH_CHECK_PATH` to a route you already have. Return `204` until the model is loaded. +- **Set the port explicitly.** Set `PORT` and `PORT_HEALTH` to your server's port (Cloud Run's default is `8080`) and expose it. +- **Replace Google identity.** `K_SERVICE`, `K_REVISION`, and the metadata server aren't present. Code that fetches tokens from `metadata.google.internal` or relies on the service's attached service account needs a key from a Runpod secret. +- **Check request duration and size.** Requests are limited to 5.5 minutes of processing and 30 MB. Move longer requests to a queue endpoint. +- **Replace Cloud Storage volume mounts.** A `cloud-storage` volume becomes a network or global volume at `/runpod-volume`, or explicit Cloud Storage client calls. + +A Cloud Run job doesn't listen on a port. It runs to completion and reads `CLOUD_RUN_TASK_INDEX` and `CLOUD_RUN_TASK_COUNT` to shard work. On Runpod, wrap the per-task work in a queue handler and submit one `/run` request per shard, passing the index in `input`. + +### Translate a Compute Engine startup script + +A startup script runs as root on every boot of the VM. On a Pod, the equivalent is the container start command. Put one-time setup (package installs, driver-independent configuration) in the image, and put per-boot work in the start command. + +Official Runpod PyTorch images start SSH and JupyterLab from `/start.sh`. If you override the start command, call `/start.sh` at the end so those services still start. + +```bash +bash -c 'bash /workspace/startup.sh && /start.sh' +``` + +This runs your script from the persistent volume, then hands off to the image's own startup. Skip GPU driver installation steps from your old script: the driver is already on the host. + +### Train with the Runpod SDK instead of a custom job + +A Vertex AI custom job runs a container on a worker pool and exits. With the Runpod SDK, `@app.task` runs a Python function on an ephemeral Pod with the same lifecycle. + +```python train.py +import runpod +from runpod import App, NetworkVolume + +app = App("training") +data = NetworkVolume("training-data", size=200) + + +@app.task(gpu="H100", gpu_count=1, mounts={"/data": data}, dependencies=["torch", "transformers"]) +def finetune(epochs: int = 3): + import torch + + # Read the dataset from /data and write checkpoints back to /data/checkpoints. + return {"device": torch.cuda.get_device_name(0), "epochs": epochs} + + +@runpod.local_entrypoint +def main(): + print(finetune.remote(epochs=3)) +``` + +Run it with `rp flash dev train.py`. Checkpoints written to `/data` replace `AIP_MODEL_DIR` and `AIP_CHECKPOINT_DIR`, and `/gcs/BUCKET` paths become paths on the mounted volume. To keep your existing training container instead, run it as a Pod with your training command as the start command. + +## Step 5: Move your data and models + +Decide first whether the data needs to move. In a hybrid setup, datasets and artifacts can stay in Cloud Storage and your Runpod workloads read them over the public API with a service account key or HMAC key stored as a [Runpod secret](/pods/templates/secrets). Move data when startup time matters (model weights) or when the workload reads it constantly (training data). + + +Google Cloud bills network usage for data that leaves Cloud Storage or Compute Engine for a destination outside Google Cloud, and colder storage classes can add retrieval charges. Estimate the one-time transfer from Google's pricing pages before you start. Runpod doesn't charge for ingress or egress. + + +| Google Cloud storage | Runpod target | Notes | +| --- | --- | --- | +| Boot disk, local SSD scratch | Container disk | Ephemeral. Cleared when the Pod stops. | +| Persistent Disk or Hyperdisk data disk on one VM | Volume disk at `/workspace` | Persists across stops. Deleted with the Pod. | +| Filestore, Managed Lustre | Network volume | Shared across Pods and workers in one data center. Choose the high-performance tier where it's offered. | +| Cloud Storage bucket of model weights, Cloud Storage FUSE mount | Global volume, network volume, or cached models | Global volumes mount in any data center and suit read-heavy data. | +| Model artifacts from Hugging Face | [Cached models](/serverless/endpoints/model-caching) | Runpod stages the model on the host before the worker starts. | + +Create the volume in the data center where you'll run compute. Check GPU stock there first, because everything that mounts a network volume runs in that data center. + +```bash +runpodctl network-volume create --name "models" --size 200 --data-center-id "US-KS-2" +``` + +Then copy the data with one of these methods. + + + + Start a Pod with the volume attached, install the [gcloud CLI](https://docs.cloud.google.com/sdk/docs/install) in it, and pull straight from the bucket. This is the most direct path for large datasets because the data goes from Cloud Storage to the volume without passing through your machine. + + ```bash + # On the Pod. The key comes from a Runpod secret mapped to GCP_SA_KEY. + echo "$GCP_SA_KEY" > /tmp/key.json + gcloud auth activate-service-account --key-file=/tmp/key.json + + gcloud storage rsync gs://BUCKET/models /workspace/models --recursive + rm /tmp/key.json + ``` + + `gcloud storage rsync` copies only what's missing or changed, so you can rerun it after an interruption. Add `--dry-run` to preview. If you prefer rclone, `rclone copy :gcs:BUCKET/models /workspace/models --gcs-service-account-file /tmp/key.json` does the same job. + + + [Cloud Sync](/pods/storage/cloud-sync) copies between a running Pod and a Cloud Storage bucket from the Runpod console. Open the Pod, click **Cloud Sync**, select **Google Cloud Storage**, paste the service account JSON key, and choose the bucket path and Pod folders. + + + If the data is already on a workstation or CI runner, upload it without a Pod. This uses the volume's [S3-compatible API](/storage/s3-api), so the volume must be in a [data center that supports it](/storage/s3-api#datacenter-availability). Create an S3 API key on the Credentials page and run `aws configure` with it, as described in [Setup and authentication](/storage/s3-api#setup-and-authentication). + + ```bash + gcloud storage rsync gs://BUCKET/models ./models --recursive + + aws s3 sync ./models s3://VOLUME_ID/models \ + --region US-KS-2 \ + --endpoint-url https://s3api-us-ks-2.runpod.io/ + ``` + + Use the region and endpoint URL listed for your volume's data center. Rerun the `sync` command to resume an interrupted transfer. `aws s3 sync` can fail on directories with more than 10,000 files, so split very large trees into batches or copy from inside a Pod instead. + + + +The network volume S3-compatible API speaks the S3 protocol at `https://s3api-DATACENTER.runpod.io/` with the volume ID as the bucket. Drive it with S3 clients such as the AWS CLI or Boto3. It isn't a Cloud Storage API, so `gcloud storage` and the Cloud Storage client libraries can't target it. + +To fill a global volume, attach it to a Pod and copy into its mount path the same way. Global volumes have no S3-compatible API. + +### Keep reading from Cloud Storage + +For data that stays in Cloud Storage, give the workload a key that can only read what it needs. + +1. Create a service account with `roles/storage.objectViewer` on the specific bucket. +2. Create a JSON key, or an HMAC key if your code uses an S3-style client against `https://storage.googleapis.com`. +3. Store the key as a Runpod secret, for example `gcp_sa_key`. +4. Map it to an environment variable on the Pod or endpoint: `GCP_SA_KEY={{ RUNPOD_SECRET_gcp_sa_key }}`. + +Load the key in code instead of relying on Application Default Credentials from a metadata server. + +```python +import json +import os + +from google.cloud import storage +from google.oauth2 import service_account + +credentials = service_account.Credentials.from_service_account_info( + json.loads(os.environ["GCP_SA_KEY"]) +) +client = storage.Client(credentials=credentials, project=credentials.project_id) +client.bucket("BUCKET").blob("models/config.json").download_to_filename("/tmp/config.json") +``` + +The key never touches the image or the template in plain text. Runpod substitutes the secret when the Pod or worker boots. + +## Step 6: Deploy on Runpod + +### Pod + +This replaces a Compute Engine GPU VM. Here's a typical VM with one L4, a Deep Learning VM image, and a startup script. + +```bash +gcloud compute instances create trainer \ + --machine-type g2-standard-8 \ + --zone us-central1-a \ + --boot-disk-size 200GB \ + --image-family pytorch-2-9-cu129-ubuntu-2204-nvidia-580 \ + --image-project deeplearning-platform-release \ + --maintenance-policy TERMINATE \ + --metadata-from-file=startup-script=startup.sh +``` + +The Runpod equivalent creates a Pod with the same GPU, a network volume for data, and a start command in place of the startup script. + +```bash +runpodctl pod create \ + --name trainer \ + --image "runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04" \ + --gpu-id "NVIDIA L4" \ + --gpu-count 1 \ + --container-disk-in-gb 50 \ + --network-volume-id VOLUME_ID \ + --data-center-ids "US-KS-2" \ + --min-cuda-version 12.8 \ + --ports "8888/http,22/tcp" \ + --env '{"GCS_BUCKET":"BUCKET","GCP_SA_KEY":"{{ RUNPOD_SECRET_gcp_sa_key }}"}' \ + --docker-args "bash -c 'bash /workspace/startup.sh && /start.sh'" +``` + +The network volume mounts at `/workspace`, so `startup.sh` must already be on it (copy it there in Step 5) or be baked into the image. The data center must be the one that holds the volume. In the console, the same flow is **Pods > Deploy**. + +Wait for the Pod to be ready, then connect. + +```bash +until [ "$(runpodctl pod get POD_ID --output json | jq -r '.runtimeStatus')" = "running" ]; do + sleep 5 +done +runpodctl ssh info POD_ID +``` + +HTTP ports are served at `https://POD_ID-8888.proxy.runpod.net`. The official PyTorch images start JupyterLab on port 8888 only when `JUPYTER_PASSWORD` is set, so add it to `--env` if you want JupyterLab on a Pod created with `runpodctl`. To create the same Pod over REST, send `POST https://api.runpod.io/v2/pods` with `name`, `image`, `gpu` (`id`, `count`, `minCudaVersion`), `disk`, `ports`, `env`, `args`, `dataCenterIds`, and `mounts.network`. See [Create a Pod](/api-reference-v2/pods/create-a-pod). + +### Load-balancing endpoint + +This replaces a Cloud Run GPU service or a Vertex AI endpoint that serves short requests. Here's the Cloud Run deployment you're replacing. + +```bash +gcloud run deploy embedder \ + --image us-central1-docker.pkg.dev/PROJECT_ID/REPO/embedder:v1 \ + --region us-central1 \ + --gpu 1 --gpu-type nvidia-l4 \ + --cpu 8 --memory 32Gi \ + --no-cpu-throttling \ + --port 8080 \ + --concurrency 8 \ + --min-instances 0 --max-instances 3 \ + --timeout 300 \ + --no-allow-unauthenticated +``` + +Load-balancing endpoints are created with REST API v2, Terraform, or the console, because the endpoint type is set at creation. This request creates the equivalent endpoint. + +```bash +curl -s -X POST https://api.runpod.io/v2/serverless \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "embedder", + "type": "LOAD_BALANCER", + "image": "us-central1-docker.pkg.dev/PROJECT_ID/REPO/embedder:v1", + "registry": "REGISTRY_AUTH_ID", + "gpu": {"pools": ["AMPERE_24"], "count": 1, "minCudaVersion": "12.8"}, + "disk": 20, + "ports": ["8080/http"], + "env": {"PORT": "8080", "PORT_HEALTH": "8080"}, + "workers": {"min": 0, "max": 3, "idleTimeout": 10}, + "scaling": {"type": "REQUEST_COUNT", "requestCount": 8}, + "flashboot": "FLASHBOOT", + "timeout": 300000 + }' +``` + +The field mapping from the `gcloud run deploy` flags is direct: `--gpu-type nvidia-l4` becomes the `AMPERE_24` pool, `--port` becomes `ports` plus `PORT` and `PORT_HEALTH`, `--concurrency` becomes `scaling.requestCount`, and `--min-instances` and `--max-instances` become `workers.min` and `workers.max`. `type` and `scaling` are required, and a load-balancing endpoint only accepts `REQUEST_COUNT` scaling. There's no `--cpu` or `--memory` equivalent because they come with the GPU. + +The response includes the endpoint ID and `requestUrls.base`, which is `https://ENDPOINT_ID.api.runpod.ai`. Send a request once a worker is ready. + +```bash +curl -s https://api.runpod.ai/v2/ENDPOINT_ID/health -H "Authorization: Bearer $RUNPOD_API_KEY" + +curl -s -X POST https://ENDPOINT_ID.api.runpod.ai/predict \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"instances": [{"text": "hello"}], "parameters": {"normalize": true}}' +``` + +The first request after scale to zero waits for a cold start. If it returns `no workers available`, retry after a few seconds, or poll `/health` until a worker reports ready. + +### Queue endpoint + +This replaces Vertex AI endpoints with long-running requests, batch inference, Cloud Run jobs, and Batch jobs. Create a Serverless template for the handler image from Step 4, then an endpoint that uses it. + +```bash +runpodctl template create --name "embedder-queue" --serverless \ + --image "us-central1-docker.pkg.dev/PROJECT_ID/REPO/embedder-queue:v1" \ + --registry-auth-id REGISTRY_AUTH_ID \ + --container-disk-in-gb 20 \ + --env '{"GCP_SA_KEY":"{{ RUNPOD_SECRET_gcp_sa_key }}"}' + +runpodctl serverless create --name "embedder-queue" \ + --template-id TEMPLATE_ID \ + --gpu-id AMPERE_24 \ + --workers-min 0 --workers-max 3 \ + --min-cuda-version 12.8 \ + --execution-timeout 600 +``` + +The endpoint scales between zero and three workers on 24 GB GPUs. Each Serverless template binds to one endpoint, so create a template per endpoint. To attach a network volume, add `--network-volume-id`. It mounts at `/runpod-volume` on each worker. + +Submit a job and read the result. + +```bash +curl -s -X POST https://api.runpod.ai/v2/ENDPOINT_ID/runsync \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"input": {"instances": [{"text": "hello"}], "parameters": {"normalize": true}}}' +``` + +For batch work, post each input to `/run` with a `webhook` URL and let Runpod call you when each job completes. See [Send API requests](/serverless/endpoints/send-requests). + +### Public Endpoint or vLLM endpoint + +To replace calls to a Model Garden open model, check whether the model is in the [Public Endpoints reference](/public-endpoints/reference). If it is, there's nothing to deploy. If it isn't, deploy it on a vLLM endpoint from the Hub. + +```bash +runpodctl hub search vllm +runpodctl serverless create --hub-id HUB_ID --name "my-llm" \ + --env MODEL_NAME=meta-llama/Llama-3.1-8B-Instruct +``` + +The endpoint exposes an OpenAI-compatible route at `https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1`. See [Get started with vLLM](/serverless/vllm/get-started) for gated models and tuning. + +### Instant Cluster + +To replace a Slurm cluster built with Cluster Toolkit, create a Slurm cluster from the [Instant Clusters page](https://console.runpod.io/cluster) with your GPU type, node count, region, and an optional network volume in the same region. Slurm and munge are preconfigured, and `sbatch`, `squeue`, and `sinfo` work on the controller node. Managed Slurm clusters use official Runpod PyTorch images. Availability depends on the GPU type and region, so confirm it in the console before you schedule a cutover. See [Slurm Clusters](/instant-clusters/slurm-clusters). + +## Step 7: Update the code that calls Google Cloud + +### Vertex AI online inference + +Calls through the Vertex AI SDK authenticate with Google credentials and send `instances`. + +```python +from google.cloud import aiplatform + +aiplatform.init(project="PROJECT_ID", location="us-central1") +endpoint = aiplatform.Endpoint("VERTEX_ENDPOINT_ID") + +prediction = endpoint.predict( + instances=[{"text": "hello"}], + parameters={"normalize": True}, +) +print(prediction.predictions) +``` + +The same applies to `gcloud ai endpoints predict` and to REST calls against `https://LOCATION_ID-aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/LOCATION_ID/endpoints/ENDPOINT_ID:predict`. + +On Runpod, the call is an HTTPS request with a Bearer API key. + + + + The body is unchanged because your server defines it. + + ```python + import os + + import requests + + response = requests.post( + "https://ENDPOINT_ID.api.runpod.ai/predict", + headers={"Authorization": f"Bearer {os.environ['RUNPOD_API_KEY']}"}, + json={"instances": [{"text": "hello"}], "parameters": {"normalize": True}}, + timeout=120, + ) + response.raise_for_status() + print(response.json()["predictions"]) + ``` + + Add a retry for the first request after scale to zero, when no worker is ready yet. + + + Wrap the body in `input` and read the result from `output`. + + ```python + import os + + import requests + + response = requests.post( + "https://api.runpod.ai/v2/ENDPOINT_ID/runsync", + headers={"Authorization": f"Bearer {os.environ['RUNPOD_API_KEY']}"}, + json={"input": {"instances": [{"text": "hello"}], "parameters": {"normalize": True}}}, + timeout=120, + ) + response.raise_for_status() + result = response.json() + print(result["status"], result["output"]["predictions"]) + ``` + + With the Runpod SDK, the same call is `runpod.Endpoint("ENDPOINT_ID").run_sync({"instances": [...], "parameters": {...}})`. + + + +### Long-running and batch requests + +Vertex AI batch inference takes a `BatchPredictionJob` with a Cloud Storage or BigQuery source. On Runpod, submit jobs to `/run` and collect results by polling or webhook. + +```python +import os + +import requests + +BASE_URL = "https://api.runpod.ai/v2/ENDPOINT_ID" +HEADERS = {"Authorization": f"Bearer {os.environ['RUNPOD_API_KEY']}"} + +job = requests.post( + f"{BASE_URL}/run", + headers=HEADERS, + json={ + "input": {"instances": [{"text": "hello"}]}, + "webhook": "https://example.com/runpod-webhook", + }, + timeout=30, +).json() + +status = requests.get(f"{BASE_URL}/status/{job['id']}", headers=HEADERS, timeout=30).json() +print(status["status"]) +``` + +Job statuses are `IN_QUEUE`, `IN_PROGRESS`, `COMPLETED`, `FAILED`, `CANCELLED`, and `TIMED_OUT`. Results from `/run` are kept for 30 minutes after completion, so write large outputs to Cloud Storage or a volume from inside the handler and return a reference. + +### Private Cloud Run services + +A caller of a private Cloud Run service fetches an ID token for the service URL. + +```python +import google.auth.transport.requests +import google.oauth2.id_token +import requests + +SERVICE_URL = "https://embedder-abc123-uc.a.run.app" + +auth_request = google.auth.transport.requests.Request() +token = google.oauth2.id_token.fetch_id_token(auth_request, SERVICE_URL) + +response = requests.post( + f"{SERVICE_URL}/predict", + headers={"Authorization": f"Bearer {token}"}, + json={"instances": [{"text": "hello"}]}, + timeout=120, +) +``` + +On Runpod, drop the token exchange and send the API key. The request is the load-balancing example above with the base URL `https://ENDPOINT_ID.api.runpod.ai`. Callers that still run on Google Cloud should read the Runpod API key from Secret Manager. + +### OpenAI-compatible calls + +If you call open models through Google's OpenAI-compatible endpoint, the client authenticates with a short-lived Google access token. + +```python +import google.auth.transport.requests +import openai +from google.auth import default + +credentials, _ = default(scopes=["https://www.googleapis.com/auth/cloud-platform"]) +credentials.refresh(google.auth.transport.requests.Request()) + +client = openai.OpenAI( + base_url="https://aiplatform.googleapis.com/v1/projects/PROJECT_ID/locations/global/endpoints/openapi", + api_key=credentials.token, +) +response = client.chat.completions.create( + model="openai/gpt-oss-120b-maas", + messages=[{"role": "user", "content": "Hello"}], +) +``` + +On Runpod, change the base URL, API key, and model name. This example uses the GPT OSS 120B Public Endpoint. + +```python +import os + +import openai + +client = openai.OpenAI( + base_url="https://api.runpod.ai/v2/gpt-oss-120b/openai/v1", + api_key=os.environ["RUNPOD_API_KEY"], +) +response = client.chat.completions.create( + model="openai/gpt-oss-120b", + messages=[{"role": "user", "content": "Hello"}], +) +``` + +The API key doesn't expire hourly, so remove the token refresh logic. For a model you deployed on a vLLM endpoint, use `https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1` and the model name the endpoint serves. + +### Control-plane automation + +Replace `gcloud` calls and Google API client calls in scripts and CI with `runpodctl`, REST API v2, or the Runpod SDK. The [CLI command map](#cli-command-map) and [API map](#api-map) list the equivalents. In CI, set `RUNPOD_API_KEY` as a secret in place of a service account key or Workload Identity Federation configuration. + +## Step 8: Verify, cut over, and shut down + + + + Call the Runpod endpoint from the same network location as your production callers, using production-shaped inputs. For Pods, connect over SSH and run the workload. + + + Send the same inputs to both platforms and compare the outputs. Measure cold-start and warm latency separately. On Runpod, the endpoint's **Metrics** tab shows execution time, delay time, and cold-start percentiles. Tune the idle timeout, active workers, and FlashBoot before you shift traffic. + + + Keep the Google Cloud deployment serving while you route a small share of traffic to Runpod from your client or gateway. Watch error rates and logs with `runpodctl serverless logs ENDPOINT_ID`. + + + Move the rest of the traffic. Set `workers.max` to the highest concurrency you intend to pay for, keep active workers at zero unless you need them, and enable [low balance notifications](/accounts-billing/billing#low-balance-notifications). Runpod accounts also have a default hourly spend limit, described in [Account spend limits](/accounts-billing/manage-accounts#account-spend-limits). + + + Stopping a VM or scaling a service down doesn't stop every charge. Work through the checklist below. + + + +### Google Cloud shutdown checklist + +Delete only what you've replaced, and take a final backup of anything you might need. + +```bash +# Vertex AI: deployed models bill while deployed. Undeploy, then delete the endpoint. +gcloud ai endpoints describe ENDPOINT_ID --region=LOCATION_ID +gcloud ai endpoints undeploy-model ENDPOINT_ID --region=LOCATION_ID --deployed-model-id=DEPLOYED_MODEL_ID +gcloud ai endpoints delete ENDPOINT_ID --region=LOCATION_ID + +# Cloud Run services and jobs +gcloud run services delete SERVICE --region=REGION +gcloud run jobs delete JOB --region=REGION + +# Compute Engine VMs +gcloud compute instances delete VM_NAME --zone=ZONE + +# Persistent disks and snapshots that outlive the VM +gcloud compute disks list +gcloud compute disks delete DISK_NAME --zone=ZONE +gcloud compute snapshots list +gcloud compute snapshots delete SNAPSHOT_NAME + +# Reserved static external IP addresses +gcloud compute addresses list +gcloud compute addresses delete ADDRESS_NAME --region=REGION + +# Cloud NAT gateways +gcloud compute routers nats list --router=ROUTER_NAME --region=REGION +gcloud compute routers nats delete NAT_NAME --router=ROUTER_NAME --region=REGION + +# Workbench instances +gcloud workbench instances list --location=ZONE +gcloud workbench instances delete INSTANCE_NAME --location=ZONE + +# Filestore instances +gcloud filestore instances list +gcloud filestore instances delete INSTANCE_NAME --zone=ZONE + +# Artifact Registry repositories (only if Runpod no longer pulls from them) +gcloud artifacts repositories list +gcloud artifacts repositories delete REPO --location=LOCATION +``` + +Points that are easy to miss: + +- **Stopped VMs still bill for disks and IP addresses.** Compute Engine bills attached disks and external IP addresses in every instance state, including `TERMINATED`. Suspended VMs also bill for memory. +- **Disks can survive VM deletion.** A disk that isn't set to auto-delete stays after the VM is gone. +- **A Vertex AI endpoint can't be deleted until every model is undeployed from it.** Uploaded models in Model Registry and their artifacts in Cloud Storage remain until you delete them. +- **Cloud Run minimum instances bill while idle.** Deleting the service removes them. +- **Managed instance groups recreate VMs.** Delete the group, not the instances. +- **GKE node pools.** Delete GPU node pools or the cluster if nothing else uses them. +- **Reservations and committed use discounts.** Reservations bill until deleted, and commitments run to the end of their term. Review them with your account team. +- **Cloud Storage.** Keep buckets your Runpod workloads still read, and delete the rest. + +Confirm the result in the Cloud Billing reports a day or two later, filtered by SKU. + +## Migrate your Terraform + +The [Runpod Terraform provider](/integrations/terraform/overview) 2.x (`runpod/runpod`) is built on REST API v2. It runs alongside `hashicorp/google` in one configuration, which fits a hybrid setup: Google Cloud keeps buckets, service accounts, and Artifact Registry, and Runpod runs the GPU compute. + +### Resource map + +| Google Cloud resource | Runpod resource | Notes | +| --- | --- | --- | +| `google_compute_instance` (accelerator-optimized machine type or `guest_accelerator`) | `runpod_pod` | `gpu_type_id` and `gpu_count` replace `machine_type` and `guest_accelerator`. | +| `metadata_startup_script` on `google_compute_instance` | `args` on `runpod_pod` | A container start command, not a boot script. | +| `scheduling` (`provisioning_model`, `on_host_maintenance`) | No argument | Pods are on-demand. | +| `google_compute_instance_template` with `google_compute_instance_group_manager` | `runpod_template` with `runpod_serverless_endpoint` | `workers_min` and `workers_max` replace the group size and autoscaler. | +| `google_vertex_ai_endpoint` and its deployed models | `runpod_serverless_endpoint` | One resource covers the endpoint, the container, and scaling. | +| `google_vertex_ai_endpoint_with_model_garden_deployment` | `runpod_serverless_endpoint` running vLLM | Or no resource, if a Public Endpoint serves the model. | +| `google_vertex_ai_custom_job` | `runpod_pod`, or `@app.task` outside Terraform | Run-to-completion jobs fit the SDK better than Terraform state. | +| `google_cloud_run_v2_service` with a GPU | `runpod_template` and `runpod_serverless_endpoint` (`endpoint_type = "LOAD_BALANCER"`) | See the example below. | +| `google_cloud_run_v2_job` | `runpod_serverless_endpoint` (`endpoint_type = "QUEUE"`) | | +| `google_compute_disk` | `disk` and `persistent_volume` on `runpod_pod`, or `runpod_network_volume` | Use a network volume for data that must outlive the Pod. | +| `google_filestore_instance` | `runpod_network_volume` | `type = "HIGH_PERFORMANCE"` where the data center offers it. | +| `google_storage_bucket` | Keep it, or `runpod_global_volume` for model weights | | +| `google_artifact_registry_repository` | Keep it, and add `runpod_registry` | The credential holds a service account key. | +| `google_secret_manager_secret` | Runpod secret, referenced in `env` | Create the secret in the console or with `POST /v2/account/secrets`. | +| `google_container_node_pool` (GPU) | `runpod_serverless_endpoint`, `runpod_pod`, or `runpod_cluster` | | +| `google_compute_network`, `google_compute_firewall`, `google_compute_router_nat` | No resource | Use `ports` and `global_networking` on `runpod_pod`. | +| `google_service_account`, IAM bindings | No resource | Runpod access is by API key and organization role. | + +### Run both providers side by side + +Declare both providers. During the transition, the Google provider keeps managing the resources you haven't moved, and it can create the pull-only service account that Runpod uses for Artifact Registry. + +```hcl providers.tf +terraform { + required_version = ">= 1.6" + + required_providers { + google = { + source = "hashicorp/google" + } + runpod = { + source = "runpod/runpod" + version = "~> 2.0" + } + } +} + +provider "google" { + project = var.project_id + region = "us-central1" +} + +# Reads RUNPOD_API_KEY from the environment. +provider "runpod" {} + +variable "project_id" { + type = string +} +``` + +Keep your existing version constraint for the Google provider. With both declared, wire Artifact Registry to Runpod in the same configuration. + +```hcl registry.tf +resource "google_service_account" "runpod_puller" { + account_id = "runpod-puller" + display_name = "Runpod image pull" +} + +resource "google_artifact_registry_repository_iam_member" "runpod_puller" { + project = var.project_id + location = "us-central1" + repository = "REPO" + role = "roles/artifactregistry.reader" + member = "serviceAccount:${google_service_account.runpod_puller.email}" +} + +resource "google_service_account_key" "runpod_puller" { + service_account_id = google_service_account.runpod_puller.name +} + +resource "runpod_registry" "artifact_registry" { + name = "artifact-registry" + username = "_json_key_base64" + password = google_service_account_key.runpod_puller.private_key +} +``` + +`google_service_account_key.private_key` is the JSON key, base64 encoded, which is the format Artifact Registry expects with the `_json_key_base64` username. The key is stored in Terraform state, so use an encrypted remote backend with restricted access. `runpod_registry` has no update operation, so rotating the key replaces the credential. + +### Compute Engine GPU instance to a Pod + +This is a GPU VM with a data disk and a startup script. + +```hcl before.tf +resource "google_compute_disk" "data" { + name = "trainer-data" + type = "pd-ssd" + zone = "us-central1-a" + size = 200 +} + +resource "google_service_account" "trainer" { + account_id = "trainer" + display_name = "Trainer VM" +} + +resource "google_compute_instance" "trainer" { + name = "trainer" + machine_type = "g2-standard-8" + zone = "us-central1-a" + + boot_disk { + initialize_params { + image = "deeplearning-platform-release/pytorch-2-9-cu129-ubuntu-2204-nvidia-580" + size = 100 + } + } + + attached_disk { + source = google_compute_disk.data.id + device_name = "data" + } + + network_interface { + network = "default" + access_config {} + } + + scheduling { + on_host_maintenance = "TERMINATE" + } + + metadata_startup_script = file("${path.module}/startup.sh") + + service_account { + email = google_service_account.trainer.email + scopes = ["cloud-platform"] + } +} +``` + +An N1 instance would add a `guest_accelerator` block with `type` and `count`. A G2 machine type carries its L4 GPU implicitly. + +The Runpod version replaces the VM and data disk with a Pod and a network volume. + +```hcl after.tf +resource "runpod_network_volume" "data" { + name = "trainer-data" + size = 200 + data_center = "US-KS-2" +} + +resource "runpod_pod" "trainer" { + name = "trainer" + image = "runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04" + gpu_type_id = "NVIDIA L4" + gpu_count = 1 + cloud = "SECURE" + disk = 50 + + min_cuda_version = "12.8" + data_center_ids = [runpod_network_volume.data.data_center] + + network_volumes = [{ + volume_id = runpod_network_volume.data.id + path = "/workspace" + }] + + ports = ["8888/http", "22/tcp"] + start_ssh = true + start_jupyter = true + + env = { + GCS_BUCKET = "BUCKET" + GCP_SA_KEY = "{{ RUNPOD_SECRET_gcp_sa_key }}" + } + + args = "bash -c 'bash /workspace/startup.sh && /start.sh'" +} + +output "trainer_status" { + value = runpod_pod.trainer.status +} +``` + +What changed: `machine_type` became `gpu_type_id` and `gpu_count`, the boot image became a container `image`, `google_compute_disk` became `runpod_network_volume`, the attached service account became a key read from a Runpod secret, and `metadata_startup_script` became `args`. There's no network interface or scheduling block. `start_ssh` needs an SSH key registered on your account. Changing `gpu_type_id`, `gpu_count`, `cloud`, `min_cuda_version`, or `data_center_ids` replaces the Pod, so keep anything you can't recreate on the network volume. To stop the Pod without removing it from the configuration, set `desired_state = "EXITED"`. + +### Cloud Run GPU service to a Serverless endpoint + +This is a Cloud Run service with one L4 GPU. + +```hcl before.tf +resource "google_cloud_run_v2_service" "embedder" { + name = "embedder" + location = "us-central1" + deletion_protection = false + ingress = "INGRESS_TRAFFIC_ALL" + + scaling { + max_instance_count = 3 + } + + template { + timeout = "300s" + max_instance_request_concurrency = 8 + + containers { + image = "us-central1-docker.pkg.dev/${var.project_id}/REPO/embedder:v1" + + ports { + container_port = 8080 + } + + env { + name = "MODEL_ID" + value = "sentence-transformers/all-MiniLM-L6-v2" + } + + resources { + limits = { + "cpu" = "8" + "memory" = "32Gi" + "nvidia.com/gpu" = "1" + } + } + } + + node_selector { + accelerator = "nvidia-l4" + } + + gpu_zonal_redundancy_disabled = true + } +} +``` + +The Runpod version is a Serverless template for the container and a load-balancing endpoint for routing and scaling. + +```hcl after.tf +resource "runpod_template" "embedder" { + name = "embedder" + image = "us-central1-docker.pkg.dev/${var.project_id}/REPO/embedder:v1" + serverless = true + disk = 20 + registry = runpod_registry.artifact_registry.id + ports = ["8080/http"] + + env = { + PORT = "8080" + PORT_HEALTH = "8080" + MODEL_ID = "sentence-transformers/all-MiniLM-L6-v2" + } + + start_ssh = false + start_jupyter = false +} + +resource "runpod_serverless_endpoint" "embedder" { + name = "embedder" + template_id = runpod_template.embedder.id + endpoint_type = "LOAD_BALANCER" + + gpu_pools = ["AMPERE_24"] + gpu_count = 1 + min_cuda_version = "12.8" + + scaling_type = "REQUEST_COUNT" + scaling_value = 8 + workers_min = 0 + workers_max = 3 + idle_timeout = 10 + flashboot = "FLASHBOOT" + timeout = 300000 +} + +output "embedder_url" { + value = runpod_serverless_endpoint.embedder.request_urls.base +} +``` + +What changed: `node_selector.accelerator` became `gpu_pools`, `max_instance_request_concurrency` became `scaling_value` with `scaling_type = "REQUEST_COUNT"`, `scaling.max_instance_count` became `workers_max`, `ports.container_port` became `ports` plus the `PORT` and `PORT_HEALTH` variables, and `timeout` is in milliseconds. `endpoint_type` can't change after creation. For a queue endpoint, set `endpoint_type = "QUEUE"` and `scaling_type = "QUEUE_DELAY"`, and drop `ports`. + +### Cut over in Terraform + +1. Add the Runpod resources and run `terraform apply`. The Google resources are untouched. +2. Verify the Runpod endpoint or Pod, then update the outputs or variables your callers read (for example, switch a service URL output to `runpod_serverless_endpoint.embedder.request_urls.base`). +3. Remove the Google resource blocks you replaced and apply again. If a resource has `deletion_protection` enabled, set it to `false` and apply before removing the block. +4. Keep the Google resources Runpod depends on: the service account, its key, the Artifact Registry repository, and any buckets. + +To bring existing Runpod resources under Terraform, use `terraform import` with the Pod, endpoint, template, or volume ID. See the [provider reference](https://registry.terraform.io/providers/runpod/runpod/latest/docs) for every argument. + +## Reference + +### CLI command map + +| gcloud | Runpod | +| --- | --- | +| `gcloud auth login`, `gcloud auth activate-service-account` | `rp login`, or set `RUNPOD_API_KEY` | +| `gcloud compute instances create` | `runpodctl pod create` | +| `gcloud compute instances list` | `runpodctl pod list` (add `--all` to include stopped Pods) | +| `gcloud compute instances describe` | `runpodctl pod get POD_ID` | +| `gcloud compute instances stop` and `start` | `runpodctl pod stop POD_ID` and `runpodctl pod start POD_ID` | +| `gcloud compute instances delete` | `runpodctl pod delete POD_ID` | +| `gcloud compute ssh` | `runpodctl ssh info POD_ID`, then `ssh` | +| `gcloud compute accelerator-types list`, `gcloud compute regions list` | `runpodctl gpu list`, `runpodctl datacenter list` | +| `gcloud ai models upload` and `gcloud ai endpoints deploy-model` | `runpodctl template create --serverless` and `runpodctl serverless create` | +| `gcloud ai endpoints list` and `describe` | `runpodctl serverless list` and `runpodctl serverless get ENDPOINT_ID` | +| `gcloud ai endpoints predict` | `curl` to the endpoint URL (see Step 6) | +| `gcloud ai endpoints undeploy-model` and `delete` | `runpodctl serverless delete ENDPOINT_ID` | +| `gcloud run deploy` | `POST /v2/serverless` with `"type": "LOAD_BALANCER"` | +| `gcloud run services update` | `runpodctl serverless update ENDPOINT_ID` | +| `gcloud run services list` and `delete` | `runpodctl serverless list` and `runpodctl serverless delete ENDPOINT_ID` | +| `gcloud storage rsync` | `aws s3 sync` against the network volume [S3-compatible API](/storage/s3-api), or `gcloud storage rsync` from inside a Pod | +| `gcloud artifacts docker images list` | Your registry's own tooling. Credentials: `runpodctl registry list` | +| `gcloud logging read` | `runpodctl pod logs POD_ID`, `runpodctl serverless logs ENDPOINT_ID` | +| Cloud Billing reports | `runpodctl billing pods`, `runpodctl billing serverless` | + +### API map + +| Google Cloud API call | Runpod REST API v2 or runtime route | +| --- | --- | +| Compute Engine `instances.insert` | `POST https://api.runpod.io/v2/pods` | +| Compute Engine `instances.list`, `instances.get` | `GET /v2/pods`, `GET /v2/pods/{id}` | +| Compute Engine `instances.stop`, `instances.start` | `POST /v2/pods/{id}/action` with `{"action": "stop"}` or `{"action": "start"}` | +| Compute Engine `instances.delete` | `DELETE /v2/pods/{id}` | +| Vertex AI model upload plus `endpoints.deployModel`, Cloud Run service create | `POST /v2/serverless` | +| Vertex AI `endpoints.mutateDeployedModel`, Cloud Run service update | `PATCH /v2/serverless/{id}` | +| Vertex AI `endpoints.undeployModel` and delete, Cloud Run service delete | `DELETE /v2/serverless/{id}` | +| Vertex AI `endpoints.predict`, `rawPredict` | `POST https://api.runpod.ai/v2/ENDPOINT_ID/runsync`, or your own path on `https://ENDPOINT_ID.api.runpod.ai` | +| Vertex AI `batchPredictionJobs.create` | `POST https://api.runpod.ai/v2/ENDPOINT_ID/run` per input, then `GET /status/JOB_ID` | +| Chat Completions at `.../endpoints/openapi` | `https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1` | +| Artifact Registry pull credentials | `POST /v2/registries` | +| Secret Manager secret create | `POST /v2/account/secrets` | +| Filestore instance create | `POST /v2/network-volumes` | +| Compute Engine accelerator and zone listings | `GET /v2/catalog/gpus`, `GET /v2/catalog/datacenters` | +| Cloud Logging entries for a VM or revision | `GET /v2/pods/{id}/logs`, `GET /v2/serverless/{id}/workers/{workerId}/logs` | + +### Configuration field map + +| Google Cloud field | Runpod REST API v2 field | Terraform argument | +| --- | --- | --- | +| Machine type and accelerator type (VM) | `gpu.id`, `gpu.count` | `gpu_type_id`, `gpu_count` | +| Accelerator type (Vertex AI `--accelerator`, Cloud Run `--gpu-type`) | `gpu.pools`, `gpu.count` | `gpu_pools`, `gpu_count` | +| Container image URI | `image` | `image` | +| Container command and args | `entrypoint`, `cmd`, or `args` | `args` | +| Environment variables | `env` | `env` | +| Container port (`--container-ports`, `--port`) | `ports`, plus `PORT` and `PORT_HEALTH` in `env` | `ports`, `env` | +| Health route (`--container-health-route`) | `HEALTH_CHECK_PATH` in `env` | `env` | +| Min and max replicas or instances | `workers.min`, `workers.max` | `workers_min`, `workers_max` | +| Cloud Run concurrency, Vertex AI autoscaling targets | `scaling.type`, `scaling.requestCount` or `scaling.queueDelay` | `scaling_type`, `scaling_value` | +| Request timeout | `timeout` (milliseconds) | `timeout` | +| Boot disk size | `disk` (GB) | `disk` | +| Data disk, Filestore share | `mounts.persistent`, `mounts.network` | `persistent_volume`, `network_volumes` | +| Region or zone | `dataCenterIds` | `data_center_ids` | +| Service account | A key in a Runpod secret, referenced from `env` | `env` | +| Artifact Registry access | `registry` (credential ID) | `registry` | + +### Environment variable map + +| Google Cloud runtime variable | Runpod equivalent | +| --- | --- | +| `AIP_HTTP_PORT` (Vertex AI), `PORT` (Cloud Run) | `PORT` and `PORT_HEALTH` on load-balancing workers (default `80`) | +| `AIP_HEALTH_ROUTE` | `HEALTH_CHECK_PATH` (default `/ping`) | +| `AIP_PREDICT_ROUTE` | Your own route on a load-balancing endpoint. Queue endpoints have no route. | +| `AIP_STORAGE_URI` | A path on a mounted volume, such as `/runpod-volume/models` | +| `AIP_MODEL_DIR`, `AIP_CHECKPOINT_DIR` | A path on a mounted volume, such as `/workspace/checkpoints` | +| `AIP_ENDPOINT_ID`, `K_SERVICE` | Set your own variable in `env` | +| `CLOUD_ML_PROJECT_ID`, metadata server project lookups | Set your own variable in `env` | +| `CLOUD_RUN_TASK_INDEX`, `CLOUD_RUN_TASK_COUNT` | Pass the shard index in the job `input` | +| VM instance name and zone from the metadata server | `RUNPOD_POD_ID`, `RUNPOD_DC_ID` on Pods | +| GPU count from machine type | `RUNPOD_GPU_COUNT` on Pods | +| Application Default Credentials | A service account key from a Runpod secret | + +### Status map + +| Google Cloud state | Runpod state | +| --- | --- | +| Compute Engine `PENDING`, `PROVISIONING` | Pod `PROVISIONING` (or a queued Pod in `QUEUED`) | +| Compute Engine `STAGING` | Pod `STARTING` | +| Compute Engine `RUNNING` | Pod `RUNNING` | +| Compute Engine `STOPPING`, `TERMINATED` (stopped) | Pod `EXITED` | +| Compute Engine `SUSPENDED` | No equivalent. Stop the Pod and keep its volume. | +| Compute Engine `REPAIRING` | Pod `ERROR` is the closest. Check logs, then recreate. | +| Deleted VM | Pod `TERMINATED` | +| Vertex AI `JOB_STATE_PENDING` | Job `IN_QUEUE` | +| Other Vertex AI `JOB_STATE_*` values | Job `IN_PROGRESS`, `COMPLETED`, `FAILED`, `CANCELLED`, `TIMED_OUT` | + +## Migrate with a coding agent + +A coding agent can run this migration with the Runpod skills and MCP server. + + + + + + ```bash + # The plugin also connects the hosted MCP server + claude plugin marketplace add runpod/runpod-plugins-official + claude plugin install runpod@runpod + ``` + + + ```bash + # Codex, Cursor, and others: skills, then the hosted MCP server + npx skills add runpod/runpod-plugins-official --skill '*' --agent AGENT_ID --global + npx @runpod/mcp-server@latest add --client CLIENT_ID + ``` + + + + The MCP server signs in with OAuth ("Sign in with Runpod"), so you don't need an API key to start. See [Agent skills](/get-started/agent-skills) and [MCP servers](/get-started/mcp-servers). + + + Run the agent in the repository that holds your application code and Terraform, with `gcloud` authenticated. + + ```text + Use the runpod-cloud-migrate skill to migrate our GPU workloads from Google Cloud to Runpod. + We're keeping Cloud Storage, IAM, and the rest of the app on Google Cloud. + + 1. Inventory first. Scan this repo for Vertex AI, Cloud Run, Compute Engine, GKE, and + Artifact Registry usage (SDK imports, gcloud calls, AIP_* and K_SERVICE variables, gs:// + paths, google_* Terraform resources). Then list the live GPU resources with read-only + gcloud commands. Don't change anything on Google Cloud. + 2. Show me a plan: each workload, its Runpod target (Pod, queue endpoint, load-balancing + endpoint, Public Endpoint, Instant Cluster, or @app.task), the GPU mapping, the storage + plan, what stays on Google Cloud, and how Runpod will authenticate back to it. Include a + cost estimate from the live Runpod catalog and flag anything with no direct equivalent + (VPC, IAM service accounts, CMEK, Spot). + 3. Wait for my approval before creating anything billable on Runpod. + 4. Migrate one workload end to end: image, registry credential, secrets, deploy, and the + client code change. If we use Terraform, add runpod provider resources next to the + google ones instead of replacing them. + 5. Verify with a real request from outside Runpod and show me the output and latency next + to the Google Cloud result. + 6. Never delete, stop, or undeploy Google Cloud resources without my explicit approval. + When I approve, list the exact gcloud commands first. + ``` + + + +## Troubleshooting + +**The image pull fails with an authentication error.** The registry credential is wrong or the service account can't read the repository. With `--username "_json_key"`, the password must be the raw JSON key file. With `_json_key_base64`, it must be the whole file base64 encoded on one line. An `oauth2accesstoken` credential stops working after 60 minutes. Confirm the service account has `roles/artifactregistry.reader` on the repository and that the image path includes a tag. + +**A load-balancing worker shows as running but never becomes ready.** The health check isn't reaching your server. Vertex AI and Cloud Run images usually listen on `8080`. Set `PORT` and `PORT_HEALTH` to that port, expose it in `ports`, and make sure `GET /ping` (or your `HEALTH_CHECK_PATH`) returns `200`. Misconfigured workers return `502` and are stopped after 8 minutes. + +**Requests return `{"error":"not allowed for QB API"}`.** You're calling a path on `https://ENDPOINT_ID.api.runpod.ai` for an endpoint that was created as queue-based. The endpoint type is fixed at creation. Recreate it with `"type": "LOAD_BALANCER"`. + +**Requests return `no workers available`.** The endpoint scaled to zero and no worker was ready in time. Retry with a delay, raise the idle timeout, or set one active worker for latency-sensitive paths. + +**Google client libraries fail with a default credentials error.** There's no metadata server on Runpod, so Application Default Credentials finds nothing. Load a service account key from a Runpod secret, as shown in Step 5, or write the key to a file and set `GOOGLE_APPLICATION_CREDENTIALS` to its path at startup. + +**The container exits with a CUDA or driver error.** The host's driver is older than your image's CUDA version. Set a CUDA floor (`--min-cuda-version`, `gpu.minCudaVersion`, or `min_cuda_version`) that matches the image. Cloud Run GPU instances use a recent driver, so images built against it may assume a newer CUDA than you had pinned elsewhere. + +**The handler receives `instances` as missing.** Queue endpoints deliver the request body's `input` object to the handler. Wrap the Vertex AI body: `{"input": {"instances": [...], "parameters": {...}}}`. + +**Long requests are cut off on a load-balancing endpoint.** Processing is limited to 5.5 minutes per request. Move the workload to a queue endpoint and call `/run`. + +**The Pod creates in the wrong data center or fails with a capacity error.** A network volume pins the Pod to its data center, and the GPU you asked for may not be in stock there. Check `runpodctl datacenter list`, pick a GPU type the data center has, or queue the Pod create through REST API v2. + +**Paths under `/gcs/` or a Cloud Storage FUSE mount don't exist.** Vertex AI training mounts buckets under `/gcs/`. On Runpod, copy the data to a volume and update the paths, or read from Cloud Storage with the client library. + +## Next steps + + + + The full worker contract, limits, and health check behavior. + + + Install the provider and manage Runpod as code. + + + Persistent storage for Pods and Serverless workers. + + + Define queue endpoints, tasks, and APIs in Python. + + diff --git a/migrate/hugging-face.mdx b/migrate/hugging-face.mdx new file mode 100644 index 000000000..985a55325 --- /dev/null +++ b/migrate/hugging-face.mdx @@ -0,0 +1,1112 @@ +--- +title: "Migrate from Hugging Face Inference Endpoints to Runpod" +sidebarTitle: "Hugging Face" +description: "Move Hugging Face Inference Endpoints, Inference Providers calls, GPU Spaces, and Jobs to Runpod while your models stay on the Hugging Face Hub." +--- + +This guide moves where your models run, not where they live. Your models, datasets, and access tokens stay on the Hugging Face Hub, and Runpod pulls weights from the Hub by repository ID, the same way [Inference Endpoints](https://huggingface.co/docs/inference-endpoints/index) does today. + +It's written for teams that run production inference on Hugging Face compute. Inference Endpoints map to Runpod [Serverless](/serverless/overview) endpoints, [Inference Providers](https://huggingface.co/docs/inference-providers/index) calls map to [Public Endpoints](/public-endpoints/overview) where the model is listed, GPU [Spaces](https://huggingface.co/docs/hub/spaces-gpus) map to [Pods](/pods/overview), and [Jobs](https://huggingface.co/docs/hub/jobs) map to `@app.task` in the Runpod SDK or to a Pod. At the end, each workload runs on Runpod, your clients call Runpod URLs with a Runpod API key, and the Hugging Face compute is paused or deleted. + + +You can hand this migration to a coding agent. See [Migrate with a coding agent](#migrate-with-a-coding-agent). + + +## How Hugging Face maps to Runpod + +| Hugging Face concept | Runpod equivalent | Notes | +| --- | --- | --- | +| Hugging Face Hub (models, datasets, tokens) | Stays where it is | Runpod downloads from the Hub. [Cached models](/serverless/endpoints/model-caching), the [Hub](/hub/overview), and `hf download` onto a network volume all use the Hub as the model source. | +| Inference Endpoint with the vLLM, TGI, or SGLang engine | Serverless [vLLM worker](/serverless/vllm/overview) | Same Hugging Face repo ID. OpenAI-compatible route at `/openai/v1`. | +| Inference Endpoint with the Inference Toolkit and a custom `handler.py` | Serverless queue-based endpoint with a [handler function](/serverless/workers/handler-functions), or `@app.queue` in the [Runpod SDK](/sdk/overview) | `EndpointHandler.__call__(data)` becomes `handler(job)`. | +| Inference Endpoint with the TEI engine | Serverless [load balancing endpoint](/serverless/load-balancing/overview) running the TEI container | You keep TEI's own routes, including `/v1/embeddings`. | +| Inference Endpoint with the llama.cpp engine or a custom container | Serverless load balancing endpoint | Your HTTP server and routes run unchanged. Health checks default to `/ping`. | +| Inference Endpoints Model Catalog | [Runpod Hub](/hub/overview) | Preconfigured repos you deploy as endpoints. | +| Replicas (min and max) | Workers (active and max) | Active workers are always on. Other workers start on demand. | +| Scale-to-zero timeout (minutes) | Idle timeout (seconds) | Runpod endpoints scale to zero by default. | +| Inference Providers (`https://router.huggingface.co/v1`) | [Public Endpoints](/public-endpoints/overview), or your own vLLM endpoint | Public Endpoints cover the models in the [model reference](/public-endpoints/reference). | +| Spaces on GPU hardware | [Pod](/pods/overview) with an exposed HTTP port | The Pod proxy URL is public and unauthenticated. | +| ZeroGPU Spaces | Queue-based endpoint behind your own UI | No direct equivalent. See [What has no direct equivalent](#what-has-no-direct-equivalent). | +| Jobs (`hf jobs run`) | `@app.task` in the Runpod SDK, or a Pod | One Pod per call that terminates when the function returns. | +| Engine container image, custom image | Container image in a Serverless template | Images must be `linux/amd64`. | +| Model mounted at `/repository` | Cached model at `/runpod-volume/huggingface-cache/hub/` | Hugging Face cache layout, not a flat directory. | +| Files you manage yourself (adapters, indexes, outputs) | [Network volume](/storage/network-volumes), global volume, container disk | Network volumes mount at `/runpod-volume` on Serverless and `/workspace` on Pods. Container disk is ephemeral. | +| Default Env and Secret Env | Environment variables and [Runpod secrets](/pods/templates/secrets) | Reference a secret as `{{ RUNPOD_SECRET_name }}`. | +| Endpoint types: public, authenticated, private | Runpod API key on every Serverless request | See [What works differently](#what-works-differently). | +| Vendor and region (AWS, Azure, Google Cloud) | Runpod data centers | Leave data centers unrestricted unless you need a specific one. | +| Instance type and size (for example `nvidia-l4` `x1`) | GPU pool or GPU type, plus GPU count | See the [GPU mapping](#gpu-mapping). | +| Billed per minute for each replica while initializing and running | Billed per second while a worker runs | No charge while a cached model downloads. See [Serverless pricing](/serverless/pricing). | +| `huggingface_hub` endpoint functions, `hf endpoints` CLI | `runpodctl serverless`, the Runpod SDK, the [MCP server](/get-started/mcp-servers) | The SDK and Terraform provider are built on REST API v2. | +| Inference Endpoints API (`https://api.endpoints.huggingface.cloud/v2`) | [REST API v2](/api-reference-v2/overview) (`https://api.runpod.io/v2/serverless`) | Bearer token auth on both. | +| Infrastructure as code through the API | [Terraform provider](/integrations/terraform/overview) (`runpod_serverless_endpoint`) | Optional. | +| Hugging Face access token (`HF_TOKEN`) | Runpod API key (`RUNPOD_API_KEY`) | You still need `HF_TOKEN` on Runpod for gated and private repos. | +| Organizations, resource groups | [Organizations](/organizations/orgs-overview) or Teams | Runpod organizations are in beta. | +| Logs tab, Analytics tab, OpenMetrics API | Endpoint logs and metrics in the console, REST v2 metrics and worker log routes | See [Logs](/serverless/development/logs). | + +## What's different + +### What works differently + +**Where the model lands inside the container.** Inference Endpoints mounts the selected repository at `/repository`. Runpod's cached models land in the standard Hugging Face cache layout at `/runpod-volume/huggingface-cache/hub/models--ORG--NAME/snapshots/HASH/`. Code that loads from `/repository` needs the snapshot path instead. Step 4 has a helper that resolves it. + +**What happens to a request when nothing is running.** An Inference Endpoint that has scaled to zero answers with status `503` while the new replica initializes, unless the client sends the `X-Scale-Up-Timeout` header to make the proxy hold the request for a set number of seconds. A Runpod queue-based endpoint puts the request in the endpoint's queue and a worker picks it up when it's ready, so you can remove 503 retry loops. Load balancing endpoints don't queue: a request that arrives before a worker is ready can return `no workers available`, and it times out after 2 minutes without a worker. + +**The first synchronous request after a cold start.** `/runsync` waits up to 90 seconds by default, and synchronous routes sit behind an edge timeout of about 100 seconds. A cold vLLM worker loading a large model can take longer than that. Use `/run` and poll `/status` for the first request, or keep one active worker for latency-sensitive endpoints. + +**Idle time is measured in seconds.** Scale-to-zero on Inference Endpoints is set in minutes (1 hour by default). Runpod's idle timeout is set in seconds, from 1 to 3600. The console default is 5 seconds and the REST v2 default is 10 seconds. A worker is billed while it waits out the idle timeout. + +**Authentication.** Inference Endpoints has three endpoint types: public (no authentication), authenticated (a Hugging Face token), and private. Requests to a Runpod Serverless endpoint carry `Authorization: Bearer RUNPOD_API_KEY`. If you run a public endpoint today, put your own API or gateway in front of the Runpod endpoint and keep the key server-side. + +**Request and response envelope.** The Inference Toolkit accepts your JSON body as is, with `inputs` at the top level. A queue-based endpoint expects `{"input": {...}}` and returns your handler's result under `output`, next to `id`, `status`, `delayTime`, and `executionTime`. + +**The `model` field on OpenAI-compatible calls.** On a vLLM worker, `model` must be the Hugging Face repo ID you deployed, or the name you set in `OPENAI_SERVED_MODEL_NAME_OVERRIDE`. If your clients send the Inference Endpoint's name as `model`, set that override to the same string and they won't need a code change beyond the URL and key. + +**TGI's native routes.** TGI serves `/generate` and `/generate_stream` next to its OpenAI-compatible routes. The vLLM worker serves `/chat/completions`, `/completions`, and `/models`. Move `/generate` callers to `/completions`. Hugging Face put TGI in maintenance mode in late 2025 and recommends vLLM or SGLang, so this is the same move you'd make on Inference Endpoints. + +**Autoscaling signals.** Inference Endpoints scales on hardware utilization, pending requests, or (in beta, for vLLM and SGLang) inference load. Runpod scales on queue delay (seconds a request waits) or request count (queued plus in-progress requests divided by a target). Request count is the closer match to pending requests. + +**Pause and resume.** A paused Inference Endpoint stays defined and isn't billed. On Runpod, an endpoint at zero workers isn't billed for compute, so there's no separate paused state. To block new workers from starting, set max workers to 0. + +**Unused endpoints scale down on their own.** After 3 days without a request, Runpod reduces an endpoint's max workers to 2 and emails you. After 7 days it sets max workers to 0, and it stays there until you raise it. An Inference Endpoint with min replicas above 0 keeps running until you change it. + +**Worker limits.** Your account balance sets the maximum number of workers across all endpoints (5 by default). See [Max worker limits](/serverless/workers/overview#max-worker-limits) before you size a large endpoint. + +### What has no direct equivalent + +| On Hugging Face | On Runpod | Workaround | +| --- | --- | --- | +| Public (unauthenticated) endpoint type | API key on every Serverless request | Proxy through your own backend. | +| AWS PrivateLink for endpoints | Serverless endpoints are reached over public HTTPS with an API key | Contact Runpod about private networking requirements before you migrate. | +| Choice of AWS, Azure, or Google Cloud as the vendor | Runpod data centers | Pin `dataCenterIds` if you need a location. Check [security and compliance](/references/security-and-compliance) for your requirements. | +| AWS Inferentia2 (`inf2`) instances | GPU and CPU workers | Redeploy the standard (non-Neuron) model on a GPU. | +| Daily autoscaling schedule | Not built in | Run `runpodctl serverless update ENDPOINT_ID --workers-min N` from your scheduler. | +| `dataParallelSize` (several model copies on one multi-GPU instance) | One model copy per worker | Use single-GPU workers and raise max workers. | +| Inference Providers routing across many providers with `:fastest` and `:cheapest` | Public Endpoints for listed models | Deploy other models on your own vLLM endpoint. | +| ZeroGPU (a GPU attached per function call inside a Gradio Space) | Not available | Host the UI anywhere and call a queue-based endpoint for the GPU step. | +| OpenMetrics export | REST v2 metrics routes | Poll `/v2/serverless/ENDPOINT_ID/metrics/requests`, `/metrics/cold-starts`, and `/metrics/workers`. | + +In the other direction, moving gives you a few things to build on: a request queue with async jobs, status polling, and webhooks on queue-based endpoints, persistent [network volumes](/storage/network-volumes) shared across workers, and per-second billing. + +## Requirements + +- A Hugging Face account with access to the endpoints, Spaces, and Jobs you're moving, and the [`hf` CLI](https://huggingface.co/docs/huggingface_hub/guides/cli) logged in (`hf auth login`). +- A Hugging Face access token with read access to every gated or private repo you serve. +- A [Runpod account](/accounts-billing/manage-accounts) with credits and a [Runpod API key](/get-started/credentials). +- Docker, if you're moving a custom handler or a custom container. +- Python 3.10 to 3.14, if you use the Runpod SDK. + +## Step 1: Take inventory of what you run on Hugging Face + +List every Inference Endpoint in every namespace you belong to, then print the full configuration of each one: + +```bash +hf endpoints ls --namespace '*' +hf endpoints describe ENDPOINT_NAME --namespace NAMESPACE +``` + +To export everything at once, dump the raw API response for each endpoint. It includes the repository and revision, engine image, environment variable names, instance type and size, vendor and region, replica limits, and the scale-to-zero timeout. + +```python export_endpoints.py +import json + +from huggingface_hub import list_inference_endpoints + +inventory = [] +for endpoint in list_inference_endpoints(namespace="*"): + inventory.append( + { + "name": endpoint.name, + "namespace": endpoint.namespace, + "repository": endpoint.repository, + "revision": endpoint.revision, + "task": endpoint.task, + "status": endpoint.status, + "type": endpoint.type, + "url": endpoint.url, + "raw": endpoint.raw, + } + ) + +with open("hf_endpoints.json", "w") as f: + json.dump(inventory, f, indent=2, default=str) + +print(f"Exported {len(inventory)} endpoints to hf_endpoints.json") +``` + +Secret values aren't returned, so collect them from wherever you keep them. For each endpoint, record which engine it runs (the key under `model.image` in `raw`, such as `vllm`, `tgi`, `tei`, `llamacpp`, `huggingface`, or `custom`) and whether the model repository contains a `handler.py`. Those two facts decide where the endpoint lands in Step 3. + +Then list the rest of your Hugging Face compute: + +```bash +# Spaces you own, with the hardware each one runs on +python -c " +from huggingface_hub import HfApi +api = HfApi() +for space in api.list_spaces(author='NAMESPACE'): + runtime = api.get_space_runtime(space.id) + print(space.id, runtime.stage, runtime.hardware) +" +hf spaces info NAMESPACE/SPACE_NAME + +# Jobs, including finished ones, and scheduled Jobs +hf jobs ps -a +hf jobs scheduled ls +``` + +Any Space whose hardware isn't `cpu-basic` is using paid compute. Note each Space's SDK (Gradio, Docker, or static), its `app_port`, and its variables and secrets. + +Finally, find the code that calls Hugging Face compute or manages it: + +```bash +grep -rnE "endpoints\.huggingface\.cloud|router\.huggingface\.co|api\.endpoints\.huggingface\.cloud|InferenceClient|AsyncInferenceClient|create_inference_endpoint|get_inference_endpoint|list_inference_endpoints|scale_to_zero|X-Scale-Up-Timeout|EndpointHandler|hf endpoints|hf jobs|run_job|@spaces\.GPU|\.hf\.space" \ + --include="*.py" --include="*.ts" --include="*.js" --include="*.sh" --include="*.yml" --include="*.yaml" --include="*.env*" . +``` + +Code that only reads from the Hub (`from_pretrained`, `hf_hub_download`, `snapshot_download`, `load_dataset`) doesn't need to change. + +## Step 2: Set up Runpod + +Install `runpodctl` and the Runpod SDK, which includes the `rp` CLI, then store your API key: + +```bash +# runpodctl (macOS: brew install runpod/runpodctl/runpodctl) +curl -fsSL https://cli.runpod.net | sudo bash +runpodctl doctor + +# Runpod SDK 2.x and the rp CLI +pip install runpod +rp login + +export RUNPOD_API_KEY="YOUR_API_KEY" +``` + +`runpodctl doctor` walks through first-time setup, and `rp login` saves credentials to `~/.runpod/config.toml`. To let a coding agent manage resources, add the [MCP server](/get-started/mcp-servers). If you manage infrastructure as code, the [Terraform provider](/integrations/terraform/overview) covers endpoints, templates, and volumes. + +Next, store your Hugging Face token as a Runpod secret so it never appears in an endpoint's configuration: + +```bash +rp secret add hf_token +``` + +The command prompts for the value. You can also create the secret on the [Secrets page](https://www.console.runpod.io/user/secrets) in the console. Secret names can't start with `runpod`. Reference the secret from any environment variable as `{{ RUNPOD_SECRET_hf_token }}`. + +## Step 3: Choose where each workload lands + +| What you run on Hugging Face | Runpod target | Why | +| --- | --- | --- | +| Endpoint serving an LLM with vLLM, TGI, or SGLang | Serverless vLLM worker (queue-based) | Same repo ID, OpenAI-compatible route, requests queue during cold starts. | +| Endpoint with a custom `handler.py` (Inference Toolkit) | Queue-based endpoint with a handler, or `@app.queue` | Closest contract: one function per request. | +| Endpoint serving embeddings or rerankers with TEI | Load balancing endpoint running the TEI image | Keeps TEI's routes and batching. | +| Endpoint with a GGUF model on llama.cpp | Load balancing endpoint running a llama.cpp server image | The server already exposes `/health`. | +| Endpoint with a custom container | Load balancing endpoint | Your server's paths and payloads stay the same. | +| Inference Providers call for a model in the Public Endpoints reference | Public Endpoint | Nothing to deploy. Billed per use. | +| Inference Providers call for any other model | Serverless vLLM worker | You run the model yourself. | +| Gradio, Streamlit, or Docker Space on GPU hardware | Pod with an exposed HTTP port | Long-running process with a URL. | +| ZeroGPU Space | UI host of your choice plus a queue-based endpoint | The endpoint gives you a GPU per request. | +| Job (`hf jobs run`, `hf jobs uv run`) | `@app.task`, or a Pod | Runs to completion, then terminates. | +| Multi-node training | [Instant Cluster](/instant-clusters) | Slurm, PyTorch, or Ray across nodes. | + +Queue-based or load balancing is the main decision for an Inference Endpoint. Choose queue-based when requests can wait for a worker, run longer than a few minutes, or need retries. Choose load balancing when you need your own routes, non-JSON payloads, or WebSockets, and your requests finish within the 5.5 minute processing limit and the 30 MB payload limit. + +### GPU mapping + +Match on VRAM and GPU generation, then confirm the model fits. Instance names come from the [Inference Endpoints pricing page](https://huggingface.co/docs/inference-endpoints/support/pricing), and pools and GPU IDs come from the [GPU types reference](/references/gpu-types). + +| Inference Endpoints instance type | VRAM per GPU | Runpod GPU pool | Runpod GPUs in the pool | +| --- | --- | --- | --- | +| `nvidia-t4` | 16 GB | `AMPERE_16` | A4000, A4500, RTX 4000, RTX 2000 | +| `nvidia-l4` | 24 GB | `AMPERE_24` | L4, A5000, 3090 | +| `nvidia-a10g` | 24 GB | `AMPERE_24` or `ADA_24` | L4, A5000, 3090, or 4090 | +| `nvidia-l40s` | 48 GB | `ADA_48_PRO` | L40, L40S, 6000 Ada | +| `nvidia-a100` | 80 GB | `AMPERE_80` | A100 | +| `nvidia-h100` | 80 GB | `ADA_80_PRO` | H100 | +| `nvidia-h200` | 141 GB | `HOPPER_141` | H200 | +| `nvidia-b200` | Check `hf endpoints hardware` | `BLACKWELL_180` | B200 (180 GB) | +| `intel-spr`, `intel-xeon` (CPU) | None | CPU endpoint (`--compute-type CPU`) | See [CPU types](/references/cpu-types) | +| `inf2` (AWS Inferentia2) | Not a GPU | No equivalent | Deploy the standard model on a GPU pool | + +The instance size (`x1`, `x2`, `x4`, `x8`) is the number of GPUs per replica. Set the same number with `--gpu-count`, or `gpu.count` in REST v2. You can list up to three GPU choices in priority order on one endpoint, which helps when your first choice is busy. Stock varies by data center, so check live availability with `runpodctl gpu list` or `GET /v2/catalog/gpus?include=AVAILABILITY&product=SERVERLESS` before you commit to a GPU. + +## Step 4: Move your images and code + +### LLM endpoints (vLLM, TGI, SGLang) + +There's no image to build. The vLLM worker takes the same repo ID, and engine settings become environment variables. Any `vllm serve` flag works as an upper snake case variable (`--max-model-len 8192` becomes `MAX_MODEL_LEN=8192`), and `VLLM_EXTRA_ARGS` passes flags through as written. + +| Inference Endpoints setting | vLLM worker variable | +| --- | --- | +| Model repository | `MODEL_NAME` (and the cached model reference) | +| Commit revision | `MODEL_REVISION` | +| TGI: Max Number of Tokens (`MAX_TOTAL_TOKENS`) | `MAX_MODEL_LEN` | +| TGI: Quantization | `QUANTIZATION` (must match the checkpoint format) | +| vLLM: Max Number of Sequences | `MAX_NUM_SEQS` | +| vLLM: Max Number of Batched Tokens | `MAX_NUM_BATCHED_TOKENS` | +| vLLM: KV Cache DType | `KV_CACHE_DTYPE` | +| vLLM: Tensor Parallel Size | `TENSOR_PARALLEL_SIZE` (set automatically to the worker's GPU count) | +| Container arguments such as `--enable-auto-tool-choice --tool-call-parser hermes` | `ENABLE_AUTO_TOOL_CHOICE=true`, `TOOL_CALL_PARSER=hermes`, or `VLLM_EXTRA_ARGS` | +| Secret Env `HF_TOKEN` | `HF_TOKEN` set to `{{ RUNPOD_SECRET_hf_token }}` | +| Name your clients send as `model` | `OPENAI_SERVED_MODEL_NAME_OVERRIDE` | + +TGI's batching limits (Max Input Tokens, Max Batch Prefill Tokens, Max Batch Total Tokens) have no one-to-one vLLM setting. Start with `MAX_MODEL_LEN` and `MAX_NUM_SEQS`, then tune from there. See [vLLM environment variables](/serverless/vllm/environment-variables) and [Configure vLLM](/serverless/vllm/configuration) for model-specific settings. + +### Custom handlers (`handler.py`) + +On Inference Endpoints, the Inference Toolkit imports `EndpointHandler` from a `handler.py` in your model repository, calls `__init__` once with the path to the weights, and calls `__call__` with the request body for every request. Here's a typical handler: + +```python handler.py (Hugging Face) +from typing import Any, Dict, List + +from transformers import pipeline + + +class EndpointHandler: + def __init__(self, path=""): + # Called once when the endpoint starts. `path` points to the model weights. + self.pipeline = pipeline("text-classification", model=path, device=0) + + def __call__(self, data: Dict[str, Any]) -> List[Dict[str, Any]]: + # Called for every request. `data` is the request body. + inputs = data.pop("inputs", data) + parameters = data.pop("parameters", None) or {} + return self.pipeline(inputs, **parameters) +``` + +On Runpod, the worker is a container that you build. Load the model at module level so it happens once per worker, and read the request from `job["input"]`: + +```python handler.py (Runpod) +import os + +import runpod +from transformers import pipeline + +MODEL_ID = os.environ.get( + "MODEL_ID", "distilbert/distilbert-base-uncased-finetuned-sst-2-english" +) +HF_CACHE_ROOT = "/runpod-volume/huggingface-cache/hub" + + +def resolve_snapshot_path(model_id: str) -> str: + """Return the local snapshot directory for a cached Hugging Face model.""" + org, name = model_id.split("/", 1) + model_root = os.path.join(HF_CACHE_ROOT, f"models--{org}--{name}") + refs_main = os.path.join(model_root, "refs", "main") + snapshots_dir = os.path.join(model_root, "snapshots") + + if os.path.isfile(refs_main): + with open(refs_main, "r") as f: + candidate = os.path.join(snapshots_dir, f.read().strip()) + if os.path.isdir(candidate): + return candidate + + if os.path.isdir(snapshots_dir): + versions = sorted( + d + for d in os.listdir(snapshots_dir) + if os.path.isdir(os.path.join(snapshots_dir, d)) + ) + if versions: + return os.path.join(snapshots_dir, versions[0]) + + raise RuntimeError(f"Cached model not found: {model_id}") + + +# Runs once when the worker starts. This replaces EndpointHandler.__init__. +classifier = pipeline( + "text-classification", model=resolve_snapshot_path(MODEL_ID), device=0 +) + + +def handler(job): + # Runs for every request. This replaces EndpointHandler.__call__. + data = job["input"] + inputs = data.get("inputs") + if inputs is None: + return {"error": "Missing 'inputs'."} + parameters = data.get("parameters") or {} + return classifier(inputs, **parameters) + + +runpod.serverless.start({"handler": handler}) +``` + +Three things changed. `path` is now the snapshot directory of the cached model, which Runpod places at `/runpod-volume/huggingface-cache/hub/` when you attach a model to the endpoint (see [Use Hugging Face models](/serverless/development/huggingface-models#use-cached-models)). The handler keeps the `inputs` and `parameters` keys, so clients only wrap their existing body in `input`. Errors are returned as `{"error": "..."}`, which marks the job as `FAILED`. + + +To change as little code as possible, copy your existing `handler.py` into the image unchanged as `hf_handler.py` and wrap it: create `EndpointHandler(path=resolve_snapshot_path(MODEL_ID))` at module level, then return `endpoint_handler(job["input"])` from `handler(job)`. + + +Package the handler with a Dockerfile. Copy the dependencies from the `requirements.txt` in your model repository and add `runpod`: + +```dockerfile Dockerfile +FROM pytorch/pytorch:2.9.1-cuda12.8-cudnn9-runtime + +WORKDIR / + +COPY requirements.txt . +RUN pip install --no-cache-dir -r requirements.txt + +COPY handler.py . + +CMD ["python", "-u", "/handler.py"] +``` + +```txt requirements.txt +runpod +transformers +``` + +Build for `linux/amd64` and push to a registry Runpod can pull from: + +```bash +docker build --platform linux/amd64 -t DOCKER_USER/sentiment-worker:v1 . +docker push DOCKER_USER/sentiment-worker:v1 +``` + +The base image uses CUDA 12.8, so set a matching CUDA floor when you create the endpoint in Step 6. The weights aren't in the image: Runpod stages them on the host. For a private registry, save credentials with `runpodctl registry create` and pass the ID with `--registry-auth-id` on the template. You can also skip the registry and [deploy from GitHub](/serverless/workers/github-integration). + +### Custom handlers without a Dockerfile + +The Runpod SDK builds and deploys the worker from a Python file. `runpod.Model` stages the weights from the Hub before the worker starts, and `model.path` points to them: + +```python main.py +import runpod + +app = runpod.App("sentiment") + +model = runpod.Model("distilbert/distilbert-base-uncased-finetuned-sst-2-english") + + +@app.queue( + gpu="4090", + workers=(0, 3), + idle_timeout=60, + dependencies=["transformers"], + model=model, +) +def classify(inputs: str, parameters: dict | None = None): + from transformers import pipeline + + classifier = pipeline("text-classification", model=str(model.path), device=0) + return classifier(inputs, **(parameters or {})) + + +@runpod.local_entrypoint +def main(): + print(classify.remote("This migration went well.")) +``` + +Run it against temporary endpoints with `rp flash dev main.py`, then deploy with `rp flash deploy`. For a gated model, add `env={"HF_TOKEN": runpod.Secret("hf_token")}` to the decorator. This example builds the pipeline inside the function, the same way the SDK docs do. To load the model once per worker, serve it from an `@app.api` class and load it in a method marked `@runpod.init`. See [Build an app](/sdk/apps/build-app) and [Resource parameters](/sdk/configuration/parameters). + +### Custom containers, TEI, and llama.cpp + +A container that serves HTTP on Inference Endpoints runs on a load balancing endpoint with three changes: + +| Inference Endpoints | Runpod load balancing endpoint | +| --- | --- | +| Container port set in the endpoint's container configuration | `PORT` environment variable (default `80`). Set `PORT_HEALTH` to the same value. | +| Health route set in the container configuration (for example `/health`), polled until it returns `200` | `HEALTH_CHECK_PATH` environment variable (default `/ping`). `200` means healthy, `204` means initializing, anything else is unhealthy. | +| Model mounted at `/repository` | Cached model under `/runpod-volume/huggingface-cache/hub/`, or downloaded at startup with `HF_TOKEN` | + +If you don't want to rebuild the image, set `HEALTH_CHECK_PATH` to your existing health route. Changing `PORT` doesn't change `PORT_HEALTH`, so set both, and expose the port on the endpoint. If the server loads the model from `/repository`, make that path configurable and point it at the cached snapshot directory or a Hub repo ID. + +For TEI, run the image from the [TEI docs](https://huggingface.co/docs/text-embeddings-inference/quick_tour) (for example `ghcr.io/huggingface/text-embeddings-inference:cuda-1.9`). TEI reads `MODEL_ID`, `HF_TOKEN`, and `PORT` from the environment and serves `/health`, `/embed`, `/rerank`, `/predict`, and `/v1/embeddings`. Runpod's docs don't cover a dedicated embeddings worker, but the Hub has an embedding category you can search: + +```bash +runpodctl hub search embedding --type SERVERLESS +``` + +Read a listing's required environment variables with `runpodctl hub get` before you deploy it. + +### Spaces and Jobs + +A Docker Space is already a container image. Build it for `linux/amd64`, push it, and run it on a Pod with the Space's `app_port` exposed (the default is `7860`). For a Gradio or Streamlit Space, add a Dockerfile that installs `requirements.txt` and starts the app bound to `0.0.0.0`. Space variables and secrets become Pod environment variables and Runpod secrets. + +A Job is an image, a command, and a hardware flavor. With the Runpod SDK, the same work is a function: + + + + ```bash + hf jobs run --flavor a10g-small --secrets HF_TOKEN --timeout 2h \ + pytorch/pytorch:2.6.0-cuda12.4-cudnn9-devel \ + python train.py --steps 1000 + ``` + + + ```python main.py + import runpod + + app = runpod.App("training") + + checkpoints = runpod.NetworkVolume("checkpoints", size=50) + + + @app.task( + gpu="4090", + dependencies=["transformers", "datasets"], + mounts={"/checkpoints": checkpoints}, + env={"HF_TOKEN": runpod.Secret("hf_token")}, + ) + def train(steps: int = 1000): + run_dir = checkpoints.path / "run-1" + run_dir.mkdir(parents=True, exist_ok=True) + # Your training code goes here. Save checkpoints to run_dir. + return {"steps": steps, "checkpoints": str(run_dir)} + + + @runpod.local_entrypoint + def main(): + print(train.remote(steps=1000)) + ``` + + + +Each call to `train` provisions one Pod and terminates it when the function returns. Scheduled Jobs (`hf jobs scheduled run`) have no built-in counterpart: trigger the task from your own scheduler or CI. + +## Step 5: Move your data and models + +In most migrations there's nothing to copy, because the weights are on the Hub. Pick how each endpoint gets them: + +| Method | Use it when | How | +| --- | --- | --- | +| [Cached models](/serverless/endpoints/model-caching) | The model is on the Hub (public, gated, or private) | `--model-reference https://huggingface.co/ORG/NAME:main`, or the **Model** field in the console | +| [Network volume](/storage/network-volumes) | You need several models on one endpoint, files that aren't on the Hub, or predictable load times | `hf download` onto the volume from a Pod | +| Baked into the image | You need a fully self-contained image | `COPY` the files in your Dockerfile | + +Cached models are the closest match to how Inference Endpoints delivers weights. Runpod schedules workers on hosts that already hold the model where it can, and you aren't billed while a model downloads. An endpoint supports one cached model, and a repo with several quantization variants downloads all of them. The reference includes a ref (`:main`, a tag, or a commit), so pin the commit you serve today if your Inference Endpoint pins a revision. + +To pre-load a network volume, create it in the data center you'll serve from, start a small Pod with the volume attached, and download from the Hub: + +```bash +runpodctl network-volume create --name models --size 100 --data-center-id US-KS-2 + +runpodctl pod create --name model-loader \ + --image "runpod/pytorch:1.0.3-cu1281-torch291-ubuntu2404" \ + --gpu-id "NVIDIA GeForce RTX 4090" \ + --data-center-ids US-KS-2 \ + --network-volume-id NETWORK_VOLUME_ID \ + --env '{"HF_TOKEN":"{{ RUNPOD_SECRET_hf_token }}"}' + +# Connect with `rp ssh POD_ID`, then run on the Pod (the volume is mounted at /workspace): +pip install -U huggingface_hub +hf download meta-llama/Llama-3.1-8B-Instruct --local-dir /workspace/models/llama-3.1-8b-instruct +``` + +A network volume pins every endpoint and Pod that mounts it to the volume's data center, so choose one that has the GPUs you need. On Serverless the same files appear under `/runpod-volume/models/`. Delete the loader Pod when the download finishes. + +Fine-tuned weights that only exist on a replica's disk or in a Space's ephemeral storage should be pushed to a Hub repository first (`hf upload`), then served from there. Datasets on the Hub load with `load_dataset` from Runpod as they do today. + +## Step 6: Deploy on Runpod + +### vLLM worker for an LLM endpoint + +Here's an Inference Endpoint definition and its Runpod equivalent: + + + + ```python + from huggingface_hub import create_inference_endpoint + + endpoint = create_inference_endpoint( + "llama-3-1-8b-instruct", + repository="meta-llama/Llama-3.1-8B-Instruct", + framework="pytorch", + task="text-generation", + accelerator="gpu", + vendor="aws", + region="us-east-1", + instance_type="nvidia-l4", + instance_size="x1", + min_replica=0, + max_replica=3, + scale_to_zero_timeout=15, + type="authenticated", + custom_image={ + "vLLM": { + "url": "vllm/vllm-openai:v0.23.0", + "healthRoute": "/health", + "port": 8000, + "tensorParallelSize": 1, + } + }, + secrets={"HF_TOKEN": "YOUR_HF_TOKEN"}, + ) + endpoint.wait() + ``` + + + ```bash + # Find the vLLM worker's Hub listing ID + runpodctl hub search vllm --type SERVERLESS + + runpodctl serverless create \ + --hub-id HUB_ID \ + --name llama-3-1-8b-instruct \ + --gpu-id AMPERE_24 \ + --workers-min 0 \ + --workers-max 3 \ + --idle-timeout 60 \ + --execution-timeout 600 \ + --model-reference https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct:main \ + --env MODEL_NAME=meta-llama/Llama-3.1-8B-Instruct \ + --env MAX_MODEL_LEN=8192 \ + --env HF_TOKEN='{{ RUNPOD_SECRET_hf_token }}' + ``` + + + +`--hub-id` pulls the image, container disk size, and default environment from the Hub release, and `--env` overrides or adds variables. `--model-reference` attaches the cached model, so workers start with the weights already on the host where possible. The command prints the endpoint ID that goes in every request URL. In the console, the same deployment is **Serverless > New Endpoint > Hugging Face**, where you enter the repo ID and token. + +Replicas translate directly: `min_replica` is `--workers-min` (active workers, billed continuously) and `max_replica` is `--workers-max`. `scale_to_zero_timeout=15` is in minutes, and `--idle-timeout` is in seconds with a maximum of 3600. A short idle timeout costs less and causes more cold starts. + +### Queue-based endpoint for a custom handler + +Create a Serverless template for the image you pushed in Step 4, then create the endpoint from it: + + + + ```bash + runpodctl template create \ + --name sentiment-worker \ + --image DOCKER_USER/sentiment-worker:v1 \ + --serverless \ + --container-disk-in-gb 20 \ + --env '{"MODEL_ID":"distilbert/distilbert-base-uncased-finetuned-sst-2-english"}' + + runpodctl serverless create \ + --template-id TEMPLATE_ID \ + --name sentiment \ + --gpu-id AMPERE_24 \ + --min-cuda-version 12.8 \ + --workers-min 0 \ + --workers-max 3 \ + --idle-timeout 30 \ + --execution-timeout 300 \ + --model-reference https://huggingface.co/distilbert/distilbert-base-uncased-finetuned-sst-2-english:main + ``` + + + ```bash + curl -X POST "https://api.runpod.io/v2/serverless" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "sentiment", + "image": "DOCKER_USER/sentiment-worker:v1", + "type": "QUEUE", + "gpu": {"pools": ["AMPERE_24"], "count": 1, "minCudaVersion": "12.8"}, + "workers": {"min": 0, "max": 3, "idleTimeout": 30}, + "scaling": {"type": "QUEUE_DELAY", "queueDelay": 4}, + "flashboot": "FLASHBOOT", + "timeout": 300000, + "disk": 20, + "env": { + "MODEL_ID": "distilbert/distilbert-base-uncased-finetuned-sst-2-english" + } + }' + ``` + + + +With `--template-id`, environment variables come from the template, so set them on `template create`. Each Serverless template binds to one endpoint. In REST v2, `type` and `scaling` are required, `timeout` is in milliseconds, and FlashBoot is off unless you set it. The REST example creates the endpoint without a cached model: attach one with `runpodctl serverless create --model-reference` or the **Model** field in the console, because the handler above loads from the cache. + +### Load balancing endpoint for a custom container or TEI + +Create load balancing endpoints through REST v2 or the console (**Serverless > New Endpoint**, endpoint type **Load Balancer**). This example runs TEI: + +```bash +curl -X POST "https://api.runpod.io/v2/serverless" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "embeddings", + "image": "ghcr.io/huggingface/text-embeddings-inference:cuda-1.9", + "type": "LOAD_BALANCER", + "gpu": {"pools": ["AMPERE_24"], "count": 1}, + "workers": {"min": 0, "max": 3, "idleTimeout": 60}, + "scaling": {"type": "REQUEST_COUNT", "requestCount": 4}, + "flashboot": "FLASHBOOT", + "ports": ["8080/http"], + "env": { + "MODEL_ID": "sentence-transformers/all-MiniLM-L6-v2", + "PORT": "8080", + "PORT_HEALTH": "8080", + "HEALTH_CHECK_PATH": "/health", + "HF_TOKEN": "{{ RUNPOD_SECRET_hf_token }}" + } + }' +``` + +Load balancing endpoints scale on request count. The endpoint is served at `https://ENDPOINT_ID.api.runpod.ai/`, and every path your server defines is available under it. For your own container, swap in your image and port, and set `HEALTH_CHECK_PATH` to the health route you configured on Inference Endpoints. This TEI configuration downloads the model when a worker starts, so expect a longer cold start than with a cached model. + +### Public Endpoint for an Inference Providers call + +There's nothing to deploy. Check the [model reference](/public-endpoints/reference) for the model you call through `router.huggingface.co`. If it's listed, switch the client in Step 7. If it isn't, deploy it with the vLLM worker above. + +### Pod for a Space + +Run the Space's image on a Pod and expose its port over HTTP: + +```bash +runpodctl pod create \ + --name gradio-app \ + --image DOCKER_USER/my-space:v1 \ + --gpu-id "NVIDIA GeForce RTX 4090" \ + --ports "7860/http" \ + --container-disk-in-gb 40 \ + --env '{"HF_TOKEN":"{{ RUNPOD_SECRET_hf_token }}"}' +``` + +The app is served at `https://POD_ID-7860.proxy.runpod.net`. + + +The Pod proxy URL is public and unauthenticated. If your Space relied on Hugging Face's access controls, add authentication in the app or put it behind your own proxy before you expose it. Proxy connections also time out after 100 seconds, so a request that takes longer needs a TCP port or a queue-based endpoint behind the app. + + +A Pod bills per second while it runs, whether or not anyone is using the app, like an upgraded Space with no sleep time. If you relied on a Space's sleep time, stop the Pod yourself with `runpodctl pod stop POD_ID` and start it again with `runpodctl pod start POD_ID`. + +## Step 7: Update the code that calls Hugging Face + +### OpenAI-compatible clients + +Swap the base URL and the key. Everything else in the call stays the same. + + + + ```python + import os + + from openai import OpenAI + + client = OpenAI( + base_url="https://NAME.REGION.VENDOR.endpoints.huggingface.cloud/v1/", + api_key=os.environ["HF_TOKEN"], + ) + + response = client.chat.completions.create( + model="llama-3-1-8b-instruct", + messages=[{"role": "user", "content": "Why is the sky blue?"}], + max_tokens=200, + ) + print(response.choices[0].message.content) + ``` + + + ```python + import os + + from openai import OpenAI + + client = OpenAI( + base_url="https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1", + api_key=os.environ["RUNPOD_API_KEY"], + ) + + response = client.chat.completions.create( + model="meta-llama/Llama-3.1-8B-Instruct", + messages=[{"role": "user", "content": "Why is the sky blue?"}], + max_tokens=200, + ) + print(response.choices[0].message.content) + ``` + + + +`model` is the repo ID you deployed, or the value of `OPENAI_SERVED_MODEL_NAME_OVERRIDE`. Streaming works with `stream=True`. Raise the client's timeout for the first request after a cold start, and drop the `X-Scale-Up-Timeout` header. + +### `InferenceClient` + +`InferenceClient` accepts a `base_url` and exposes the same `chat.completions.create` method, so you can keep it and point it at Runpod: + +```python +import os + +from huggingface_hub import InferenceClient + +client = InferenceClient( + base_url="https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1", + api_key=os.environ["RUNPOD_API_KEY"], +) + +response = client.chat.completions.create( + model="meta-llama/Llama-3.1-8B-Instruct", + messages=[{"role": "user", "content": "Why is the sky blue?"}], + max_tokens=200, +) +print(response.choices[0].message.content) +``` + +Task methods that call TGI's native routes, such as `client.text_generation()` or `endpoint.client.text_generation()`, have no route on the vLLM worker. Replace them with `chat.completions.create` or the OpenAI client's `completions.create`. + +### TGI `/generate` callers + +Move raw `/generate` requests to the completions route and rename the parameters: + + + + ```bash + curl "https://NAME.REGION.VENDOR.endpoints.huggingface.cloud/generate" \ + -H "Authorization: Bearer $HF_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"inputs": "What is deep learning?", "parameters": {"max_new_tokens": 200}}' + ``` + + + ```bash + curl "https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1/completions" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "prompt": "What is deep learning?", "max_tokens": 200}' + ``` + + + +The response is an OpenAI completion object, so read `choices[0].text`. `/generate_stream` callers move to the same route with `"stream": true`. + +### Custom handler clients + +Wrap the body in `input`, change the URL and key, and read the result from `output`: + + + + ```python + import os + + import requests + + response = requests.post( + "https://NAME.REGION.VENDOR.endpoints.huggingface.cloud", + headers={"Authorization": f"Bearer {os.environ['HF_TOKEN']}"}, + json={"inputs": "I like you. I love you.", "parameters": {"top_k": 2}}, + timeout=120, + ) + response.raise_for_status() + print(response.json()) + ``` + + + ```python + import os + + import requests + + response = requests.post( + "https://api.runpod.ai/v2/ENDPOINT_ID/runsync", + headers={"Authorization": f"Bearer {os.environ['RUNPOD_API_KEY']}"}, + json={"input": {"inputs": "I like you. I love you.", "parameters": {"top_k": 2}}}, + timeout=120, + ) + response.raise_for_status() + result = response.json() + if result["status"] != "COMPLETED": + raise RuntimeError(f"Job {result['id']} is {result['status']}") + print(result["output"]) + ``` + + + +`/runsync` returns the result when the job finishes within the wait window. If the job is still queued or running when the window closes, the response has a status of `IN_QUEUE` or `IN_PROGRESS`, and you fetch the result from `/status/JOB_ID`. For anything longer than a few seconds, submit to `/run`, which returns a job ID right away, then poll `/status/JOB_ID` or pass a `webhook` URL in the request. Results are kept for 30 minutes after an async job and 1 minute after a sync job. Payloads are limited to 10 MB on `/run` and 20 MB on `/runsync`, so send large inputs as URLs. The same calls are available from Python through `runpod.Endpoint("ENDPOINT_ID").run_sync({...})`. See [Send API requests](/serverless/endpoints/send-requests). + +### Custom container and TEI clients + +Only the host and the key change. Paths and payloads stay the same: + +```python +import os + +from openai import OpenAI + +# Before: base_url="https://NAME.REGION.VENDOR.endpoints.huggingface.cloud/v1/", api_key=HF_TOKEN +client = OpenAI( + base_url="https://ENDPOINT_ID.api.runpod.ai/v1", + api_key=os.environ["RUNPOD_API_KEY"], +) + +result = client.embeddings.create( + model="sentence-transformers/all-MiniLM-L6-v2", + input=["Runpod pulls models from the Hugging Face Hub."], +) +print(len(result.data[0].embedding)) +``` + +Add retries for the cold start window. A request that arrives before a worker is healthy can return `no workers available`. Use at least 3 retries with a 5 to 10 second delay. + +### Inference Providers clients + +For a model that's listed as a Public Endpoint with OpenAI compatibility, such as [Qwen3 32B](/public-endpoints/models/qwen3-32b), swap the router URL for the model's endpoint: + + + + ```python + import os + + from openai import OpenAI + + client = OpenAI( + base_url="https://router.huggingface.co/v1", + api_key=os.environ["HF_TOKEN"], + ) + + completion = client.chat.completions.create( + model="Qwen/Qwen3-32B:fastest", + messages=[{"role": "user", "content": "Hello!"}], + ) + ``` + + + ```python + import os + + from openai import OpenAI + + client = OpenAI( + base_url="https://api.runpod.ai/v2/qwen3-32b-awq/openai/v1", + api_key=os.environ["RUNPOD_API_KEY"], + ) + + completion = client.chat.completions.create( + model="Qwen/Qwen3-32B-AWQ", + messages=[{"role": "user", "content": "Hello!"}], + ) + ``` + + + +Each Public Endpoint has its own URL, so there's no single router and no provider suffix (`:fastest`, `:cheapest`, `:groq`) on the model name. The Public Endpoint may serve a different variant of the model than the provider you used (here, an AWQ quantization), so compare outputs before you switch. Image, video, and audio models use `/runsync` with an `input` object. See [Make API requests](/public-endpoints/requests). + +### Control plane automation + +| `huggingface_hub` or `hf` CLI | Runpod | +| --- | --- | +| `create_inference_endpoint(...)`, `hf endpoints deploy` | `runpodctl serverless create`, or `POST /v2/serverless` | +| `create_inference_endpoint_from_catalog(...)`, `hf endpoints catalog deploy` | `runpodctl serverless create --hub-id HUB_ID` | +| `list_inference_endpoints()`, `hf endpoints ls` | `runpodctl serverless list`, or `GET /v2/serverless` | +| `get_inference_endpoint(name)`, `hf endpoints describe` | `runpodctl serverless get ENDPOINT_ID --include-template --include-workers` | +| `endpoint.wait()` | Poll `GET https://api.runpod.ai/v2/ENDPOINT_ID/health` until `workers.idle` (ready to start) or `workers.running` is above 0, or submit to `/run` and let the job wait in the queue | +| `endpoint.update(min_replica=1, max_replica=6)` | `runpodctl serverless update ENDPOINT_ID --workers-min 1 --workers-max 6` | +| `endpoint.update(scale_to_zero_timeout=15)` | `runpodctl serverless update ENDPOINT_ID --idle-timeout 900` | +| `endpoint.scale_to_zero()` | `runpodctl serverless update ENDPOINT_ID --workers-min 0` (workers stop after the idle timeout) | +| `endpoint.pause()` | `runpodctl serverless update ENDPOINT_ID --workers-min 0 --workers-max 0` | +| `endpoint.resume()` | `runpodctl serverless update ENDPOINT_ID --workers-max 3` | +| `endpoint.delete()`, `hf endpoints delete` | `runpodctl serverless delete ENDPOINT_ID` | +| `list_inference_endpoints_hardware()`, `hf endpoints hardware` | `runpodctl gpu list`, or `GET /v2/catalog/gpus` | + +After any update that sets a worker count to 0, confirm it with `runpodctl serverless get ENDPOINT_ID`, and update `runpodctl` first if it's an old build. + +A batch script that resumes an endpoint, waits, runs inference, and pauses it again gets simpler: submit the jobs to `/run` on an endpoint with zero active workers. Workers start for the queue and stop when it's empty. + +## Step 8: Verify, cut over, and shut down + +1. **Send a real request from outside Runpod.** Check the endpoint's health, then send the same prompt or payload you send to Hugging Face: + + ```bash + curl "https://api.runpod.ai/v2/ENDPOINT_ID/health" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" + + curl "https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1/chat/completions" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"model": "meta-llama/Llama-3.1-8B-Instruct", "messages": [{"role": "user", "content": "Reply with the word ready."}], "max_tokens": 10}' + ``` + +2. **Compare outputs and latency.** Run a fixed set of requests against both platforms with the same sampling settings (set `temperature` to 0 where you can). Expect small differences when the engine changes, for example TGI to vLLM, or when the quantization differs. Measure a cold start as well as warm latency: the response includes `delayTime` and `executionTime`, and the endpoint's **Metrics** tab shows cold start percentiles. +3. **Run both in parallel.** Keep the Inference Endpoint up and send a share of traffic to Runpod through a feature flag or your gateway. +4. **Shift the rest of the traffic**, then watch error rates and queue delay for a full traffic cycle. +5. **Stop billing on Hugging Face.** Pause first so you can roll back, then delete once you're sure: + + ```bash + # Inference Endpoints: a paused endpoint isn't billed and keeps its configuration + hf endpoints pause ENDPOINT_NAME --namespace NAMESPACE + hf endpoints delete ENDPOINT_NAME --namespace NAMESPACE --yes + + # Spaces: paused time isn't billed + hf spaces pause NAMESPACE/SPACE_NAME + + # Jobs + hf jobs cancel JOB_ID + hf jobs scheduled delete SCHEDULED_JOB_ID + ``` + + Deleting an Inference Endpoint is permanent and removes its configuration, logs, and usage metrics. Keep the model repositories and the token that Runpod uses to read them. Revoke tokens that only the old clients used. + +6. **Set cost guards on Runpod.** Keep `--workers-min` at 0 unless you need warm capacity, set `--workers-max` to the concurrency you're willing to pay for, set an execution timeout on every endpoint, and keep the idle timeout short. Delete loader Pods and unused network volumes, which bill for storage until you remove them. Track spend on the [billing page](/accounts-billing/billing). + +## Reference + +### CLI command map + +| `hf` CLI | `runpodctl` or `rp` | +| --- | --- | +| `hf auth login` | `runpodctl doctor`, `rp login` | +| `hf endpoints ls` | `runpodctl serverless list` | +| `hf endpoints describe NAME` | `runpodctl serverless get ENDPOINT_ID` | +| `hf endpoints deploy NAME --repo ...` | `runpodctl serverless create --template-id ...` or `--hub-id ...` | +| `hf endpoints catalog ls` | `runpodctl hub list`, `runpodctl hub search` | +| `hf endpoints catalog deploy --repo ...` | `runpodctl serverless create --hub-id HUB_ID` | +| `hf endpoints update NAME --min-replica 2 --max-replica 6` | `runpodctl serverless update ENDPOINT_ID --workers-min 2 --workers-max 6` | +| `hf endpoints pause`, `resume`, `scale-to-zero` | `runpodctl serverless update` with `--workers-min` and `--workers-max` | +| `hf endpoints delete NAME --yes` | `runpodctl serverless delete ENDPOINT_ID` | +| `hf endpoints hardware` | `runpodctl gpu list`, `runpodctl datacenter list` | +| `hf jobs run`, `hf jobs uv run` | `rp flash dev main.py --once` for an `@app.task`, or `runpodctl pod create` | +| `hf jobs logs JOB_ID` | `rp logs POD_ID --follow` | +| `hf spaces pause`, `hf spaces restart` | `runpodctl pod stop POD_ID`, `runpodctl pod start POD_ID` | +| `hf spaces secrets add` | `rp secret add SECRET_NAME` | +| `hf download REPO_ID` | Unchanged. Run it on a Pod to fill a network volume. | + +### API map + +The Inference Endpoints API is at `https://api.endpoints.huggingface.cloud`. Runpod's REST API v2 is at `https://api.runpod.io`. Both use `Authorization: Bearer`. + +| Inference Endpoints API | Runpod REST API v2 | +| --- | --- | +| `POST /v2/endpoint/{namespace}` | `POST /v2/serverless` | +| `GET /v2/endpoint/{namespace}` | `GET /v2/serverless` | +| `GET /v2/endpoint/{namespace}/{name}` | `GET /v2/serverless/{id}` | +| `PUT /v2/endpoint/{namespace}/{name}` | `PATCH /v2/serverless/{id}` | +| `DELETE /v2/endpoint/{namespace}/{name}` | `DELETE /v2/serverless/{id}` | +| `POST .../pause`, `.../resume`, `.../scale-to-zero` | `PATCH /v2/serverless/{id}` with `workers.min` and `workers.max` | +| `GET /v2/endpoint/{namespace}/{name}/replica` | `GET /v2/serverless/{id}/workers` | +| `GET /v2/endpoint/{namespace}/{name}/logs` | `GET /v2/serverless/{id}/workers/{workerId}/logs` | +| `POST /v2/endpoint/{namespace}/{name}/metrics/...`, `GET .../open-metrics` | `GET /v2/serverless/{id}/metrics/requests`, `/metrics/cold-starts`, `/metrics/workers` | +| `GET /v2/provider` (vendors, regions, compute) | `GET /v2/catalog/gpus`, `GET /v2/catalog/datacenters` | + +### Configuration field map + +| `create_inference_endpoint` argument | `runpodctl` flag | REST v2 field | +| --- | --- | --- | +| `name` | `--name` | `name` | +| `repository`, `revision` | `--model-reference URL:REF`, `--env MODEL_NAME=...` | `env.MODEL_NAME` | +| `accelerator` | `--compute-type` | `gpu` or `cpu` | +| `instance_type` | `--gpu-id` | `gpu.pools` | +| `instance_size` | `--gpu-count` | `gpu.count` | +| `vendor`, `region` | `--data-center-ids` | `dataCenterIds` | +| `min_replica` | `--workers-min` | `workers.min` | +| `max_replica` | `--workers-max` | `workers.max` | +| `scale_to_zero_timeout` (minutes) | `--idle-timeout` (seconds) | `workers.idleTimeout` | +| `scaling_metric`, `scaling_threshold` | `--scale-by`, `--scale-threshold` | `scaling.type`, `scaling.queueDelay` or `scaling.requestCount` | +| `custom_image["url"]` | `--image` on `template create` | `image` | +| `custom_image["port"]` | `--ports` on `template create` | `ports`, `env.PORT`, `env.PORT_HEALTH` | +| `custom_image["healthRoute"]` | Template `--env` | `env.HEALTH_CHECK_PATH` | +| `container_command`, `container_args` | `--docker-entrypoint`, `--docker-start-cmd` on `template create` | `entrypoint`, `cmd` | +| `container_registry_username`, `container_registry_password` | `runpodctl registry create`, then `--registry-auth-id` | `registry` | +| `env` | `--env` | `env` | +| `secrets` | `--env KEY='{{ RUNPOD_SECRET_name }}'` | `env` with a secret reference | +| `type` (`public`, `authenticated`, `private`) | Not applicable | Not applicable (API key required) | +| `task`, `framework` | Not applicable | Not applicable (decided by the worker image) | +| No equivalent | `--execution-timeout` (seconds) | `timeout` (milliseconds) | + +### Environment variable map + +| Inside an Inference Endpoints container | Inside a Runpod worker | +| --- | --- | +| Model files at `/repository` | Cached model at `/runpod-volume/huggingface-cache/hub/models--ORG--NAME/snapshots/HASH/`. Network volume at `/runpod-volume`. | +| `HF_TOKEN` from Secret Env | `HF_TOKEN` from a Runpod secret | +| TGI: `MODEL_ID`, `MAX_INPUT_LENGTH`, `MAX_TOTAL_TOKENS` | vLLM worker: `MODEL_NAME`, `MAX_MODEL_LEN` | +| Port and health route from the container configuration | `PORT`, `PORT_HEALTH`, `HEALTH_CHECK_PATH` (load balancing endpoints) | +| Not applicable | `RUNPOD_INIT_TIMEOUT`: seconds a worker may take to initialize (default 420) | + +### Status map + +Inference Endpoints reports one state per endpoint. Runpod reports a state per worker, and a state per job on queue-based endpoints. + +| Inference Endpoint state | Runpod equivalent | +| --- | --- | +| `pending` | Endpoint created, no worker yet. Workers show **Initializing** while the image and cached model are pulled. | +| `initializing` | Worker **Initializing**, then **Running** during the cold start. Jobs wait as `IN_QUEUE`. | +| `running` | Worker **Running** (processing or waiting out the idle timeout) or **Idle** (ready to start). | +| `scaledToZero` | All workers **Idle**. Not billed. The next request starts a worker. | +| `paused` | Max workers set to 0. | +| `updating` | A new release is rolling out. Workers on the old version show **Outdated** until they're replaced. | +| `failed`, `updateFailed` | Workers **Unhealthy**. Runpod retries them with increasing delays. | +| No equivalent | Worker **Throttled**: the host's GPUs are busy, so the worker can't start there yet. | + +Job states on queue-based endpoints are `IN_QUEUE`, `IN_PROGRESS`, `COMPLETED`, `FAILED`, `CANCELLED`, and `TIMED_OUT`. + +## Migrate with a coding agent + +Install the Runpod skills and connect the MCP server: + + + + ```bash + # The plugin also connects the hosted MCP server + claude plugin marketplace add runpod/runpod-plugins-official + claude plugin install runpod@runpod + ``` + + + ```bash + # Codex, Cursor, and others: skills, then the hosted MCP server + npx skills add runpod/runpod-plugins-official --skill '*' --agent AGENT_ID --global + npx @runpod/mcp-server@latest add --client CLIENT_ID + ``` + + + +The MCP server signs in with OAuth (**Sign in with Runpod**), so you don't need an API key to start. See [Agent skills](/get-started/agent-skills) and [MCP servers](/get-started/mcp-servers). + +Then give your agent this prompt from the root of your repository: + +```text +Use the runpod-cloud-migrate skill to migrate this project from Hugging Face +Inference Endpoints to Runpod. Models, datasets, and tokens stay on the Hugging +Face Hub. Only the compute moves. + +1. Inventory first. Run `hf endpoints ls --namespace '*'` and + `hf endpoints describe` for each endpoint, list GPU Spaces and Jobs, and grep + this repo for endpoints.huggingface.cloud, router.huggingface.co, + InferenceClient, create_inference_endpoint, EndpointHandler, and hf jobs. + Don't change anything yet. +2. For each workload, tell me the engine (vLLM, TGI, SGLang, TEI, llama.cpp, + Inference Toolkit with handler.py, or custom container) and the Runpod target + you propose: vLLM worker, queue-based endpoint, load balancing endpoint, + Public Endpoint, Pod, or @app.task. +3. Show me a plan with the GPU pool, worker limits, idle timeout, and a cost + estimate for each endpoint. Wait for my approval before you create anything + billable. +4. Migrate one workload first. Keep the same Hugging Face repo ID, attach it as + a cached model, and read HF_TOKEN from a Runpod secret. Never print the token. +5. Verify with a real request to the new endpoint and compare the output with + the Hugging Face endpoint. +6. Update the client code (base URL, API key, request envelope) on a branch. +7. Don't pause or delete any Hugging Face endpoint, Space, or Job unless I + approve it explicitly. +``` + +## Troubleshooting + +**The worker fails with `Cached model not found` or tries to load `/repository`.** The handler still expects Hugging Face's mount path, or the endpoint has no cached model attached. Attach the model with `--model-reference` or the **Model** field, and load from the snapshot directory under `/runpod-volume/huggingface-cache/hub/`. + +**A gated model returns 401 or 403 while a worker starts.** The endpoint has no `HF_TOKEN`, or the token's account hasn't accepted the model's terms. Set `HF_TOKEN` to `{{ RUNPOD_SECRET_hf_token }}` and confirm the token can read the repo. Workers pick up a changed secret on their next start. + +**The first request after idle returns a 524 or times out.** The cold start took longer than the synchronous wait. Send the first request to `/run` and poll `/status`, raise the idle timeout, or set one active worker. + +**OpenAI calls return an invalid model error.** The client still sends the Inference Endpoint's name as `model`. Send the Hugging Face repo ID, or set `OPENAI_SERVED_MODEL_NAME_OVERRIDE` to the name your clients use. + +**`/generate` returns 404 or an error.** TGI's native routes aren't served by the vLLM worker. Call `/openai/v1/completions` or `/openai/v1/chat/completions`. + +**A load balancing worker never becomes healthy.** The load balancer polls `/ping` on port 80 by default. Set `HEALTH_CHECK_PATH` to your health route, set `PORT` and `PORT_HEALTH` to your server's port, and expose that port on the endpoint. With misconfigured ports, workers stay up for 8 minutes and return `502`. + +**Requests return `no workers available`.** A load balancing endpoint has no healthy worker yet. Retry with a delay, or keep an active worker. If the request can wait, a queue-based endpoint holds it for you. + +**The vLLM worker runs out of GPU memory on a model that fit before.** vLLM pre-allocates VRAM for its KV cache, and your new GPU may have less memory than the old instance. Lower `MAX_MODEL_LEN` or `GPU_MEMORY_UTILIZATION`, or move up a pool. + +**Workers stay Throttled or Initializing for a long time.** The selected GPU is busy in the allowed data centers, or the model is still downloading to the host. Add a second and third GPU choice, remove data center restrictions, and for large models raise `RUNPOD_INIT_TIMEOUT`. + +**An endpoint that worked last week returns no workers.** Runpod set max workers to 0 after 7 days without requests. Raise max workers again. + +## Next steps + + + + Configure and tune LLM endpoints. + + + Serve Hugging Face models without baking them into images. + + + Set workers, timeouts, and autoscaling. + + + Define endpoints and tasks in Python. + + diff --git a/migrate/lambda.mdx b/migrate/lambda.mdx new file mode 100644 index 000000000..02bdd45c2 --- /dev/null +++ b/migrate/lambda.mdx @@ -0,0 +1,1008 @@ +--- +title: "Migrate from Lambda to Runpod" +sidebarTitle: "Lambda" +description: "Move Lambda On-Demand Cloud instances, filesystems, firewall rules, and 1-Click Clusters to Runpod Pods, network volumes, and Instant Clusters." +--- + +This guide is for teams that run training, fine-tuning, or self-hosted inference on [Lambda](https://lambda.ai), the GPU cloud formerly called Lambda Labs. It maps [On-Demand Cloud](https://docs.lambda.com/public-cloud/on-demand/) instances to Pods, [filesystems](https://docs.lambda.com/public-cloud/filesystems/) to network volumes, [firewall rules](https://docs.lambda.com/public-cloud/firewalls/) to exposed ports, [1-Click Clusters](https://docs.lambda.com/public-cloud/1-click-clusters/) to Instant Clusters, and [Lambda Cloud API](https://docs.lambda.com/api/cloud) calls to the Runpod REST API v2. At the end, your workloads run on Runpod, your data lives on a network volume, and your automation calls Runpod instead of Lambda. + + +This page is about Lambda the GPU cloud. If you're moving functions off AWS Lambda, see [Migrate from AWS to Runpod](/migrate/aws). + + + +A coding agent can do most of this migration for you. See [Migrate with a coding agent](#migrate-with-a-coding-agent). + + +## How Lambda maps to Runpod + +The biggest change is the unit of compute. A Lambda instance is a virtual machine that you configure after it boots. A Runpod Pod is a container that starts from an image, so your environment moves into a Dockerfile or a template. + +| Lambda concept | Runpod equivalent | Notes | +| --- | --- | --- | +| On-Demand Cloud instance (VM, Ubuntu, `ubuntu` user with `sudo`) | [Pod](/pods/overview) (container, `root` user) | Billed while it runs on both platforms. A Pod can also be stopped and restarted. | +| Instance type (for example `gpu_1x_a10`) | GPU type ID plus GPU count (for example `NVIDIA RTX A5000`, count 1) | vCPU and RAM come from the host machine. Set a floor with `gpu.minVcpuCountPerGpu` and `gpu.minRamPerGpu` in REST v2. | +| Lambda Stack image (`lambda-stack-24-04`, `lambda-stack-22-04`) | Official Runpod PyTorch [templates](/pods/templates/overview) and `runpod/pytorch` images | PyTorch, CUDA, SSH, and JupyterLab are preinstalled. Add TensorFlow or JAX yourself if you use them. | +| GPU Base and Ubuntu Server images | Any `linux/amd64` container image | For example an `nvidia/cuda` or `ubuntu` base image. | +| cloud-init `user_data` | Dockerfile, template start command, and environment variables | Build setup into the image instead of running it at boot. | +| Root volume (512 GiB to 22 TiB, lost on terminate) | Container disk (ephemeral) plus volume disk at `/workspace` | You choose both sizes at create time. The container disk defaults to 20 GB in `runpodctl`. | +| Filesystem (`/lambda/nfs/NAME`, region-bound, attach at launch) | [Network volume](/storage/network-volumes) (`/workspace`, pinned to one data center, attach at create) | A network volume has a provisioned size. A Lambda filesystem bills for what you use. | +| Filesystem S3 Adapter (`files.REGION.lambda.com`) | [S3-compatible API](/storage/s3-api) for network volumes (`s3api-DC.runpod.io`) | Both are available in select locations only. | +| Firewall rules and rulesets (TCP, UDP, ICMP, source CIDR) | [Exposed ports](/pods/configuration/expose-ports) (`PORT/http`, `PORT/tcp`) | Runpod exposes TCP and HTTP. Access control lives in your application. | +| Public IPv4 on every instance | HTTPS proxy URL, or public IP with a mapped external port | See [What works differently](#what-works-differently). | +| Private IP between instances | [Global networking](/pods/networking) (`POD_ID.runpod.internal`) | Private network between your NVIDIA GPU Pods, in supported data centers. | +| SSH keys (per workspace, one key per launch) | [SSH public keys](/get-started/credentials#ssh-public-keys) (per account, all keys injected) | | +| JupyterLab (**Launch** in the console) | JupyterLab on port 8888 in official PyTorch templates | Set `JUPYTER_PASSWORD` when you create Pods outside the console. | +| 1-Click Cluster (reserved by the week) | [Instant Cluster](/instant-clusters) (on demand) | See the size ranges in [What works differently](#what-works-differently). | +| Managed Slurm, unmanaged Slurm | [Slurm Clusters](/instant-clusters/slurm-clusters) | | +| Self-hosted inference server on an instance | Pod, or [Serverless](/serverless/overview) endpoint that scales to zero | Serverless has no Lambda counterpart, so this is a redesign, not a rename. | +| Regions (`us-east-1`, `europe-central-1`, and others) | Data centers (`US-GA-1`, `EU-RO-1`, and others) | List them with `runpodctl datacenter list`. | +| Per-minute billing, weekly invoices | Per-second billing from a prepaid credit balance | Organizations on Runpod are invoiced post-paid. | +| Lambda Cloud API (`https://cloud.lambda.com/api/v1`) | [REST API v2](/api-reference-v2/overview) (`https://api.runpod.io/v2`) | Plus `runpodctl`, the Runpod SDK, an MCP server, and a Terraform provider. | +| API key (`Authorization: Bearer` or Basic) | API key (`Authorization: Bearer`), `RUNPOD_API_KEY` | Runpod keys can be restricted or read-only. | +| Account, workspaces, IAM roles, SSO | Account, teams, [Organizations](/organizations/orgs-overview) with roles and groups | | +| Audit events (`GET /api/v1/audit-events`) | Audit logs in REST v2 | | +| Guest Agent metrics in the console | Telemetry tab and logs in the console, `runpodctl pod logs` | | +| Private Cloud (reserved single-tenant clusters) | Custom plans through [Runpod sales](https://ecykq.share.hsforms.com/2MZdZATC3Rb62Dgci7knjbA) | | + +Lambda sunset its hosted Inference API (the shutdown date was September 25, 2025), so this guide covers inference that you host yourself on instances. If you still have code that calls a Lambda-hosted model, [Public Endpoints](/public-endpoints/overview) and [Serverless vLLM](/serverless/vllm/get-started) are the Runpod options. + +## What's different + +### What works differently + +**You get a container, not a VM.** On Lambda you have a full virtual machine with its own kernel, `systemd`, and a Docker daemon. On a Pod, Runpod runs Docker for you and your image's start command is the container's main process. That has four practical consequences: + +- Docker and Docker Compose don't run inside a Pod. If your instance runs `docker run` or `docker compose up`, deploy each image as its own Pod, or combine the services into one image with a start script. +- Services you manage with `systemctl` need to start from your image's start command instead. Official Runpod images start SSH and JupyterLab from `/start.sh`. +- The container shares the host's kernel and NVIDIA driver. You pick a host by CUDA version (`--min-cuda-version`) instead of installing a driver or kernel module. +- Anything you installed by hand over SSH has to be captured in a Dockerfile, or reinstalled onto `/workspace`. + +**A Pod can stop without losing its volume.** Lambda instances can be launched, restarted, or terminated, and Lambda's docs recommend containers and backups to preserve state. A Pod can also be stopped: the GPU is released, the volume disk at `/workspace` is kept, and you pay for volume storage while it's stopped. When you start it again, the GPUs on that host may be in use. See [zero GPU Pods](/pods/troubleshooting/zero-gpus). + +**Only `/workspace` persists.** On a Lambda instance, the whole root volume lasts until you terminate. On a Pod, the container disk is cleared when the Pod stops, restarts, or has its configuration edited. Keep anything you need on the volume disk or a network volume. + +**Ports are exposed, not firewalled.** Lambda opens ports with firewall rules that can restrict the source IP range. A Pod exposes the ports you list, to the public internet, in one of two ways: + +- `PORT/http` gives you `https://POD_ID-PORT.proxy.runpod.net`. It's HTTPS only and has a 100-second response timeout. +- `PORT/tcp` gives you a public IP and a mapped external port that usually differs from the internal port. Request a port above 70000 to get matching internal and external ports, and read the assigned port from `RUNPOD_TCP_PORT_70000`. + +**SSH connects as `root` on a mapped port.** Instead of `ssh ubuntu@INSTANCE_IP`, you use `ssh root@POD_IP -p EXTERNAL_PORT`, or the proxied `ssh POD_USER@ssh.runpod.io` form. The proxied form gives you a shell only. SCP, SFTP, rsync, and port forwarding need the direct connection. + +**Storage has a provisioned size.** A Lambda filesystem has no size to set and bills for the GiB you use. A network volume has a size from 10 GB to 4,096 GB that you choose and can grow later (contact support for more than 4 TB). A [global volume](/storage/globalvolume/overview) (beta) is the usage-billed option on Runpod. + +**Billing is prepaid.** Lambda bills in one-minute increments and sends weekly invoices. Runpod bills per second against a credit balance. You need at least one hour of credit for a configuration to deploy it, and Pods stop when your balance reaches \$0. Set up [auto-pay or low balance notifications](/accounts-billing/billing) before you move production. + +**Clusters start on demand, at a different size range.** A 1-Click Cluster is reserved in weeks, covers 16 to 512 H100 or B200 GPUs, and includes three CPU head nodes. An Instant Cluster deploys on demand with 2 to 8 nodes (16 to 64 GPUs) of B200, H200, H100, or A100, and one of the GPU nodes is the primary node. Larger clusters go through Runpod sales. Availability depends on current stock in each data center. + +### What has no direct equivalent + +**On Runpod (Lambda features to plan around):** + +- **Docker, Docker Compose, and kernel access inside the machine.** Use one image per Pod. To build images from inside a Pod, see [Build Docker images with Bazel](/tutorials/pods/build-docker-images), or build in CI and push to a registry. +- **UDP and ICMP.** Pods accept TCP and HTTP connections. Move UDP protocols to TCP, or keep that service elsewhere. +- **Source IP allowlists.** Exposed ports are reachable by anyone who has the address. Require authentication in your service, keep internal services on global networking without exposing a port, or reach them through an SSH tunnel. +- **ARM instances (GH200).** Runpod images must be `linux/amd64`. Rebuild ARM images for x86-64. +- **Several filesystems on one instance.** `runpodctl pod create` and the console attach one network volume per Pod, plus one global volume if you need a second mount. REST API v2 accepts up to four network volumes in `mounts.network`, all in the Pod's data center. The simplest plan is to merge your filesystems into one network volume with subdirectories. +- **Managed Kubernetes on a cluster.** Instant Clusters come as Slurm, Ray, PyTorch training, or general application clusters. Port Kubernetes workloads to one of those, or to Pods and Serverless endpoints. +- **Fixed vCPU and RAM per instance type.** Pod CPU and memory depend on the host. Use the REST v2 placement floors if your job needs a minimum. + +**On Lambda (Runpod features you can adopt):** + +- **Scale to zero.** A [Serverless](/serverless/overview) endpoint runs your image only while requests are in flight. A self-hosted inference server that sits idle on an instance is a good candidate. +- **Stop and resume.** Stop a Pod overnight and keep its volume. +- **Hosted models.** [Public Endpoints](/public-endpoints/overview) serve models over an API with no deployment. +- **Restricted API keys.** Lambda API keys have full access to all API operations. Runpod keys can be read-only or restricted. + +## Requirements + +- A Lambda account with access to the workspaces you're migrating, and a [Lambda Cloud API key](https://cloud.lambda.com/api-keys). +- A [Runpod account](/accounts-billing/manage-accounts) with credits and a [Runpod API key](/get-started/credentials#api-keys). +- An SSH key pair. You can reuse the public key you already use on Lambda. +- `curl` and `jq` for the inventory commands. +- Docker on your local machine or in CI, and a container registry, if you build a custom image. + +## Step 1: Take inventory of what you run on Lambda + +Lambda's docs describe two interfaces, the console and the Cloud API. The API gives you an inventory you can save and diff. Export your key first. + +```bash +export LAMBDA_API_KEY="YOUR_LAMBDA_API_KEY" +export LAMBDA_API="https://cloud.lambda.com/api/v1" +``` + +List every running instance with the fields you need to recreate it. + +```bash +curl -s -H "Authorization: Bearer $LAMBDA_API_KEY" "$LAMBDA_API/instances" \ + | jq '.data[] | {id, name, status, ip, region: .region.name, + type: .instance_type.name, gpus: .instance_type.specs.gpus, + vcpus: .instance_type.specs.vcpus, memory_gib: .instance_type.specs.memory_gib, + image: .image, filesystems: .file_system_names, ssh_keys: .ssh_key_names, + firewall_rulesets, tags}' > lambda-instances.json +``` + +To list the nodes of one 1-Click Cluster, add the `cluster_id` query parameter. The Lambda API allows about one request per second, so run these commands one at a time. + +Next, export storage, SSH keys, and firewall configuration. + +```bash +# Filesystems: name, region, mount point, and approximate bytes used +curl -s -H "Authorization: Bearer $LAMBDA_API_KEY" "$LAMBDA_API/filesystems" \ + | jq '.data[] | {id, name, region: .region.name, mount_point, is_in_use, bytes_used}' \ + > lambda-filesystems.json + +# SSH public keys +curl -s -H "Authorization: Bearer $LAMBDA_API_KEY" "$LAMBDA_API/ssh-keys" \ + | jq '.data[] | {name, public_key}' > lambda-ssh-keys.json + +# Global firewall rules and per-instance rulesets +curl -s -H "Authorization: Bearer $LAMBDA_API_KEY" "$LAMBDA_API/firewall-rules" \ + | jq '.data' > lambda-firewall-rules.json +curl -s -H "Authorization: Bearer $LAMBDA_API_KEY" "$LAMBDA_API/firewall-rulesets" \ + | jq '.data' > lambda-firewall-rulesets.json +``` + +`bytes_used` tells you how large each network volume needs to be. Lambda updates it every few hours, so add headroom. The firewall files tell you which ports each workload serves: every TCP rule becomes an exposed port, and every UDP rule or source restriction needs a decision (see [What has no direct equivalent](#what-has-no-direct-equivalent)). + +The API doesn't know what's installed inside each VM, so capture that over SSH. Run this on each instance you plan to move. + +```bash +ssh ubuntu@INSTANCE_IP 'bash -s' <<'EOF' > instance-snapshot.txt +echo "== OS and GPU"; lsb_release -ds; nvidia-smi --query-gpu=name,memory.total,driver_version --format=csv +echo "== Python packages"; python3 --version; pip freeze +echo "== Apt packages you added"; apt-mark showmanual +echo "== Containers"; sudo docker ps --format '{{.Image}} {{.Ports}} {{.Command}}' +echo "== Services"; systemctl list-units --type=service --state=running --no-pager +echo "== Cron"; crontab -l 2>/dev/null +echo "== Listening ports"; sudo ss -tlnup +echo "== Disk"; df -h / /lambda/nfs/* 2>/dev/null +echo "== Environment"; env | cut -d= -f1 | sort +EOF +``` + +The snapshot is the input for your Dockerfile in Step 4. The environment section prints variable names only, so secrets stay out of the file. If you launched instances with cloud-init, find the `user_data` you passed at launch in your own scripts, because it's write-only in the API. + +Finally, find the code and configuration that reference Lambda. + +```bash +grep -rnE "cloud\.lambda(labs)?\.(com|ai)|LAMBDA_API_KEY|instance-operations/|/lambda/nfs|lambda-stack|files\.[a-z0-9-]+\.lambda\.com|lambdaspaces\.com" \ + --exclude-dir={.git,node_modules,.venv} . +``` + +The pattern covers all three API hostnames (`cloud.lambda.com`, `cloud.lambda.ai`, and `cloud.lambdalabs.com`), the filesystem mount path, the S3 Adapter endpoints, and JupyterLab URLs. Also search for hardcoded instance IP addresses and for `ubuntu@` in SSH configs and CI scripts. + +## Step 2: Set up Runpod + +Create an account, add credits, and create an API key on the [Credentials page](/get-started/credentials). Then install the CLI and register your SSH key. + +```bash +# Install runpodctl (or: brew install runpod/runpodctl/runpodctl) +curl -fsSL https://cli.runpod.net | sudo bash + +# Configure your API key and SSH access interactively +runpodctl doctor + +# Register the same public key you use on Lambda +runpodctl ssh add-key --key-file ~/.ssh/id_ed25519.pub +runpodctl ssh list-keys +``` + +Lambda attaches exactly one SSH key per launch. Runpod injects every key on your account into your Pods, so add each teammate's key once instead of editing `authorized_keys`. Keys must be in OpenSSH format (`ssh-ed25519 AAAA...`). If you saved Lambda keys in another format, convert them with `ssh-keygen -i`, or read the public key back from your private key with `ssh-keygen -y -f KEY_FILE`. + +Install the other tools as you need them: + +- **Runpod SDK**: `pip install runpod` installs the Python SDK and the `rp` CLI. Run `rp login` to store your key. See the [SDK overview](/sdk/overview). +- **MCP server**: lets a coding agent manage your Runpod resources. See [MCP servers](/get-started/mcp-servers). +- **Terraform**: see [Migrate your Terraform](#migrate-your-terraform). + +## Step 3: Choose where each workload lands + +Use the inventory to place each workload. Most Lambda instances become Pods, and the exceptions are worth finding early. + +| What the instance does on Lambda | Runpod target | Why | +| --- | --- | --- | +| Interactive development, notebooks, SSH sessions | [Pod](/pods/overview) with an official PyTorch template | Closest match to a Lambda Stack instance. | +| Long training or fine-tuning run on one machine | Pod with a network volume | Checkpoints survive the Pod. | +| Always-on inference server (vLLM, Triton, a custom API) | Pod with an exposed HTTP port | Same shape as today. | +| Inference server with idle periods or bursty traffic | [Serverless load-balancing endpoint](/serverless/load-balancing/overview) | Keeps your HTTP server and scales to zero. | +| OpenAI-compatible LLM serving | [Serverless vLLM](/serverless/vllm/get-started) | Prebuilt worker with an OpenAI-compatible route. | +| Batch jobs triggered by a queue or script | [Serverless queue endpoint](/serverless/overview) | Per-request billing with async job status. | +| Calls to a hosted model | [Public Endpoint](/public-endpoints/overview) | No deployment. | +| Multi-node training on a 1-Click Cluster | [Instant Cluster](/instant-clusters) | Slurm, PyTorch, or Ray across nodes. | +| CPU node next to a cluster | CPU Pod | `runpodctl pod create --compute-type cpu`. | + +Then map each instance type to a Runpod GPU type. Lambda instance type names encode the GPU count and model, for example `gpu_1x_a10` or `gpu_8x_h100_sxm5gdr`. Get the exact names you use from your inventory or from `GET /api/v1/instance-types`. On Runpod, the GPU model is the `--gpu-id` and the count is `--gpu-count`. + +| Lambda GPU (VRAM per GPU) | Runpod GPU type ID | Notes | +| --- | --- | --- | +| B200 (180 GB) | `NVIDIA B200` | | +| H100 SXM (80 GB) | `NVIDIA H100 80GB HBM3` | | +| H100 PCIe (80 GB) | `NVIDIA H100 PCIe` | | +| A100 SXM (80 GB) | `NVIDIA A100-SXM4-80GB` | | +| A100 SXM or PCIe (40 GB) | `NVIDIA A100 80GB PCIe` or `NVIDIA A100-SXM4-80GB` | Runpod lists the 80 GB A100. For a 48 GB card, use `NVIDIA A40`, `NVIDIA RTX A6000`, or `NVIDIA L40S`. | +| A6000 (48 GB) | `NVIDIA RTX A6000` | | +| A10 (24 GB) | `NVIDIA RTX A5000`, `NVIDIA L4`, or `NVIDIA GeForce RTX 4090` | Runpod's GPU list has no A10. Choose a 24 GB card that fits the workload. | +| RTX 6000 (24 GB) | `NVIDIA RTX A5000` or another 24 GB card | | +| Tesla V100 (16 GB) | `Tesla V100-PCIE-16GB` or `Tesla V100-SXM2-16GB` | | +| GH200 (96 GB, ARM) | `NVIDIA H100 NVL` (94 GB) or `NVIDIA RTX PRO 6000 Blackwell Server Edition` (96 GB) | Rebuild your image for `linux/amd64`. | + +The full list is in [GPU types](/references/gpu-types). Stock varies by data center and changes through the day, so check what's deployable before you commit to a type. + +```bash +# GPU types with stock right now +runpodctl gpu list + +# Stock per data center +runpodctl datacenter list + +# REST v2: availability for an 8-GPU Pod, with CUDA 12.8 or later +curl -s -H "Authorization: Bearer $RUNPOD_API_KEY" \ + "https://api.runpod.io/v2/catalog/gpus?include=AVAILABILITY&product=POD&count=8&minCudaVersion=12.8" +``` + +The catalog also reports `maxCount`, the largest GPU count a single Pod of that type can have. If a multi-GPU instance type has no single-Pod match, consider an Instant Cluster. + +Pick the data center before you create storage. A network volume is pinned to one data center, and Pods that mount it run there, the same way a Lambda filesystem ties you to its region. + +## Step 4: Move your images and code + +How much work this step takes depends on how you use the instance today. + +### If you already run containers on Lambda + +Lambda instances ship with Docker and the NVIDIA Container Toolkit, so many teams already run `docker run --gpus all`. That image runs on a Pod with three checks: + +- **Architecture**: the image must be `linux/amd64`. Build with `docker build --platform linux/amd64` if you build on an ARM machine. +- **Registry access**: for a private image, add a registry credential with `runpodctl registry create` and pass its ID as `--registry-auth-id`. For Amazon ECR, see [Use private ECR images](/tutorials/pods/use-private-ecr-images). +- **Run flags**: translate your `docker run` flags with the table below. + +| `docker run` on a Lambda instance | Runpod Pod | +| --- | --- | +| `--gpus all` | `--gpu-id` and `--gpu-count`. All requested GPUs are visible to the container. | +| `-p 8000:8000` | `--ports "8000/http"` or `--ports "8000/tcp"` | +| `-e KEY=value` | `--env '{"KEY":"value"}'`, or a [secret](/pods/templates/secrets) reference | +| `-v /lambda/nfs/NAME:/data` | `--network-volume-id`, mounted at `/workspace` | +| `-v $HOME/.cache/huggingface:/root/.cache/huggingface` | Set `HF_HOME=/workspace/huggingface` so the cache persists | +| Image arguments after the image name | `--docker-args "..."` | +| `--ipc=host`, `--privileged`, `--network host` | The [`runpodctl pod create` reference](/runpodctl/reference/runpodctl-pod) lists no matching flags. Test that your workload runs without them. | +| `docker compose up` | One Pod per service, connected with global networking | + +### If you work directly on the VM with Lambda Stack + +Pick the official PyTorch template closest to your Lambda Stack image and reinstall the rest of your environment on top. + +| Lambda image | Default Python | Closest Runpod template | Image | +| --- | --- | --- | --- | +| Lambda Stack 24.04 (`lambda-stack-24-04`) | 3.12 | `runpod-torch-v280` | `runpod/pytorch:1.0.2-cu1281-torch280-ubuntu2404` (Ubuntu 24.04, Python 3.12, PyTorch 2.8.0, CUDA 12.8.1) | +| Lambda Stack 22.04 (`lambda-stack-22-04`) | 3.10 | `runpod-torch-v220` | `runpod/pytorch:2.2.0-py3.10-cuda12.1.1-devel-ubuntu22.04` | + +Run `runpodctl template list --type official` to see the current list. Check the PyTorch version in your snapshot against the template, because Lambda Stack updates its packages over time. + +For anything you run more than once, capture the environment in an image. This Dockerfile turns the snapshot from Step 1 into a Pod image. + +```dockerfile Dockerfile +FROM runpod/pytorch:1.0.2-cu1281-torch280-ubuntu2404 + +ENV PYTHONUNBUFFERED=1 \ + HF_HOME=/workspace/huggingface + +# Apt packages from the "Apt packages you added" section of your snapshot +RUN apt-get update --yes && \ + DEBIAN_FRONTEND=noninteractive apt-get install --yes --no-install-recommends \ + git tmux htop rsync && \ + rm -rf /var/lib/apt/lists/* + +# Python packages from pip freeze, minus the ones the base image provides +COPY requirements.txt /tmp/requirements.txt +RUN pip install --no-cache-dir --break-system-packages -r /tmp/requirements.txt + +WORKDIR /workspace +``` + +Remove `torch`, `torchvision`, and the `nvidia-*` packages from `requirements.txt` so `pip` doesn't replace the CUDA build of PyTorch that the base image ships. The Ubuntu 24.04 base marks the system Python as externally managed, which is why the install uses `--break-system-packages`. The image keeps the base image's `/start.sh`, so SSH and JupyterLab still start. + +Build for `linux/amd64` and push the image to your registry. + +```bash +docker build --platform linux/amd64 -t YOUR_REGISTRY/lambda-env:v1 . +docker push YOUR_REGISTRY/lambda-env:v1 +``` + +If your instance starts a long-running process from `systemd` or cron, add a start script that launches the base services and then your process. See [Create a custom template](/pods/templates/create-custom-template#step-4-configure-container-startup-behavior) for the three startup patterns. + +### Paths and environment variables + +Update anything that assumed the Lambda VM layout. + +| On a Lambda instance | On a Runpod Pod | +| --- | --- | +| `/home/ubuntu` (user `ubuntu`, `sudo` for root) | `/root` (user `root`, no `sudo` needed) | +| `/lambda/nfs/FILESYSTEM_NAME` | `/workspace` | +| Root volume persists until terminate | Only `/workspace` persists across stop and restart | +| `sudo systemctl restart SERVICE` | Restart the process, or `runpodctl pod restart POD_ID` | +| Variables in `~/.bashrc` | Pod environment variables. They're set for the container's main process, so export them in your shell profile too if you need them in SSH sessions. | + +Runpod sets `RUNPOD_POD_ID`, `RUNPOD_PUBLIC_IP`, `RUNPOD_TCP_PORT_22`, `RUNPOD_GPU_COUNT`, and other variables in every Pod. See the [environment variable map](#environment-variable-map). + +## Step 5: Move your data and models + +Create the network volume first, in the data center you chose in Step 3. Size it from `bytes_used` in your filesystem inventory. + + + + ```bash + runpodctl network-volume create --name "training-data" --size 500 --data-center-id "US-KS-2" + ``` + + + ```bash + curl --request POST \ + --url https://api.runpod.io/v2/network-volumes \ + --header "Authorization: Bearer $RUNPOD_API_KEY" \ + --header 'Content-Type: application/json' \ + --data '{"name": "training-data", "size": 500, "dataCenter": "US-KS-2"}' + ``` + + + +The command returns the volume ID you'll attach to Pods. Network volumes are available on Secure Cloud. A volume can grow later but can't shrink. + +Then start a transfer Pod with the volume attached. A CPU Pod is enough for copying data, and data transfer in and out of Runpod has no ingress or egress fee. + +```bash +runpodctl pod create \ + --name "lambda-transfer" \ + --compute-type cpu \ + --image "runpod/pytorch:1.0.2-cu1281-torch280-ubuntu2404" \ + --network-volume-id "NETWORK_VOLUME_ID" \ + --data-center-ids "US-KS-2" \ + --ports "22/tcp" + +runpodctl ssh info POD_ID +``` + +`runpodctl ssh info` prints the direct SSH command with the Pod's public IP and external port. You need the direct form, because rsync doesn't work over the proxied SSH connection. If the command reports no public IP, recreate the transfer Pod as a GPU Pod on Secure Cloud. Then connect and install the copy tools on the Pod with `apt update && apt install -y rsync`. + +Choose a transfer path based on where the data is. + + + + This works in every Lambda region. The filesystem must be attached to a running instance, and Lambda only attaches filesystems at launch. If the filesystem isn't mounted anywhere, launch a small instance with it attached. + + Connect to the Lambda instance with agent forwarding so it can authenticate to the Pod with your local key, then push the data. + + ```bash + ssh -A ubuntu@INSTANCE_IP + + # On the Lambda instance + tmux new-session -s migrate + rsync -avP --inplace -e "ssh -p POD_SSH_PORT" \ + /lambda/nfs/FILESYSTEM_NAME/ root@POD_IP:/workspace/ + ``` + + The trailing slash on the source copies the directory's contents into `/workspace`. If the transfer is interrupted, run the same command again and rsync continues where it stopped. Running inside `tmux` keeps the copy alive if your SSH session drops. + + + If your filesystem is in a region with the [Filesystem S3 Adapter](https://docs.lambda.com/public-cloud/s3-adapter-filesystems/) (Lambda lists `us-east-2`, `us-east-3`, and `us-midwest-2`) and was created in March 2025 or later, you can pull the data without a running Lambda instance. Generate an access key pair on the Filesystem S3 Adapter Keys page of the Lambda console and copy the filesystem's bucket name from the Filesystems page. + + On the transfer Pod, install s5cmd from its releases page, then copy the bucket onto the volume. + + ```bash + export AWS_ACCESS_KEY_ID="LAMBDA_S3_ACCESS_KEY" + export AWS_SECRET_ACCESS_KEY="LAMBDA_S3_SECRET_KEY" + export AWS_REGION="us-east-2" + export S3_ENDPOINT_URL="https://files.us-east-2.lambda.com" + + s5cmd ls "s3://LAMBDA_BUCKET_NAME/" + s5cmd cp "s3://LAMBDA_BUCKET_NAME/*" /workspace/ + ``` + + These are the environment variables and commands from Lambda's S3 Adapter docs, run from the Pod so the data lands on the network volume. Lambda's docs note that `aws s3` needs `request_checksum_calculation = WHEN_REQUIRED` and `response_checksum_validation = WHEN_REQUIRED` in the profile to work with the adapter. + + + If the source of truth is an S3-compatible bucket, copy from the bucket onto the volume instead of from Lambda. From the transfer Pod, use the same tools Lambda documents for imports. + + ```bash + # From S3-compatible storage with rclone (configure the remote first with: rclone config) + rclone -P copy REMOTE:BUCKET_NAME /workspace/datasets --transfers 16 + + # Or use Cloud Sync from the Pod's page in the console + ``` + + [Cloud Sync](/pods/storage/cloud-sync) supports Amazon S3, Google Cloud Storage, Azure Blob Storage, Backblaze B2, and Dropbox. Model weights that came from a public hub are often faster to download again than to copy. Set `HF_HOME=/workspace/huggingface` so the cache lands on the volume. + + + For a few files, use `runpodctl send` and `runpodctl receive`. `runpodctl` is preinstalled on Pods. Install it on the Lambda instance with the same install command as in Step 2. + + ```bash + # On the Lambda instance + runpodctl send checkpoints.tar + + # On the Pod, with the code printed by the send command + cd /workspace && runpodctl receive CODE + ``` + + + +Verify the copy before you rely on it. Compare file counts and total size on both sides. + +```bash +# On the Lambda instance +find /lambda/nfs/FILESYSTEM_NAME -type f | wc -l; du -sb /lambda/nfs/FILESYSTEM_NAME + +# On the Pod +find /workspace -type f | wc -l; du -sb /workspace +``` + +Delete hidden `.Trash-*` directories on the Lambda side before you copy if you used JupyterLab there, because they count toward size and aren't needed. When the copy checks out, delete the transfer Pod with `runpodctl pod delete POD_ID`. The data stays on the network volume. + +After the migration, you can read and write the volume without a Pod through the [S3-compatible API](/storage/s3-api), in data centers that support it. + +## Step 6: Deploy on Runpod + +### Deploy a Pod + +This is the direct replacement for launching an instance. The example recreates an 8-GPU H100 instance with a filesystem and two open ports. + + + + ```bash + curl --request POST "https://cloud.lambda.com/api/v1/instance-operations/launch" \ + --header "Authorization: Bearer $LAMBDA_API_KEY" \ + --header 'Content-Type: application/json' \ + --data '{ + "region_name": "us-east-1", + "instance_type_name": "gpu_8x_h100_sxm5gdr", + "ssh_key_names": ["my-key"], + "file_system_names": ["training-data"], + "name": "trainer", + "image": {"family": "lambda-stack-24-04"} + }' + ``` + + + ```bash + runpodctl pod create \ + --name "trainer" \ + --template-id "runpod-torch-v280" \ + --gpu-id "NVIDIA H100 80GB HBM3" \ + --gpu-count 8 \ + --data-center-ids "US-KS-2" \ + --network-volume-id "NETWORK_VOLUME_ID" \ + --container-disk-in-gb 100 \ + --ports "22/tcp,8888/http,8000/http" \ + --min-cuda-version 12.8 \ + --env '{"JUPYTER_PASSWORD":"YOUR_JUPYTER_PASSWORD","HF_HOME":"/workspace/huggingface"}' + ``` + + + ```bash + curl --request POST "https://api.runpod.io/v2/pods" \ + --header "Authorization: Bearer $RUNPOD_API_KEY" \ + --header 'Content-Type: application/json' \ + --data '{ + "name": "trainer", + "templateId": "runpod-torch-v280", + "gpu": {"id": "NVIDIA H100 80GB HBM3", "count": 8, "minCudaVersion": "12.8"}, + "dataCenterIds": ["US-KS-2"], + "mounts": {"network": [{"volumeId": "NETWORK_VOLUME_ID", "path": "/workspace"}]}, + "disk": 100, + "ports": ["22/tcp", "8888/http", "8000/http"], + "startSsh": true, + "startJupyter": true, + "env": {"HF_HOME": "/workspace/huggingface"} + }' + ``` + + + +A few things in the Runpod version have no line in the Lambda request. Ports are part of the Pod, not a separate firewall object. The container disk size is explicit, and it's much smaller than a Lambda root volume, so large files belong on `/workspace`. The data center must be the one that holds the network volume. To use your own image from Step 4, replace the template with `--image YOUR_REGISTRY/lambda-env:v1`. In the console, the same flow is **Pods**, then **Deploy**. + +The create call returns before the container is up. Wait for it, then connect. + +```bash +until [ "$(runpodctl pod get POD_ID --output json | jq -r '.runtimeStatus')" = "running" ]; do + sleep 5 +done + +runpodctl ssh info POD_ID +``` + +JupyterLab is at `https://POD_ID-8888.proxy.runpod.net`, and a service on port 8000 is at `https://POD_ID-8000.proxy.runpod.net`. + +### Deploy a self-hosted inference server + +Lambda's tutorials serve models with vLLM in Docker on an instance. On Runpod, the vLLM image is the Pod's image, and the arguments after the image name become the start command. + + + + ```bash + # On the instance, after SSH + sudo docker run --gpus all --ipc=host \ + -v "$HOME/.cache/huggingface":/root/.cache/huggingface \ + -p 8000:8000 \ + --env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" \ + vllm/vllm-openai --model meta-llama/Meta-Llama-3.1-8B-Instruct + ``` + + + ```bash + runpodctl pod create \ + --name "llama-server" \ + --image "vllm/vllm-openai:latest" \ + --gpu-id "NVIDIA A100 80GB PCIe" \ + --volume-in-gb 100 \ + --ports "8000/http" \ + --env '{"HUGGING_FACE_HUB_TOKEN":"YOUR_HF_TOKEN","HF_HOME":"/workspace/huggingface"}' \ + --docker-args "--model meta-llama/Meta-Llama-3.1-8B-Instruct --api-key YOUR_SERVER_KEY" + ``` + + + +On Lambda, the firewall kept port 8000 closed until you opened it. On Runpod, the proxy URL is public as soon as the Pod runs, so the example starts vLLM with `--api-key`. Pin the image to a version tag for production, and store the Hugging Face token as a [secret](/pods/templates/secrets). Requests through the proxy must respond within 100 seconds, so use streaming for long generations or expose the port as `8000/tcp`. + +If the server sits idle for part of the day, deploy the vLLM worker on Serverless instead. It scales to zero and bills per second while workers run. + +```bash +runpodctl hub search vllm +runpodctl serverless create --hub-id HUB_ID --name "llama-server" \ + --env MODEL_NAME=meta-llama/Meta-Llama-3.1-8B-Instruct +``` + +The endpoint serves an OpenAI-compatible API at `https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1`. See [Deploy vLLM on Serverless](/serverless/vllm/get-started) for model settings and gated models. + +### Deploy an Instant Cluster + +For a 1-Click Cluster workload, create an Instant Cluster in the same data center as your network volume. In the console, open **Instant Clusters**, click **Create Cluster**, choose the cluster type (for example **Slurm Cluster**), the GPU type, the Pod count, and the network volume. With REST v2: + +```bash +curl --request POST "https://api.runpod.io/v2/clusters" \ + --header "Authorization: Bearer $RUNPOD_API_KEY" \ + --header 'Content-Type: application/json' \ + --data '{ + "name": "distributed-training", + "type": "SLURM", + "compute": {"gpuTypeId": "NVIDIA H100 80GB HBM3", "gpuCountPerPod": 8, "podCount": 4}, + "templateId": "9yo900pjtt", + "dataCenterIds": ["US-KS-2"], + "mounts": {"network": [{"volumeId": "NETWORK_VOLUME_ID", "path": "/workspace"}]}, + "startSsh": true + }' +``` + +`type` is `SLURM`, `RAY`, `TRAINING`, or `APPLICATION`. The template ID in the example is the official PyTorch 2.8.0 template for clusters. Confirm the current ID with `runpodctl template search pytorch --type official`. Slurm Clusters need an official Runpod PyTorch image. + +What changes from a 1-Click Cluster: + +- **No separate head nodes.** The primary node (`NODE_RANK=0`) is one of the GPU nodes. On a Slurm Cluster it's the controller, and you submit jobs there with `sbatch`. +- **Shared storage is the network volume** at `/workspace` on every node, in place of the shared `/home` and `/data` on Lambda's Slurm clusters. +- **Inter-node traffic uses `ens1` to `ens8`.** Set `NCCL_SOCKET_IFNAME=ens1`. Runpod sets `PRIMARY_ADDR`, `NODE_ADDR`, `NUM_NODES`, and `WORLD_SIZE` on every node. See the [configuration reference](/instant-clusters/configuration). +- **No reservation window.** The cluster bills while it exists, so delete it when the job finishes. Whether you can create it depends on current stock of that GPU in the data center. + +## Step 7: Update the code that calls Lambda + +### Clients of your services + +Every client that reaches a service by instance IP needs the new address. + +| On Lambda | On Runpod | +| --- | --- | +| `http://INSTANCE_IP:8000` | `https://POD_ID-8000.proxy.runpod.net` | +| `tcp://INSTANCE_IP:5432` | `POD_IP:EXTERNAL_PORT` from the Pod's **Connect** menu | +| `ssh ubuntu@INSTANCE_IP` | The command from `runpodctl ssh info POD_ID` | +| `ssh -L 8000:127.0.0.1:8000 ubuntu@INSTANCE_IP` | `ssh -L 8000:127.0.0.1:8000 root@POD_IP -p EXTERNAL_PORT` | +| Private IP between instances | `POD_ID.runpod.internal` with global networking | +| JupyterLab URL from the console | `https://POD_ID-8888.proxy.runpod.net` | + +For an OpenAI-compatible server, the change is the base URL and key. + +```python client.py +from openai import OpenAI + +# Before: vLLM on a Lambda instance +# client = OpenAI(base_url="http://INSTANCE_IP:8000/v1", api_key="YOUR_SERVER_KEY") + +# After: the same vLLM server on a Runpod Pod +client = OpenAI( + base_url="https://POD_ID-8000.proxy.runpod.net/v1", + api_key="YOUR_SERVER_KEY", +) + +# After, if you moved to Serverless vLLM instead: +# client = OpenAI( +# base_url="https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1", +# api_key="YOUR_RUNPOD_API_KEY", +# ) + +response = client.chat.completions.create( + model="meta-llama/Meta-Llama-3.1-8B-Instruct", + messages=[{"role": "user", "content": "Say hello."}], +) +print(response.choices[0].message.content) +``` + +The Pod's public IP and external ports can change when a Pod is reset or recreated, and the proxy URL changes when the Pod ID changes. Put production traffic behind your own DNS name or configuration value instead of hardcoding either one. A Serverless endpoint URL stays the same across worker restarts. + +### Automation that calls the Lambda Cloud API + +Scripts that launch, poll, and terminate instances translate one call at a time. This is a typical launch-and-wait script before and after. + + + + ```python launch.py + import os + import time + + import requests + + API = "https://cloud.lambda.com/api/v1" + HEADERS = {"Authorization": f"Bearer {os.environ['LAMBDA_API_KEY']}"} + + launch = requests.post( + f"{API}/instance-operations/launch", + headers=HEADERS, + json={ + "region_name": "us-east-1", + "instance_type_name": "gpu_1x_a10", + "ssh_key_names": ["my-key"], + "file_system_names": ["training-data"], + "name": "worker", + }, + ) + launch.raise_for_status() + instance_id = launch.json()["data"]["instance_ids"][0] + + while True: + instance = requests.get(f"{API}/instances/{instance_id}", headers=HEADERS).json()["data"] + if instance["status"] == "active": + break + time.sleep(15) + + print(f"ssh ubuntu@{instance['ip']}") + + # Later + requests.post( + f"{API}/instance-operations/terminate", + headers=HEADERS, + json={"instance_ids": [instance_id]}, + ).raise_for_status() + ``` + + + ```python launch.py + import os + import time + + import requests + + API = "https://api.runpod.io/v2" + HEADERS = {"Authorization": f"Bearer {os.environ['RUNPOD_API_KEY']}"} + + create = requests.post( + f"{API}/pods", + headers=HEADERS, + json={ + "name": "worker", + "templateId": "runpod-torch-v280", + "gpu": {"id": "NVIDIA RTX A5000", "count": 1}, + "dataCenterIds": ["US-KS-2"], + "mounts": {"network": [{"volumeId": "NETWORK_VOLUME_ID", "path": "/workspace"}]}, + "ports": ["22/tcp"], + "startSsh": True, + }, + ) + create.raise_for_status() + pod_id = create.json()["id"] + + while True: + pod = requests.get(f"{API}/pods/{pod_id}", headers=HEADERS).json() + if pod["status"] == "RUNNING" and pod["ssh"]["direct"]: + break + if pod["status"] == "ERROR": + raise RuntimeError(f"Pod {pod_id} failed to start") + time.sleep(10) + + print(pod["ssh"]["direct"]["command"]) + + # Later + requests.delete(f"{API}/pods/{pod_id}", headers=HEADERS).raise_for_status() + ``` + + + +The differences that matter in code: + +- **Response envelope**: Lambda wraps results in `data` and errors in `error.code`. REST v2 returns the resource at the top level. +- **Batching**: Lambda's terminate and restart calls take a list of instance IDs. REST v2 acts on one Pod per request. +- **Capacity errors**: both APIs reject a create when the GPU type has no stock. REST v2 places exactly the GPU type you ask for and doesn't fall back to another type, so keep your retry or fallback logic. +- **SSH readiness**: `RUNNING` means the container is up. The script also waits for `ssh.direct`, which appears once port 22 has a public mapping. +- **Rate limits**: Lambda allows one request per second and one launch per 12 seconds. REST v2 reports its limits in `RateLimit-Policy` response headers. + +The full route mapping is in the [API map](#api-map). + +## Step 8: Verify, cut over, and shut down + +Test from outside Runpod, the way your users or jobs will reach the workload. + +```bash +# HTTP service through the proxy +curl -s https://POD_ID-8000.proxy.runpod.net/v1/models -H "Authorization: Bearer YOUR_SERVER_KEY" + +# GPU, CUDA, and the volume, without opening an SSH session +curl -s --request POST "https://api.runpod.io/v2/pods/POD_ID/exec" \ + --header "Authorization: Bearer $RUNPOD_API_KEY" \ + --header 'Content-Type: application/json' \ + --data '{"command": ["sh", "-c", "nvidia-smi --query-gpu=name,memory.total --format=csv && python -c \"import torch; print(torch.__version__, torch.cuda.is_available())\" && df -h /workspace"]}' +``` + +Then work through the cutover: + +1. **Compare results.** Run the same training step, evaluation, or inference request on both platforms and compare outputs and throughput. Differences usually trace back to a package version in the image. +2. **Run both in parallel.** Keep the Lambda instance up while the Pod takes a share of jobs or traffic. +3. **Do a final data sync.** Run the rsync from Step 5 again to pick up files written since the first copy. +4. **Shift traffic.** Point DNS, client configuration, and schedulers at Runpod. +5. **Set cost guards on Runpod.** Enable low balance notifications or auto-pay in [Billing](/accounts-billing/billing), stop or delete Pods that finish their work, and review spend with `runpodctl billing pods`. Accounts have a default spend limit of \$80 per hour across all resources. Contact support if your workloads need more. + +When Runpod has carried the workload for long enough that you trust it, stop billing on Lambda. Lambda bills instances until you terminate them and filesystems for as long as they exist, mounted or not. + + +Terminating a Lambda instance destroys its root volume, and deleting a filesystem destroys its data. Lambda can't recover either one. Confirm your copy on Runpod first. + + +```bash +# 1. Terminate instances (shutting down from inside the VM doesn't stop billing) +curl --request POST "$LAMBDA_API/instance-operations/terminate" \ + --header "Authorization: Bearer $LAMBDA_API_KEY" \ + --header 'Content-Type: application/json' \ + --data '{"instance_ids": ["INSTANCE_ID_1", "INSTANCE_ID_2"]}' + +# 2. Delete filesystems once no instance has them mounted +curl --request DELETE "$LAMBDA_API/filesystems/FILESYSTEM_ID" \ + --header "Authorization: Bearer $LAMBDA_API_KEY" + +# 3. Confirm nothing is left +curl -s -H "Authorization: Bearer $LAMBDA_API_KEY" "$LAMBDA_API/instances" | jq '.data | length' +curl -s -H "Authorization: Bearer $LAMBDA_API_KEY" "$LAMBDA_API/filesystems" | jq '.data | length' +``` + +Lambda's docs warn that `sudo shutdown` or `sudo systemctl poweroff` inside an instance doesn't end billing, so always terminate through the API or console. 1-Click Cluster reservations run for their reserved term and can't be extended, so time the cluster cutover to the end of the reservation. Afterward, delete SSH keys, firewall rulesets, and API keys you no longer need in the Lambda console, and check the **Usage** page for your last charges. + +## Migrate your Terraform + +Lambda's docs don't cover a Terraform provider, and the Lambda providers on the Terraform Registry are community projects. If you use one, or you want to put the migrated resources under Terraform for the first time, the Runpod provider is built on REST API v2 and covers the same ground. + +| What you manage on Lambda | Runpod resource | +| --- | --- | +| Instance | `runpod_pod` | +| Filesystem | `runpod_network_volume` | +| SSH key | `runpod_account_ssh_keys` (one resource that owns every key on the account) | +| Firewall rule or ruleset | `ports` on `runpod_pod` | +| 1-Click Cluster | `runpod_cluster` | +| Image and startup configuration | `runpod_template` | + +This configuration creates the SSH keys, a network volume, and a Pod that mounts it. + +```hcl main.tf +terraform { + required_version = ">= 1.6" + required_providers { + runpod = { + source = "runpod/runpod" + version = "~> 2.0" + } + } +} + +# Reads RUNPOD_API_KEY from the environment +provider "runpod" {} + +resource "runpod_account_ssh_keys" "this" { + keys = [trimspace(file("~/.ssh/id_ed25519.pub"))] +} + +resource "runpod_network_volume" "training_data" { + name = "training-data" + size = 500 + data_center = "US-KS-2" +} + +resource "runpod_pod" "trainer" { + name = "trainer" + image = "runpod/pytorch:1.0.2-cu1281-torch280-ubuntu2404" + gpu_type_id = "NVIDIA H100 80GB HBM3" + gpu_count = 8 + cloud = "SECURE" + disk = 100 + min_cuda_version = "12.8" + data_center_ids = [runpod_network_volume.training_data.data_center] + + network_volumes = [{ + volume_id = runpod_network_volume.training_data.id + path = "/workspace" + }] + + ports = ["22/tcp", "8888/http"] + start_ssh = true + + depends_on = [runpod_account_ssh_keys.this] +} + +output "pod_id" { + value = runpod_pod.trainer.id +} +``` + +`runpod_account_ssh_keys` replaces the account's whole key set, so list every key your team needs. Changing `gpu_type_id`, `gpu_count`, or `data_center_ids` replaces the Pod, which is why data belongs on the network volume. Set `desired_state = "EXITED"` to stop a Pod while keeping it in your configuration. See [Manage Runpod infrastructure with Terraform](/integrations/terraform/overview) and the [provider reference](https://registry.terraform.io/providers/runpod/runpod/latest/docs) for every argument. + +## Reference + +### API map + +Lambda routes are relative to `https://cloud.lambda.com/api/v1`. Runpod routes are relative to `https://api.runpod.io`. + +| Lambda Cloud API | Runpod REST v2 | `runpodctl` | +| --- | --- | --- | +| `GET /instances` | `GET /v2/pods` | `runpodctl pod list` | +| `GET /instances/{id}` | `GET /v2/pods/{id}` | `runpodctl pod get POD_ID` | +| `POST /instances/{id}` (rename, tags) | `PATCH /v2/pods/{id}` | `runpodctl pod update POD_ID` | +| `POST /instance-operations/launch` | `POST /v2/pods` | `runpodctl pod create` | +| `POST /instance-operations/restart` | `POST /v2/pods/{id}/action` with `{"action": "restart"}` | `runpodctl pod restart POD_ID` | +| `POST /instance-operations/terminate` | `DELETE /v2/pods/{id}` | `runpodctl pod delete POD_ID` | +| No stop operation | `POST /v2/pods/{id}/action` with `{"action": "stop"}` or `{"action": "start"}` | `runpodctl pod stop`, `runpodctl pod start` | +| `GET /instance-types` | `GET /v2/catalog/gpus?include=AVAILABILITY&product=POD` | `runpodctl gpu list` | +| `GET /regions` | `GET /v2/catalog/datacenters` | `runpodctl datacenter list` | +| `GET /images` | `GET /v2/catalog/templates`, `GET /v2/templates` | `runpodctl template list` | +| `GET /ssh-keys` | `GET /v2/account/ssh-keys` | `runpodctl ssh list-keys` | +| `POST /ssh-keys`, `DELETE /ssh-keys/{id}` | `PUT /v2/account/ssh-keys` (replaces the full set) | `runpodctl ssh add-key`, `runpodctl ssh remove-key` | +| `GET /filesystems` | `GET /v2/network-volumes` | `runpodctl network-volume list` | +| `POST /filesystems` | `POST /v2/network-volumes` | `runpodctl network-volume create` | +| `DELETE /filesystems/{id}` | `DELETE /v2/network-volumes/{id}` | `runpodctl network-volume delete VOLUME_ID` | +| `GET` and `PUT /firewall-rules`, `/firewall-rulesets` | `ports` on `POST /v2/pods` and `PATCH /v2/pods/{id}` | `--ports` on `runpodctl pod create` and `pod update` | +| `GET /audit-events` | Audit logs in REST v2 | | +| No equivalent | `POST /v2/pods/{id}/exec` (run a command without SSH) | | + +Lambda can generate an SSH key pair for you through `POST /ssh-keys`. On Runpod, generate the pair locally with `ssh-keygen` and register the public key. + +### Configuration field map + +| Lambda launch field | REST v2 field | `runpodctl pod create` flag | +| --- | --- | --- | +| `name` | `name` | `--name` | +| `region_name` | `dataCenterIds` | `--data-center-ids` | +| `instance_type_name` | `gpu.id` and `gpu.count` | `--gpu-id` and `--gpu-count` | +| `image` (`family` or `id`) | `image` or `templateId` | `--image` or `--template-id` | +| `ssh_key_names` | `startSsh` (uses all account keys) | `--ssh` (on by default) | +| `file_system_names`, `file_system_mounts` | `mounts.network` (`volumeId`, `path`) | `--network-volume-id` | +| `firewall_rulesets` | `ports` | `--ports` | +| `user_data` (cloud-init) | `args`, `entrypoint`, `cmd`, `env` | `--docker-args`, `--env` | +| `hostname` | No field. Use `RUNPOD_POD_HOSTNAME` or the Pod name. | | +| `tags` | Not set at Pod create | | +| `workspace_id` | The account or team that owns the API key | | +| Root volume size (fixed per instance type) | `disk`, `mounts.persistent.size` | `--container-disk-in-gb`, `--volume-in-gb` | + +### Environment variable map + +Lambda instances don't document platform-set environment variables, so this table maps where values come from. + +| Value | On a Lambda instance | On a Runpod Pod | +| --- | --- | --- | +| Instance or Pod ID | From the API or console | `RUNPOD_POD_ID` | +| Region or data center | From the API or console | `RUNPOD_DC_ID` | +| Public IP | Instance `ip` field | `RUNPOD_PUBLIC_IP` | +| SSH port | 22 | `RUNPOD_TCP_PORT_22` | +| GPU count | `nvidia-smi` | `RUNPOD_GPU_COUNT` | +| Attached storage | `/lambda/nfs/NAME` | `RUNPOD_VOLUME_ID`, mounted at `/workspace` | +| Authorized SSH keys | `~/.ssh/authorized_keys` | `PUBLIC_KEY` | +| JupyterLab token | `jupyter_token` field in the API | `JUPYTER_PASSWORD` | +| Cluster rendezvous | Node hostnames from the console | `PRIMARY_ADDR`, `NODE_RANK`, `NUM_NODES`, `WORLD_SIZE` | + +### Status map + +| Lambda instance status | Runpod Pod status | Notes | +| --- | --- | --- | +| `booting` | `PROVISIONING`, `STARTING` | `runpodctl` reports both as `runtimeStatus: initializing`. | +| `active` | `RUNNING` | | +| `unhealthy` | `ERROR` | A Pod in `ERROR` still exists and can still incur charges. | +| `terminating` | `TERMINATED` | | +| `terminated` | `TERMINATED`, then the Pod no longer appears | Treat "not found" as the signal that a Pod is gone. | +| `preempted` | No equivalent for on-demand Pods | On-demand Pods aren't displaced by other users. | +| No stopped state | `EXITED` | Stopped. Volume disk kept and billed. | + +## Migrate with a coding agent + +A coding agent with the Runpod skills and MCP server can run the inventory, write the Dockerfile, create the Runpod resources, and rewrite your API calls. Install them first. + + + + ```bash + # The plugin also connects the hosted MCP server + claude plugin marketplace add runpod/runpod-plugins-official + claude plugin install runpod@runpod + ``` + + + ```bash + # Skills, then the hosted MCP server (Codex, Cursor, and others) + npx skills add runpod/runpod-plugins-official --skill '*' --agent AGENT_ID --global + npx @runpod/mcp-server@latest add --client CLIENT_ID + ``` + + + +The MCP server signs in with OAuth ("Sign in with Runpod"), so you don't need an API key to start. See [Agent skills](/get-started/agent-skills) and [MCP servers](/get-started/mcp-servers) for supported agents and clients. + +Then give your agent this prompt from the root of the repository that holds your Lambda automation. + +```text +Migrate this project from Lambda (the GPU cloud at lambda.ai, not AWS Lambda) to Runpod. +Use the runpod-cloud-migrate skill and its Lambda reference. + +1. Inventory first, read-only. Use the Lambda Cloud API with LAMBDA_API_KEY from my + environment to list instances, filesystems, SSH keys, firewall rules, and firewall + rulesets. Search this repo for cloud.lambda.com, cloud.lambda.ai, cloud.lambdalabs.com, + LAMBDA_API_KEY, instance-operations, and /lambda/nfs. For each instance I approve, SSH in + and capture pip freeze, manually installed apt packages, docker ps, running systemd + services, cron jobs, and listening ports. +2. Show me a plan before creating anything: the Runpod target for each workload (Pod, + Serverless endpoint, or Instant Cluster), the GPU type ID and count, the data center, + network volume sizes, exposed ports, anything that has no direct equivalent (Docker + Compose, UDP or source-IP firewall rules, ARM images, multiple filesystems), and an + hourly and monthly cost estimate from live Runpod pricing. Check live GPU stock. +3. Wait for my approval before you create anything billable on Runpod. +4. Migrate one workload end to end: build or choose the image, create the network volume, + copy the data, create the Pod, and update the code that calls Lambda. +5. Verify with a real request from outside the Pod and a GPU check inside it, and show me + the output. +6. Never terminate Lambda instances, delete Lambda filesystems, or remove Lambda keys + without my explicit approval. Don't print secrets or API keys. +``` + +## Troubleshooting + +**`docker: command not found` or the Docker daemon won't start in a Pod.** +A Pod is already a container, and Runpod runs Docker for you. Deploy the image as the Pod's image instead of running it inside another Pod, and split Docker Compose services into separate Pods. + +**Files outside `/workspace` are gone after a stop, restart, or edit.** +The container disk is ephemeral. On Lambda the whole root volume lasted until terminate. Move data, model caches (`HF_HOME`), and virtual environments to `/workspace`, and put system packages in the image. + +**`sudo: command not found` or scripts fail on `/home/ubuntu`.** +Pods run as `root`, and official images don't need `sudo`. Remove `sudo` from scripts and replace `/home/ubuntu` with `/root` or `/workspace`. + +**`systemctl` fails with a message about `systemd` not being the init system.** +The container's main process is your image's start command. Start services from a start script, as shown in [Create a custom template](/pods/templates/create-custom-template#step-4-configure-container-startup-behavior). + +**SSH asks for a password, or `scp` and `rsync` fail.** +A password prompt means the key isn't registered: run `runpodctl ssh list-keys` and check that the public key is in OpenSSH format. If the shell works but `scp` and `rsync` fail, you're on the proxied connection. Expose `22/tcp` and use the direct command from `runpodctl ssh info POD_ID`. + +**The service works on `localhost` but the proxy URL returns 502.** +The service is bound to `127.0.0.1`, isn't listening yet, or the port isn't in the Pod's port list. Bind to `0.0.0.0`, check the port with `ss -tlnp`, and confirm the port is exposed as `PORT/http`. JupyterLab on official templates only starts when `JUPYTER_PASSWORD` is set. + +**Requests through the proxy fail with 524 after 100 seconds.** +The HTTP proxy closes connections that take longer than 100 seconds to respond. Stream the response, return a job ID and poll, or expose the port as TCP. + +**The container fails to start with a CUDA or driver error, or PyTorch reports no GPU.** +The host's CUDA version is older than your image needs. On Lambda the driver was part of your VM. On Runpod it belongs to the host, so recreate the Pod with `--min-cuda-version` set to the version your image was built for. + +**`pip install` fails with `externally-managed-environment`.** +The Ubuntu 24.04 PyTorch image protects the system Python. Use `pip install --break-system-packages`, or install into a virtual environment created with `--system-site-packages` so it can see the preinstalled PyTorch. + +**The Pod can't be created with the network volume attached.** +A Pod that mounts a network volume must run in the volume's data center, and that data center needs stock of your GPU type. Check with `runpodctl datacenter list`, choose another GPU type that fits, or create the volume in a data center that has the GPU you need. + +## Next steps + + + + Match GPU type, memory, and storage to your workload. + + + Share persistent storage across Pods, clusters, and endpoints. + + + Package your environment as a reusable image and template. + + + Move idle inference servers to endpoints that scale to zero. + + diff --git a/migrate/modal.mdx b/migrate/modal.mdx new file mode 100644 index 000000000..f3acda3da --- /dev/null +++ b/migrate/modal.mdx @@ -0,0 +1,1219 @@ +--- +title: "Migrate from Modal to Runpod" +sidebarTitle: "Modal" +description: "Move Modal Functions, Clses, web endpoints, Volumes, and Secrets to the Runpod SDK and Serverless, with before and after code." +--- + +This guide is for teams that run production workloads on [Modal](https://modal.com/docs/guide/intro) and want to run them on Runpod. Modal [Functions](https://modal.com/docs/guide/functions) and [Clses](https://modal.com/docs/guide/lifecycle-functions) map to queue resources in the [Runpod SDK](/sdk/overview), [Web Functions](https://modal.com/docs/guide/webhooks) and [Servers](https://modal.com/docs/guide/servers) map to API resources and load-balancing endpoints, long jobs map to tasks, [Volumes](https://modal.com/docs/guide/volumes) map to network volumes, and [Secrets](https://modal.com/docs/guide/secrets) map to Runpod secrets. When you finish, each workload runs on Runpod, your callers use Runpod URLs or the Runpod client, and your Modal Apps are stopped. + +Most of the work is a code translation, because both platforms define infrastructure with Python decorators. Apps that depend on a long `modal.Image` build chain have a second path: a Docker image with a handler function or an HTTP server. + + + You can hand this migration to a coding agent. See [Migrate with a coding agent](#migrate-with-a-coding-agent). + + +## How Modal maps to Runpod + +| Modal concept | Runpod equivalent | Notes | +| --- | --- | --- | +| `modal.App` | `runpod.App` | Both group resources that deploy together. | +| `@app.function` | `@app.queue` (request and response work) or `@app.task` (long jobs) | A queue resource is an autoscaling [queue-based Serverless endpoint](/serverless/overview). A task is one Pod per call. | +| `@app.cls` with `@modal.enter` and `@modal.method` | `@app.queue` with a module-level cache, or `@app.api` with `@runpod.init` | A queue resource wraps a function, not a class. An API resource wraps a class and serves routes. | +| `@modal.fastapi_endpoint`, `@modal.asgi_app`, `@modal.wsgi_app`, `@modal.web_server` | `@app.api`, or a [load-balancing endpoint](/serverless/load-balancing/overview) running your own HTTP server | `@app.api` routes are `@runpod.get` and `@runpod.post` methods. To keep a full FastAPI or Flask app, use a load-balancing endpoint. | +| `@app.server` (Modal Servers) | Load-balancing endpoint | Both route HTTP straight to a process listening on a port, with no queue. | +| Modal Endpoints (Shared and Dedicated LLM endpoints) | [Public Endpoints](/public-endpoints/overview), or a [vLLM worker](/serverless/vllm/overview) on your own endpoint | Hosted models versus your own autoscaling endpoint. | +| `modal.Image` builder chain | `dependencies=`, `system_dependencies=`, `image=`, or a Dockerfile | The SDK installs pip and apt packages on Runpod's runtime image. There's no chained image builder. | +| `modal.Volume` | [Network volume](/storage/network-volumes) (`runpod.NetworkVolume`) or [global volume](/storage/globalvolume/overview) (`runpod.GlobalVolume`, beta) | Network volumes live in one data center. Global volumes are object-backed and GPU only. | +| `modal.CloudBucketMount` | Read the bucket from your code, or copy it to a network volume | See [Step 5](#step-5-move-your-data-and-models). | +| `modal.Secret` | Runpod secret (`runpod.Secret`) | A Modal Secret holds several key and value pairs. A Runpod secret holds one value, and you choose the environment variable name where you use it. | +| `modal.Dict`, `modal.Queue` | An external store such as Redis or a database | See [What has no direct equivalent](#what-has-no-direct-equivalent). | +| `modal.Cron`, `modal.Period` | An external scheduler that calls the endpoint | See [What has no direct equivalent](#what-has-no-direct-equivalent). | +| `modal.Sandbox` | A [Pod](/pods/overview) you create and delete through the API | The closest match in the sources for this guide. Check the differences before you move sandbox workloads. | +| `@modal.clustered` (multi-node) | [Instant Clusters](/instant-clusters/pytorch) | Multi-node training and serving. | +| GPU strings (`gpu="H100"`, `"L4"`, `"H100:8"`) | `gpu="H100"`, GPU pools (`runpod.GpuGroup`), exact devices (`runpod.GpuType`), `gpu_count=8` | See the [GPU mapping table](#gpu-mapping). | +| `cpu=`, `memory=` | `cpu="cpu3c-2-4"` instance IDs for CPU resources | CPU resources use fixed vCPU and memory shapes. `gpu` and `cpu` are mutually exclusive. | +| `min_containers`, `max_containers`, `scaledown_window` | `workers=(min, max)`, `idle_timeout` | Old Modal names: `keep_warm`, `concurrency_limit`, `container_idle_timeout`. | +| `region=`, `cloud=` | `datacenter=` | Runpod places workloads by data center ID, such as `US-KS-2`. | +| Workspace | Runpod account or [team](/accounts-billing/manage-accounts) | | +| Environment (`modal deploy --env`) | App environment (`rp flash deploy --env`) | Modal Environments also scope Secrets and Volumes. Runpod secrets and volumes belong to the account. | +| Token ID and token secret (`MODAL_TOKEN_ID`, `MODAL_TOKEN_SECRET`) | One API key (`RUNPOD_API_KEY`) | `rp login` stores it in `~/.runpod/config.toml`. | +| Proxy Tokens (`Modal-Key`, `Modal-Secret` headers) | `Authorization: Bearer` with your Runpod API key | Modal Web Functions are public unless you set `requires_proxy_auth=True`. Requests to Runpod endpoints carry an API key. | +| `modal` CLI | `rp` (installed with the SDK) and [`runpodctl`](/runpodctl/overview) | `rp flash` handles apps. `runpodctl` manages endpoints, Pods, templates, and volumes. | +| Python, JavaScript, and Go SDKs | [Python, JavaScript, and Go SDKs](/serverless/sdks), plus [REST API v2](/api-reference-v2/overview) | The decorator-based app SDK is Python. | +| Billing: CPU, memory, and GPU metered separately per second, invoiced monthly | Billing: each worker or Pod billed per second, drawn from a prepaid balance | See [Serverless pricing](/serverless/pricing) and [Billing](/accounts-billing/billing). Modal's Starter plan includes monthly credits. | +| `modal app logs`, dashboard | Dev session log streaming, `rp logs POD_ID`, endpoint logs and metrics in the console | See [Logs](/serverless/development/logs). | + +## What's different + +### What works differently + +**Arguments and results are JSON on queue resources.** Modal serializes arguments and return values with cloudpickle, so you can pass most Python objects. Runpod queue jobs are JSON. Pass strings, numbers, lists, and dicts, send binary data as base64 or as a URL, and write large outputs to a volume or object storage. Whole-number floats can arrive as integers, so cast numeric inputs. + +**Payload and result limits are smaller.** Modal Web Functions accept request bodies up to 4 GiB, and results from `.spawn()` are kept for 7 days. Runpod queue endpoints accept 10 MB on `/run` and 20 MB on `/runsync`, load-balancing endpoints accept 30 MB per request and response, and job results are kept for 30 minutes after an async job and 1 minute after a sync job. If a caller fetches results hours later, send them to a webhook or write them to storage. + +**Images are assembled differently.** A `modal.Image` is built by chaining methods and cached by Modal. In the Runpod SDK, you list pip packages in `dependencies` and apt packages in `system_dependencies`, and `rp flash deploy` packages your code and dependencies into an artifact of up to 1,500 MB that runs on Runpod's runtime image. For anything beyond that, set `image=` to your own image or build a Docker worker. + +**Classes become functions or route classes.** `@app.queue` wraps a function. To load a model once per worker, cache it in a module-level variable, or move to `@app.api`, where a method marked `@runpod.init` runs once when each worker starts. The SDK docs show a startup hook only, so move `@modal.exit` cleanup into your own shutdown handling. + +**Web endpoints are authenticated and have a different URL shape.** A Modal Web Function gets a `modal.run` URL and is public by default. A Runpod queue endpoint is called at `https://api.runpod.ai/v2/ENDPOINT_ID/runsync` with a JSON body under `input`, and a load-balancing endpoint at `https://ENDPOINT_ID.api.runpod.ai/PATH`. Both expect `Authorization: Bearer` with a Runpod API key. If browsers or third parties call your Modal URL without credentials, put a proxy that holds the key in front of the endpoint. + +**Timeout units change.** Modal's `timeout` is in seconds and defaults to 300. The SDK's `execution_timeout_ms` is in milliseconds and defaults to 0, which means no limit. `timeout=600` becomes `execution_timeout_ms=600000`. + +**Volumes don't need commits.** A Modal Volume is snapshotted with `commit()` and refreshed with `reload()`. A network volume behaves like a mounted file system: writes are visible to other workers right away, so delete the `commit()` and `reload()` calls. Queue and API resources mount one volume, always at `/runpod-volume`. Tasks can mount one network volume and one global volume at paths you choose. + +**A network volume pins compute to its data center.** Every resource that mounts a network volume runs in that volume's data center, which narrows the GPUs available to it. GPU stock varies by data center, so check live stock before you choose where a volume lives. + +**Idle endpoints are scaled down.** Runpod reduces max workers to 2 on an endpoint that receives no requests for 3 days, and to 0 after 7 days. Raise max workers again before you send traffic to an endpoint that sat unused. + +**Requests to an empty load-balancing endpoint can fail.** Queue endpoints hold jobs until a worker is ready, like Modal Functions. Load-balancing endpoints have no queue, like Modal Servers, so clients should retry while the first worker starts. + +**Billing is per worker, from a prepaid balance.** Modal meters CPU cores, memory, and GPUs as separate line items and invoices monthly. Runpod bills each Serverless worker per second from start to stop, including cold start, execution, and the idle timeout, plus storage, and deducts charges from a balance you fund in advance. Size workers and `idle_timeout` with that in mind. + +### What has no direct equivalent + +| Modal feature | On Runpod | +| --- | --- | +| `modal.Cron` and `modal.Period` schedules | Call the endpoint's `/run` URL from an external scheduler, such as a GitHub Actions schedule, cron on a host you run, or your workflow orchestrator. For a task, run `rp flash dev main.py --once` on the schedule. | +| `modal.Dict` and `modal.Queue` | Use an external store such as Redis or a database, with the connection string in a Runpod secret. If a Queue only feeds work to a Function, the queue-based endpoint replaces it. | +| `.map()`, `.starmap()`, `.for_each()`, `.spawn_map()` | Submit jobs in a loop with `.spawn()` and collect them with `job.result()`, or use `asyncio.gather` over `fn.remote.aio(...)`. For large offline runs, see [Batch jobs](/serverless/batch-jobs). | +| `@modal.batched` (dynamic batching) | Accept a list in one job and batch on the caller, or raise `max_concurrency` and batch inside the worker. | +| `retries=` and `modal.Retries` | Retry in the caller, or call `job.retry()` (`/retry/JOB_ID`) on a failed job. Tasks are plain Pods, so make training resumable from checkpoints. | +| `buffer_containers` | Raise the minimum in `workers=(min, max)`, or tune `scaler_type` and `scaler_value`. | +| `enable_memory_snapshot` | FlashBoot is on by default for queue and API resources. It's a different mechanism, so measure cold starts instead of assuming parity. | +| `modal.parameter` (parametrized Clses) and `with_options` | Pass the value as input, or deploy one resource per configuration. | +| Custom domains on Web Functions | Point your domain at a proxy or CDN you control that forwards to the endpoint URL. | +| `modal.Proxy` (static outbound IPs) | This guide has no mapping for it. If a database or API allowlists your Modal Proxy IP, plan another egress path and test it before cutover. | +| `modal.forward` tunnels | Run the process on a [Pod and expose its port](/pods/configuration/expose-ports). | +| `modal.Sandbox` (exec, file, and snapshot APIs) | Create a Pod, run commands in it, and delete it. Startup time and isolation differ, so benchmark before you move agent or code-execution workloads. | +| Separate `cpu=` and `memory=` requests on a GPU Function | The SDK's resource parameters have no separate core or memory request for GPU resources. Choose the GPU that fits the workload, and `gpu_count` for more. | + +Runpod has features without a Modal counterpart that affect how you design the port: `runpod.Model` stages Hugging Face weights on the host before a worker starts, `@app.task` gives a job a dedicated Pod with no execution limit, and network volumes expose an [S3-compatible API](/storage/s3-api) so you can load data without running compute. + +## Requirements + +- A Modal workspace you can read with the `modal` CLI (`modal token info` shows the active token). +- A [Runpod account](/accounts-billing/manage-accounts) with a funded balance. +- Python 3.10 to 3.14 on your machine. Dev sessions and tasks run your code on the same Python version as your local interpreter. +- Docker, only if you use the custom image path. +- The AWS CLI, only if you copy data with the S3-compatible API. + +## Step 1: Take inventory of what you run on Modal + +List every App, in every Environment, with its Functions, Servers, and deployment history: + +```bash +modal profile current +modal environment list +modal app list --json +modal app info APP_NAME +modal app history APP_NAME +modal container list --json +``` + +Repeat the App commands with `--env ENVIRONMENT_NAME` for each Environment. `modal app info` lists each Function and Server in the App, and `modal container list` shows what's running now. + +Next, list the storage and configuration objects that Apps reference by name: + +```bash +modal volume list --json +modal volume ls VOLUME_NAME / +modal secret list --json +modal dict list +modal queue list +modal endpoint list +``` + +`modal secret list` returns names, not values. Collect the values from the system you created them from, because you'll re-enter them on Runpod. + +Then find the code. This search covers Modal imports, decorators, lookups, CLI calls in scripts and CI, credentials, and hard-coded URLs: + +```bash +grep -rnE --include='*.py' --include='*.ts' --include='*.js' --include='*.go' \ + --include='*.yml' --include='*.yaml' --include='*.sh' --include='*.toml' --include='Dockerfile*' \ + 'import modal|from modal|modal\.(App|Stub|Image|Volume|Secret|Dict|Queue|Cron|Period|Sandbox|Function|Cls|CloudBucketMount)|@modal\.|\.from_name\(|modal (run|serve|deploy|shell|volume|secret)|MODAL_TOKEN_(ID|SECRET)|MODAL_ENVIRONMENT|\.modal\.run|\.modal\.host' . +``` + +For each Function, Cls, and Server, record these settings from its decorator: `gpu`, `cpu`, `memory`, `ephemeral_disk`, `timeout`, `retries`, `schedule`, `region`, `min_containers`, `max_containers`, `buffer_containers`, `scaledown_window`, `volumes`, `secrets`, the image definition, and any `@modal.concurrent` or `@modal.batched` settings. Older code uses names that Modal has since renamed. Treat them as the same settings: + +| Name in older code | Current Modal name | +| --- | --- | +| `modal.Stub` | `modal.App` | +| `keep_warm` | `min_containers` | +| `concurrency_limit` | `max_containers` | +| `container_idle_timeout` | `scaledown_window` | +| `allow_concurrent_inputs=N` | `@modal.concurrent(max_inputs=N)` | +| `@modal.web_endpoint` | `@modal.fastapi_endpoint` | +| `mount=`, `modal.Mount`, `Image.copy_local_dir` | `Image.add_local_dir`, `Image.add_local_file`, `Image.add_local_python_source` | +| `@modal.build` | A Volume, or `Image.run_function` | +| `.lookup()` | `.from_name()` | + +Finally, record who calls each workload: Python callers that use `modal.Function.from_name` or `modal.Cls.from_name`, HTTP clients that call `modal.run` URLs, which of those URLs require Proxy Tokens, and any custom domains. + +## Step 2: Set up Runpod + +Install the Runpod SDK, which includes the `rp` CLI, and log in: + +```bash +pip install runpod +rp login +``` + +`rp login` opens your browser and saves credentials to `~/.runpod/config.toml`. In CI, set the `RUNPOD_API_KEY` environment variable instead. Create keys on the [Credentials page](/get-started/credentials). + +Install `runpodctl` to manage endpoints, templates, and volumes from the terminal: + +```bash +curl -fsSL https://cli.runpod.net | sudo bash +runpodctl doctor +``` + +`runpodctl doctor` configures the API key and checks the setup. On macOS you can also install with `brew install runpod/runpodctl/runpodctl`. If you automate with an agent, connect the [Runpod MCP server](/get-started/mcp-servers). + +## Step 3: Choose where each workload lands + +| Modal workload | Runpod target | Why | +| --- | --- | --- | +| `@app.function` or `@app.cls` called with `.remote()` or `.spawn()`, seconds to minutes per call | `@app.queue` | Autoscaling job queue, scales to zero, every job is queued. | +| Function with `timeout` near the 24-hour maximum, training, fine-tuning, batch pipelines | `@app.task` | One dedicated Pod per call, terminated when the function returns. | +| `@modal.fastapi_endpoint` or a small `@modal.asgi_app` with JSON routes | `@app.api` | Load-balanced HTTP routes on a class, with a once-per-worker init hook. | +| Full FastAPI, Flask, or Django app, streaming responses, `@modal.web_server`, `@app.server` | Load-balancing endpoint with a custom image | You keep your HTTP server and add a `/ping` health route. | +| App with a long `modal.Image` chain, `from_dockerfile`, or a prebuilt inference server | Custom Docker image with a [handler function](/serverless/workers/handler-functions) | Full control of the image. Queue semantics stay the same. | +| Modal Shared Endpoint (per-token LLM API) | [Public Endpoint](/public-endpoints/overview) | Hosted model, no infrastructure. | +| Modal Dedicated Endpoint, or a vLLM Server | [vLLM worker](/serverless/vllm/overview) | OpenAI-compatible route at `https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1`. | +| `modal.Sandbox`, `modal shell`, notebooks, tunnels | [Pod](/pods/overview) | Long-lived container with SSH and exposed ports. | +| `@modal.clustered` multi-node jobs | [Instant Cluster](/instant-clusters/pytorch) | Multi-node networking for training frameworks. | +| Scheduled Function | The matching target above, triggered by an external scheduler | Runpod runs the work. Your scheduler starts it. | + +### GPU mapping + +Choose the GPU that fits the workload's memory and throughput needs, then confirm it's in stock where your data lives. In the SDK, a short name such as `"H100"` matches every device whose name contains it, and a `runpod.GpuGroup` pool allows any GPU in that pool. Queue and API resources select hardware by pool. Tasks select by device. + +| Modal `gpu=` | Runpod SDK `gpu=` | Pool | Notes | +| --- | --- | --- | --- | +| `"T4"` | `runpod.GpuGroup.AMPERE_16` | `AMPERE_16` | The pool holds 16 GB cards (RTX A4000, RTX A4500, RTX 4000 Ada, RTX 2000 Ada). | +| `"L4"` | `"L4"` | `AMPERE_24` | 24 GB. | +| `"A10"` (also written `"A10G"`) | `runpod.GpuGroup.AMPERE_24` or `runpod.GpuGroup.ADA_24` | `AMPERE_24`, `ADA_24` | 24 GB class (RTX A5000, L4, RTX 3090, or RTX 4090). | +| `"L40S"` | `"L40S"` | `ADA_48_PRO` | 48 GB. | +| `"A100"`, `"A100-40GB"` | `runpod.GpuGroup.AMPERE_48` or `runpod.GpuGroup.AMPERE_80` | `AMPERE_48`, `AMPERE_80` | Runpod's A100 pool is 80 GB. A 40 GB workload also fits a 48 GB pool. | +| `"A100-80GB"` | `runpod.GpuGroup.AMPERE_80` | `AMPERE_80` | 80 GB. | +| `"RTX-PRO-6000"` | `runpod.GpuGroup.BLACKWELL_96` | `BLACKWELL_96` | 96 GB. | +| `"H100"`, `"H100!"` | `"H100"` | `ADA_80_PRO` | Modal may run `"H100"` on an H200. On Runpod, `"H100"` stays on H100 variants. | +| `"H200"` | `"H200"` | `HOPPER_141` | 141 GB. | +| `"B200"`, `"B200+"` | `"B200"` | `BLACKWELL_180` | 180 GB. | +| `"B300"` | Check `runpodctl gpu list` | | The [GPU reference](/references/gpu-types) lists a B300 device. Confirm the ID and stock before you plan on it. | +| `"any"` | `runpod.GpuGroup.ANY` | | Any available GPU. | +| `"H100:8"` | `gpu="H100", gpu_count=8` | | The count is a separate argument. | +| `["H100", "A100-80GB"]` | `["H100", runpod.GpuGroup.AMPERE_80]` | | A list sets fallbacks. Automatic switching between listed GPUs needs a worker maximum of at least 5. | + +See live stock with `runpodctl gpu list`, and see [GPU types](/sdk/configuration/gpu-types) for every pool and device name. For CPU-only Functions, pick an instance ID from [CPU types](/sdk/configuration/cpu-types), for example `cpu="cpu3c-2-4"` for 2 vCPUs and 4 GB. Modal's `cpu=` counts physical cores, and Runpod CPU instances count vCPUs. + +## Step 4: Move your images and code + +### Translate the image + +Start by deciding whether the `modal.Image` fits the SDK's built-in path. This table maps each builder method: + +| `modal.Image` method | Runpod SDK | Dockerfile | +| --- | --- | --- | +| `debian_slim(python_version="3.12")` | Runpod runtime image. Workers use your local Python version, or `rp flash deploy --python-version`. | `FROM python:3.12-slim` | +| `pip_install(...)`, `uv_pip_install(...)`, `pip_install_from_requirements(...)` | `dependencies=[...]` and `requirements.txt` | `RUN pip install ...` | +| `apt_install(...)` | `system_dependencies=[...]` | `RUN apt-get update && apt-get install -y ...` | +| `env({...})` | `env={...}` | `ENV` | +| `add_local_python_source(...)`, `add_local_dir(...)`, `add_local_file(...)` | Deploy packages the project directory, filtered by `.gitignore` and `.runpodignore`. | `COPY` | +| `from_registry("org/image:tag")`, `from_aws_ecr(...)` | `image="org/image:tag"` with `registry_auth=` for private images | `FROM org/image:tag` | +| `from_dockerfile("Dockerfile")` | Build and push the image, then set `image=` | Use the Dockerfile as is. | +| `run_commands(...)`, `dockerfile_commands(...)` | Bake into a custom image | `RUN` | +| `run_function(download_weights)` | `model=runpod.Model(...)` or a volume | `RUN python download.py`, or keep weights out of the image. | +| `workdir(...)`, `entrypoint(...)`, `cmd(...)` | Bake into a custom image | `WORKDIR`, `ENTRYPOINT`, `CMD` | + +On Runpod's built-in GPU images, `torch`, `torchvision`, `torchaudio`, and `triton` are already installed and are left out of the artifact, so don't pin them. A custom `image=` needs Python and `pip`, because the worker installs the SDK runtime when it starts. All images must be built for `linux/amd64`, which is the same requirement Modal places on external images. + +Two habits carry over unchanged: import heavy packages inside the function body, and keep blocking work out of module scope. Replace `modal.is_local()` with `runpod.is_local()`. + + + In a dev session (`rp flash dev`), only the file that defines the app is sent to workers. A project that imports sibling modules works with `rp flash deploy`, which packages the project directory, but fails in a dev session. Keep helpers in the app file or in an installable package while you iterate. + + +### Construct-by-construct mapping + +| Modal | Runpod SDK | Notes | +| --- | --- | --- | +| `app = modal.App("name")` | `app = runpod.App("name")` | `runpod.App("name", env="production")` sets a default environment. | +| `@app.function(...)` | `@app.queue(...)` or `@app.task(...)` | Resource names need at least 3 characters to deploy. | +| `@app.cls(...)` + `@modal.method()` | `@app.queue(...)` per method, or `@app.api(...)` on the class | | +| `@modal.enter()` | Module-level cache in a queue function, or `@runpod.init` on an API class | | +| `@modal.exit()` | Your own shutdown handling | The SDK docs show a startup hook only. | +| `@app.local_entrypoint()` | `@runpod.local_entrypoint` | No parentheses. Runs on your machine under `rp flash dev`. | +| `fn.remote(x)` | `fn.remote(x)` | | +| `await fn.remote.aio(x)` | `await fn.remote.aio(x)` | | +| `fn.spawn(x)` then `call.get(timeout=...)` | `fn.spawn(x)` then `job.result(timeout=...)` | Queue jobs also have `job.status()`, `job.cancel()`, and `job.retry()`. | +| `fn.remote_gen(x)` | `fn.stream(x)` | For generator functions on queue resources. | +| `fn.local(x)` | `fn.local(x)` | Volumes aren't mounted locally. | +| `fn.map(xs)` | `[fn.spawn(x) for x in xs]`, then `job.result()` on each | No built-in map. | +| `gpu="H100:2"` | `gpu="H100", gpu_count=2` | | +| `cpu=2.0, memory=4096` (CPU Function) | `cpu="cpu3c-2-4"` | Fixed shapes. | +| `ephemeral_disk=` (MiB) | `container_disk_gb=` (GB) | Defaults: 30 GB on GPU, 10 GB on CPU. | +| `min_containers=1, max_containers=10` | `workers=(1, 10)` | Default is `(0, 3)`. | +| `scaledown_window=300` | `idle_timeout=300` | Both in seconds. Both default to 60. | +| `timeout=600` | `execution_timeout_ms=600000` | Queue and API only. Default is no limit. | +| `@modal.concurrent(max_inputs=8)` | `max_concurrency=8` | Queue resources only. | +| `retries=3` | Retry in the caller or with `job.retry()` | | +| `schedule=modal.Cron("0 8 * * 1")` | External scheduler | | +| `region="us-east"` | `datacenter="US-KS-2"` | Accepts a list. | +| `volumes={"/data": vol}` | `mounts={"/runpod-volume": vol}` | Queue and API mount at `/runpod-volume` only. Tasks choose the path. | +| `modal.Volume.from_name("x", create_if_missing=True)` | `runpod.NetworkVolume("x", size=100)` | Created on first use. Pass `create=False` to require an existing volume. | +| `vol.commit()`, `vol.reload()` | Delete the call | | +| `secrets=[modal.Secret.from_name("x")]` | `env={"KEY": runpod.Secret("x")}` | One secret per value. | +| `modal.Secret.from_dict({...})`, `from_dotenv()` | Plain values in `env={...}` for non-sensitive settings, `runpod.Secret` for the rest | Local `.env` files are never sent to workers. | +| `image=modal.Image...` | `dependencies=`, `system_dependencies=`, `image=` | | +| `enable_memory_snapshot=True` | `flashboot=True` (the default) | Different mechanism. | +| `modal.Function.from_name("app", "fn")` | `runpod.Queue(app="app", name="fn")` | Keyword arguments only. | +| `modal.Cls.from_name("app", "Model")` | `runpod.Api(app="app", name="Model")` | Call routes with `.post(path, body)` and `.get(path)`. | + +### Example 1: A GPU function called with .remote() + +This Modal App embeds text on an L4 and is called from a local entrypoint: + +```python embed_modal.py +import modal + +app = modal.App("embed") + +image = modal.Image.debian_slim(python_version="3.12").uv_pip_install( + "sentence-transformers" +) + + +@app.function( + image=image, + gpu="L4", + timeout=600, + scaledown_window=120, + max_containers=5, +) +def embed(texts: list[str]) -> list[list[float]]: + from sentence_transformers import SentenceTransformer + + model = SentenceTransformer("all-MiniLM-L6-v2", device="cuda") + return model.encode(texts).tolist() + + +@app.local_entrypoint() +def main(): + vectors = embed.remote(["hello", "world"]) + print(len(vectors), len(vectors[0])) +``` + +The Runpod version keeps the function body and moves the configuration onto `@app.queue`: + +```python embed_runpod.py +import runpod + +app = runpod.App("embed") + + +@app.queue( + gpu="L4", + dependencies=["sentence-transformers"], + workers=(0, 5), + idle_timeout=120, + execution_timeout_ms=600000, +) +def embed(texts: list[str]) -> list[list[float]]: + from sentence_transformers import SentenceTransformer + + model = SentenceTransformer("all-MiniLM-L6-v2", device="cuda") + return model.encode(texts).tolist() + + +@runpod.local_entrypoint +def main(): + vectors = embed.remote(["hello", "world"]) + print(len(vectors), len(vectors[0])) +``` + +Run it with `rp flash dev embed_runpod.py` in place of `modal run embed_modal.py`. The image definition became `dependencies`, the timeout changed from seconds to milliseconds, and `max_containers=5` became the upper bound of `workers`. The function already returns a list, so its result is valid JSON. + +### Example 2: A class that loads a model in @modal.enter + +This Modal Cls loads a text-generation model once per container, caches weights on a Volume, and keeps one container warm: + +```python textgen_modal.py +import modal + +app = modal.App("textgen") + +image = ( + modal.Image.debian_slim(python_version="3.12") + .uv_pip_install("torch", "transformers", "accelerate") + .env({"HF_HUB_CACHE": "/cache"}) +) +hf_cache = modal.Volume.from_name("hf-hub-cache", create_if_missing=True) + + +@app.cls( + image=image, + gpu="A10", + volumes={"/cache": hf_cache}, + secrets=[modal.Secret.from_name("huggingface")], # Provides HF_TOKEN + min_containers=1, + max_containers=4, + scaledown_window=300, +) +class Generator: + @modal.enter() + def load(self): + from transformers import pipeline + + self.pipe = pipeline( + "text-generation", model="Qwen/Qwen2.5-7B-Instruct", device=0 + ) + + @modal.method() + def generate(self, prompt: str, max_new_tokens: int = 64) -> str: + out = self.pipe(prompt, max_new_tokens=max_new_tokens) + return out[0]["generated_text"] + + +@app.local_entrypoint() +def main(): + print(Generator().generate.remote("Runpod is")) +``` + +On Runpod you have two shapes. Use a queue resource when callers used `.remote()` or `.spawn()`, and an API resource when you want HTTP routes. + + + + The queue version caches the pipeline in a module-level variable, so the load runs once per worker: + + ```python textgen_runpod.py + import runpod + + app = runpod.App("textgen") + + qwen = runpod.Model("Qwen/Qwen2.5-7B-Instruct") + _PIPE = None + + + @app.queue( + gpu=runpod.GpuGroup.AMPERE_24, + workers=(1, 4), + idle_timeout=300, + model=qwen, + dependencies=["transformers", "accelerate"], + env={"HF_TOKEN": runpod.Secret("hf-token")}, + ) + def generate(prompt: str, max_new_tokens: int = 64) -> str: + global _PIPE + if _PIPE is None: + from transformers import pipeline + + _PIPE = pipeline("text-generation", model=str(qwen.path), device=0) + out = _PIPE(prompt, max_new_tokens=max_new_tokens) + return out[0]["generated_text"] + + + @runpod.local_entrypoint + def main(): + print(generate.remote("Runpod is")) + ``` + + `Generator().generate.remote(...)` became `generate.remote(...)`. The Volume used as a Hugging Face cache became `runpod.Model`, which stages the weights on the host before the worker starts, and `qwen.path` points to them inside the worker. + + + + The API version keeps the class. `@runpod.init` replaces `@modal.enter()`, and each method becomes a route: + + ```python textgen_api_runpod.py + import runpod + + app = runpod.App("textgen") + + qwen = runpod.Model("Qwen/Qwen2.5-7B-Instruct") + + + @app.api( + gpu=runpod.GpuGroup.AMPERE_24, + workers=(1, 4), + idle_timeout=300, + model=qwen, + dependencies=["transformers", "accelerate"], + env={"HF_TOKEN": runpod.Secret("hf-token")}, + ) + class Generator: + @runpod.init + def load(self): + from transformers import pipeline + + self.pipe = pipeline("text-generation", model=str(qwen.path), device=0) + + @runpod.post("/generate") + async def generate(self, body: dict): + out = self.pipe( + body["prompt"], max_new_tokens=body.get("max_new_tokens", 64) + ) + return {"text": out[0]["generated_text"]} + + @runpod.get("/health") + async def health(self): + return {"status": "ok"} + + + @runpod.local_entrypoint + def main(): + print(Generator.post("/generate", {"prompt": "Runpod is"})) + ``` + + A POST route receives the request's JSON body as its one argument. In a dev session you call routes through the class. After deploy, the same routes are served at `https://ENDPOINT_ID.api.runpod.ai/generate`. + + + +`min_containers=1` became `workers=(1, 4)`, which keeps one worker running and billed at all times. The Modal Secret named `huggingface` held an `HF_TOKEN` key. On Runpod, the secret `hf-token` holds the token value and the `env` mapping names the variable. If your weights aren't on Hugging Face, mount a network volume at `/runpod-volume` and see [Volume cache](/serverless/development/volume-cache) for loading them quickly. + +### Example 3: A FastAPI web endpoint + +This Modal ASGI app serves two routes, handles 20 concurrent requests per container, and requires a Proxy Token: + +```python classify_modal.py +import modal + +app = modal.App("classify") + +image = modal.Image.debian_slim(python_version="3.12").uv_pip_install( + "fastapi[standard]", "torch", "transformers" +) + + +@app.function(image=image, gpu="T4", min_containers=1) +@modal.concurrent(max_inputs=20) +@modal.asgi_app(requires_proxy_auth=True) +def web(): + from fastapi import FastAPI + from transformers import pipeline + + api = FastAPI() + classifier = pipeline("sentiment-analysis", device=0) + + @api.get("/health") + async def health(): + return {"status": "ok"} + + @api.post("/classify") + async def classify(body: dict): + return classifier(body["text"])[0] + + return api +``` + + + + When the routes take and return JSON, translate them to an `@app.api` class: + + ```python classify_runpod.py + import runpod + + app = runpod.App("classify") + + + @app.api( + gpu=runpod.GpuGroup.AMPERE_16, + workers=(1, 3), + dependencies=["transformers"], + ) + class Classifier: + @runpod.init + def load(self): + from transformers import pipeline + + self.classifier = pipeline("sentiment-analysis", device=0) + + @runpod.get("/health") + async def health(self): + return {"status": "ok"} + + @runpod.post("/classify") + async def classify(self, body: dict): + return self.classifier(body["text"])[0] + + + @runpod.local_entrypoint + def main(): + print(Classifier.get("/health")) + print(Classifier.post("/classify", {"text": "This works."})) + ``` + + A GET route takes no arguments and a POST route takes the JSON body. FastAPI features such as path parameters, Pydantic request models, middleware, and streaming responses aren't part of this route model, so use the custom image tab if you depend on them. + + + + To keep the FastAPI app as it is, run it as the server of a load-balancing endpoint. Add a `/ping` health route and listen on the port in the `PORT` environment variable: + + ```python app.py + import os + + import uvicorn + from fastapi import FastAPI + from transformers import pipeline + + api = FastAPI() + classifier = pipeline("sentiment-analysis", device=0) + + + @api.get("/ping") + async def ping(): + return {"status": "healthy"} + + + @api.get("/health") + async def health(): + return {"status": "ok"} + + + @api.post("/classify") + async def classify(body: dict): + return classifier(body["text"])[0] + + + if __name__ == "__main__": + uvicorn.run(api, host="0.0.0.0", port=int(os.getenv("PORT", "80"))) + ``` + + Package it with a Dockerfile: + + ```dockerfile Dockerfile + FROM runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04 + + WORKDIR /app + RUN pip install --no-cache-dir "fastapi[standard]" transformers + COPY app.py . + + CMD ["python", "-u", "app.py"] + ``` + + The load balancer polls `/ping` and sends traffic once it returns `200`. Return `204` while the model loads if startup is slow. Deploy this image in [Step 6](#load-balancing-endpoint-from-a-custom-image). + + + +On both paths, callers send `Authorization: Bearer` with a Runpod API key in place of the `Modal-Key` and `Modal-Secret` headers. `@modal.concurrent` has no argument on API resources: the routes are `async`, and the endpoint scales on request count (`scaler_type="REQUEST_COUNT"`, `scaler_value=4` by default). + +### Example 4: A training job with a Volume and Secrets + +This Modal Function trains on an H100, checkpoints to a Volume, logs to Weights & Biases, and retries after a failure: + +```python train_modal.py +import modal + +app = modal.App("finetune") + +image = modal.Image.debian_slim(python_version="3.12").uv_pip_install("torch", "wandb") +ckpt = modal.Volume.from_name("checkpoints", create_if_missing=True) + + +@app.function( + image=image, + gpu="H100", + volumes={"/ckpt": ckpt}, + secrets=[modal.Secret.from_name("wandb")], # Provides WANDB_API_KEY + timeout=24 * 60 * 60, + retries=modal.Retries(max_retries=2, initial_delay=10.0, backoff_coefficient=2.0), +) +def train(run_name: str, epochs: int = 3) -> dict: + from pathlib import Path + + import torch + import wandb + + run_dir = Path("/ckpt") / run_name + run_dir.mkdir(parents=True, exist_ok=True) + state_file = run_dir / "state.pt" + + model = torch.nn.Linear(128, 1).cuda() + opt = torch.optim.AdamW(model.parameters(), lr=1e-3) + start = 0 + if state_file.exists(): # Resume after a retry or preemption + state = torch.load(state_file) + model.load_state_dict(state["model"]) + start = state["epoch"] + 1 + + wandb.init(project="finetune", name=run_name, resume="allow") + loss = torch.tensor(0.0) + for epoch in range(start, epochs): + x = torch.randn(4096, 128, device="cuda") + loss = ((model(x) - x.sum(dim=1, keepdim=True)) ** 2).mean() + opt.zero_grad() + loss.backward() + opt.step() + wandb.log({"epoch": epoch, "loss": loss.item()}) + torch.save({"model": model.state_dict(), "epoch": epoch}, state_file) + ckpt.commit() + + return {"checkpoint": str(state_file), "loss": loss.item()} + + +@app.local_entrypoint() +def main(): + print(train.remote("run-1", epochs=3)) +``` + +On Runpod this is a task. Each call gets its own Pod, and the Pod is terminated when the function returns: + +```python train_runpod.py +import runpod + +app = runpod.App("finetune") + +ckpt = runpod.NetworkVolume("checkpoints", size=100) + + +@app.task( + gpu="H100", + mounts={"/ckpt": ckpt}, + env={"WANDB_API_KEY": runpod.Secret("wandb-api-key")}, + dependencies=["wandb"], + container_disk_gb=50, +) +def train(run_name: str, epochs: int = 3) -> dict: + import torch + import wandb + + run_dir = ckpt.path / run_name + run_dir.mkdir(parents=True, exist_ok=True) + state_file = run_dir / "state.pt" + + model = torch.nn.Linear(128, 1).cuda() + opt = torch.optim.AdamW(model.parameters(), lr=1e-3) + start = 0 + if state_file.exists(): # Resume when you run the task again + state = torch.load(state_file) + model.load_state_dict(state["model"]) + start = state["epoch"] + 1 + + wandb.init(project="finetune", name=run_name, resume="allow") + loss = torch.tensor(0.0) + for epoch in range(start, epochs): + x = torch.randn(4096, 128, device="cuda") + loss = ((model(x) - x.sum(dim=1, keepdim=True)) ** 2).mean() + opt.zero_grad() + loss.backward() + opt.step() + wandb.log({"epoch": epoch, "loss": loss.item()}) + torch.save({"model": model.state_dict(), "epoch": epoch}, state_file) + + return {"checkpoint": str(state_file), "loss": loss.item()} + + +@runpod.local_entrypoint +def main(): + print(train.remote("run-1", epochs=3)) +``` + +Run it with `rp flash dev train_runpod.py --once`, which runs the entrypoint a single time and exits with an error if it fails. The changes from the Modal version: + +- `ckpt.path` returns the mount path inside the task. Read it inside the function, not at module level. +- `ckpt.commit()` is gone. Writes to a network volume are durable as they happen. +- `torch` isn't in `dependencies`, because the built-in GPU image provides it. +- There's no `timeout`. Tasks have no execution limit, so a hung job keeps billing until you stop it. To bound a run, call `job = train.spawn(...)` and then `await job.wait(timeout=...)`, which terminates the Pod when the wait ends. +- There's no `retries`. The resume logic stays, and you run the task again after a failure. + + + A task started with `.spawn()` that you never wait on or cancel keeps its Pod running after your script exits. Check the [Pods page](https://console.runpod.io/pods) after interrupted runs. + + +### The custom image path: a handler function + +When an App can't fit the SDK's dependency model, build a Docker image with a [handler function](/serverless/workers/handler-functions). The handler receives each job as a dict and returns a JSON-serializable result. This handler does the same job as the Modal Cls in Example 2: + +```python handler.py +import runpod +from transformers import pipeline + +# Loaded once when the worker starts, like @modal.enter() +pipe = pipeline("text-generation", model="Qwen/Qwen2.5-7B-Instruct", device=0) + + +def handler(job): + job_input = job["input"] + out = pipe(job_input["prompt"], max_new_tokens=job_input.get("max_new_tokens", 64)) + return {"text": out[0]["generated_text"]} + + +runpod.serverless.start({"handler": handler}) +``` + +The Dockerfile replaces the `modal.Image` chain line for line: + +```dockerfile Dockerfile +FROM runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04 + +WORKDIR / +RUN pip install --no-cache-dir runpod transformers accelerate +COPY handler.py . + +CMD ["python", "-u", "/handler.py"] +``` + +Test the handler locally with `python handler.py --test_input '{"input": {"prompt": "Runpod is"}}'` before you build. A generator handler streams results the way `remote_gen` does, and a `concurrency_modifier` replaces `@modal.concurrent`. See [Concurrent handlers](/serverless/workers/concurrent-handler). You can also let Runpod build the image from a repository with the [GitHub integration](/serverless/workers/github-integration). + +## Step 5: Move your data and models + +### Secrets + +Re-create each key of each Modal Secret as a Runpod secret: + +```bash +rp secret add hf-token +rp secret add wandb-api-key +rp secret list +``` + +`rp secret add` prompts for the value, which keeps it out of your shell history. Secret names must be unique in the account, start with a letter or underscore, and can't start with `runpod`. In SDK code, reference a secret with `env={"HF_TOKEN": runpod.Secret("hf-token")}`. In a template or endpoint you create outside the SDK, set the environment variable's value to `{{ RUNPOD_SECRET_hf-token }}`. A changed secret takes effect the next time a worker or Pod boots. See [Manage secrets](/pods/templates/secrets). + +### Hugging Face models + +If a Volume only caches Hugging Face downloads, don't copy it. Use `model=runpod.Model("owner/name")` on a queue or API resource, or `--model-reference` on an endpoint you create with `runpodctl`. Runpod stages the weights on the host before the worker starts. Gated models need `HF_TOKEN` on the same resource. See [Cached models](/serverless/endpoints/model-caching). + +### Volumes + +Create a network volume in the data center where you plan to run. Check GPU stock there first, because every resource that mounts the volume runs in that data center: + +```bash +runpodctl gpu list +runpodctl datacenter list +runpodctl network-volume create --name checkpoints --size 100 --data-center-id US-KS-2 +``` + +The command returns the volume ID. Size is in GB, from 10 to 4096, and you can grow a volume later but not shrink it. In SDK code, `runpod.NetworkVolume("checkpoints", create=False)` then refers to this volume by name. + +Download the Modal Volume and upload it to the network volume through the volume's [S3-compatible API](/storage/s3-api). You need an S3 API key from the [Credentials page](/get-started/credentials) and a data center that supports the API. Configure the AWS CLI with that key (`aws configure`), then sync: + +```bash +modal volume get checkpoints / ./checkpoints-export + +aws s3 sync ./checkpoints-export s3://VOLUME_ID/ \ + --endpoint-url https://s3api-us-ks-2.runpod.io/ \ + --region US-KS-2 +``` + +Files land at the volume root, which is `/runpod-volume` on Serverless workers and `/workspace` on Pods. `aws s3 sync` can have trouble with more than 10,000 files or deeply nested trees, so for large trees copy from inside a Pod as shown next. See [S3-compatible API](/storage/s3-api) for the supported data centers and endpoint URLs. + +For a Volume too large to stage on your machine, copy from inside Runpod. Start a CPU Pod with the network volume attached, then run the Modal CLI there: + +```bash +runpodctl pod create --name modal-copy --compute-type cpu --image python:3.12 \ + --network-volume-id VOLUME_ID --data-center-ids US-KS-2 \ + --env '{"MODAL_TOKEN_ID":"YOUR_MODAL_TOKEN_ID","MODAL_TOKEN_SECRET":"YOUR_MODAL_TOKEN_SECRET"}' + +runpodctl ssh info POD_ID +# Then, in a shell on the Pod: +pip install modal +modal volume get checkpoints / /workspace/ + +# When the copy finishes, back on your machine: +runpodctl pod delete POD_ID +``` + +Use a short-lived Modal token for this and rotate it afterward, because environment variables are visible in the Pod's configuration. + +To serve the same files in several data centers, either create one network volume per data center and copy the data to each, or use a [global volume](/storage/globalvolume/overview). Global volumes are in beta, attach to GPU compute, are billed for storage and requests, and don't support file locking or atomic rename. The S3-compatible API works with network volumes only, so load a global volume from a Pod. + +### Cloud bucket mounts + +`modal.CloudBucketMount` presents an S3, R2, or GCS bucket as a directory. On Runpod, choose one of these: + +- Read and write the bucket from your code with `boto3` or the provider's SDK, with credentials in Runpod secrets. This suits buckets that stay the source of truth. +- Copy the data to a network volume once, with `aws s3 sync` or `rclone` from a Pod, when workers need file system access to a fixed dataset. +- Use [Cloud Sync](/pods/storage/cloud-sync) to move data between a Pod and S3, Google Cloud Storage, Azure Blob Storage, Backblaze B2, or Dropbox. + +### Dicts and Queues + +Export any state you need from `modal.Dict` objects before you shut down (`modal dict items DICT_NAME --all --json`), and load it into the store that replaces them. Values are pickled Python objects, so convert them to a portable format as you export. + +## Step 6: Deploy on Runpod + +### SDK apps + +Deploy from the project directory: + +```bash +rp flash deploy +rp flash deploy --env production +rp flash app list +rp flash env get production +``` + +`rp flash deploy` finds every app in the directory, builds an artifact, and creates or updates the endpoints for queue and API resources. It prints a link to each endpoint when it finishes. `--env` plays the role of `modal deploy --env`, and running the command again updates the deployment in place. Tasks aren't deployed as endpoints. They provision a Pod each time your code calls them. + +Before you deploy to production, pin GPUs instead of using `GpuGroup.ANY`, set `execution_timeout_ms`, and set `workers` explicitly. See [Configuration best practices](/sdk/configuration/best-practices). + + + Deploys and dev sessions add your account API key as `RUNPOD_API_KEY` to each endpoint's environment so that one resource can call another. Anyone who can read the endpoint's template can read that key, so limit who has access to the account or team. + + +### Queue endpoint from a custom image + +Build and push the handler image for `linux/amd64`, then create a Serverless template and an endpoint: + +```bash +docker build --platform linux/amd64 -t YOUR_REGISTRY/textgen:v1 . +docker push YOUR_REGISTRY/textgen:v1 + +runpodctl template create --name textgen-v1 --image YOUR_REGISTRY/textgen:v1 --serverless \ + --container-disk-in-gb 40 \ + --env '{"HF_TOKEN":"{{ RUNPOD_SECRET_hf-token }}"}' + +runpodctl serverless create --name textgen --template-id TEMPLATE_ID \ + --gpu-id AMPERE_24 --workers-min 1 --workers-max 4 \ + --idle-timeout 300 --execution-timeout 600 --min-cuda-version 12.8 \ + --model-reference https://huggingface.co/Qwen/Qwen2.5-7B-Instruct:main +``` + +`--gpu-id` accepts a pool ID or a device ID, and `--execution-timeout` is in seconds here. Add `--network-volume-id VOLUME_ID` to mount a network volume at `/runpod-volume`. For a private registry, save credentials with `runpodctl registry create` and pass `--registry-auth-id` when you create the template. In the console, the same flow is **Serverless**, then **New Endpoint**, then **Import from Docker Registry**. + +### Load-balancing endpoint from a custom image + +The endpoint type is set at creation through REST API v2 or the console. `runpodctl serverless create` creates queue-based endpoints. Create a load-balancing endpoint with a POST to `/v2/serverless`: + +```bash +curl -s -X POST https://api.runpod.io/v2/serverless \ + -H "Authorization: Bearer YOUR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "classify", + "image": "YOUR_REGISTRY/classify:v1", + "type": "LOAD_BALANCER", + "gpu": {"pools": ["AMPERE_16"], "count": 1, "minCudaVersion": "12.8"}, + "disk": 20, + "ports": ["8000/http"], + "env": {"PORT": "8000", "PORT_HEALTH": "8000"}, + "workers": {"min": 1, "max": 3, "idleTimeout": 60}, + "scaling": {"type": "REQUEST_COUNT", "requestCount": 4}, + "flashboot": "FLASHBOOT" + }' +``` + +`type` and `scaling` are required, and load-balancing endpoints scale on `REQUEST_COUNT`. Set `PORT_HEALTH` whenever you change `PORT`, because each defaults to 80 on its own. The response includes the endpoint ID and its request URLs. REST v2 defaults differ from the console's: `flashboot` defaults to `OFF` and `workers.idleTimeout` to 10 seconds, so set them. In the console, choose **Load Balancer** under **Endpoint Type**. See [Build a load balancing worker](/serverless/load-balancing/build-a-worker). + +## Step 7: Update the code that calls Modal + +### Python callers + +Code that looks up a deployed Modal Function changes to a Runpod client. Before: + +```python caller_modal.py +import modal + +embed = modal.Function.from_name("embed", "embed") +vectors = embed.remote(["hello", "world"]) + +call = embed.spawn(["later"]) +result = call.get(timeout=120) + +Generator = modal.Cls.from_name("textgen", "Generator") +text = Generator().generate.remote("Runpod is") +``` + +After, using resources deployed with the Runpod SDK: + +```python caller_runpod.py +import runpod + +embed = runpod.Queue(app="embed", name="embed") +vectors = embed.remote(texts=["hello", "world"]) # Keyword arguments only + +job = embed.spawn(texts=["later"]) +result = job.result(timeout=120) + +generator = runpod.Api(app="textgen", name="Generator") +text = generator.post("/generate", {"prompt": "Runpod is"})["text"] +``` + +`runpod.Queue` and `runpod.Api` target the `default` environment unless you pass `env=`. They authenticate with `rp login` credentials or `RUNPOD_API_KEY`, in place of `MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET`. + +For any queue endpoint, including one built from a custom image, call it by ID: + +```python caller_endpoint.py +import runpod + +endpoint = runpod.Endpoint("ENDPOINT_ID") + +result = endpoint.run_sync({"prompt": "Runpod is"}, timeout=120) + +run_request = endpoint.run({"prompt": "Runpod is"}) +print(run_request.status()) +print(run_request.output()) +``` + +The client wraps your dict as `{"input": {...}}`. For a function deployed with `@app.queue`, the keys of `input` are the function's keyword arguments. + +### HTTP callers + +A request to a Modal Web Function looks like this: + +```bash +curl -X POST https://WORKSPACE--classify-web.modal.run/classify \ + -H "Modal-Key: YOUR_MODAL_TOKEN_ID" \ + -H "Modal-Secret: YOUR_MODAL_TOKEN_SECRET" \ + -H "Content-Type: application/json" \ + -d '{"text": "This works."}' +``` + +For an API resource or a load-balancing endpoint, change the host and the auth header. The path and body stay the same: + +```bash +curl -X POST https://ENDPOINT_ID.api.runpod.ai/classify \ + -H "Authorization: Bearer YOUR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"text": "This works."}' +``` + +For a queue endpoint, the body moves under `input` and the result comes back under `output`: + +```bash +curl -X POST https://api.runpod.ai/v2/ENDPOINT_ID/runsync \ + -H "Authorization: Bearer YOUR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"input": {"prompt": "Runpod is", "max_new_tokens": 64}}' +``` + +The response is a JSON envelope with `id`, `status`, `delayTime`, `executionTime`, and `output`, so update clients that read the bare return value. + +### Long requests, polling, streaming, and webhooks + +| Modal pattern | Runpod pattern | +| --- | --- | +| Web Function request longer than 150 seconds, followed through `303` redirects | `/runsync` waits 90 seconds by default (`?wait=` up to 300000 ms). For longer jobs, use `/run` and poll `/status/JOB_ID`. | +| `fn.spawn()` then `FunctionCall.from_id(id).get()` | `POST /run` returns a job ID. `GET /status/JOB_ID` returns the state and output. | +| `call.cancel()` | `POST /cancel/JOB_ID` | +| Generator Function with `remote_gen`, or a streaming response | Generator handler or generator queue function, read with `GET /stream/JOB_ID` or `fn.stream()`. On load-balancing endpoints, stream from your own server. | +| Polling a web endpoint you wrote for job status | Add `"webhook": "https://your-service.example.com/hook"` next to `input`, and Runpod calls it when the job finishes. | +| OpenAI-compatible client pointed at a Modal Endpoint or vLLM Server | Set the base URL to `https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1` for a vLLM worker, with the Runpod API key as the API key. | + +Job states are `IN_QUEUE`, `IN_PROGRESS`, `COMPLETED`, `FAILED`, `CANCELLED`, and `TIMED_OUT`. A job that isn't picked up or finished within its TTL (24 hours by default) is removed, so set `policy.ttl` and `policy.executionTimeout` on long jobs. See [Send API requests](/serverless/endpoints/send-requests) for rate limits and retry guidance. + +### Automation and CI + +Replace Modal CLI calls in CI with their Runpod counterparts from the [CLI command map](#cli-command-map), and replace the `MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET` secrets with `RUNPOD_API_KEY`. Use `rp flash dev main.py --once` as the smoke test that `modal run` provided. For infrastructure you manage outside the SDK, use [REST API v2](/api-reference-v2/overview) or the [Terraform provider](/integrations/terraform/overview), which is built on REST API v2. + +## Step 8: Verify, cut over, and shut down + +1. **Send a real request from outside.** Call the deployed endpoint from the network your clients use, with a production-shaped payload, and check the health of a queue endpoint: + + ```bash + curl -s https://api.runpod.ai/v2/ENDPOINT_ID/health \ + -H "Authorization: Bearer YOUR_API_KEY" + ``` + +2. **Compare outputs and latency.** Run the same inputs through Modal and Runpod. Compare results, then measure a cold request (no workers running) and a warm request separately. The endpoint's **Metrics** tab shows delay time, execution time, and cold start time. +3. **Run both in parallel.** Keep the Modal App deployed. Send a small share of traffic to Runpod from your client or gateway, and watch error rates, `429` responses, and queue delay. +4. **Shift traffic.** Raise `workers` to cover peak load before you move the rest. Check that the maximum fits your account's worker limit. +5. **Set cost guards on Runpod.** Set `execution_timeout_ms` on every queue and API resource, keep `workers` maximums deliberate, and remember that a minimum above 0 bills continuously. Review spend on the [Billing page](/accounts-billing/billing). +6. **Stop billing on Modal.** After the parallel run, stop each App, then delete storage and Secrets you've already copied: + + ```bash + modal app list + modal app stop APP_NAME --yes + modal endpoint list + modal endpoint stop ENDPOINT_NAME + + modal volume delete VOLUME_NAME --yes + modal dict delete DICT_NAME --yes + modal queue delete QUEUE_NAME --yes + modal secret delete SECRET_NAME --yes + modal container list + ``` + + `modal app stop` is permanent, so scheduled Functions in the App stop running too. Deleting a Volume deletes its data, so verify the copy on Runpod first. Check `modal endpoint stop --help` for its arguments, and repeat for each Environment with `--env`. Then rotate or delete the Modal tokens and Proxy Tokens your callers used. + + + Undeploying a Runpod app doesn't delete the network volumes it created, and they keep billing. If you roll back to Modal, delete volumes you no longer need from the [Storage page](https://console.runpod.io/user/storage) or with `runpodctl network-volume delete`. + + +## Reference + +### CLI command map + +| Modal CLI | Runpod | Notes | +| --- | --- | --- | +| `modal setup`, `modal token new` | `rp login` | Opens a browser. `rp login --api-key YOUR_API_KEY` skips it. | +| `modal token set --token-id ... --token-secret ...` | `rp login --api-key YOUR_API_KEY`, or `export RUNPOD_API_KEY=...` | | +| `modal profile activate NAME` | `rp login --profile NAME` | | +| `modal run app.py` | `rp flash dev app.py --once` | Runs the local entrypoint once and cleans up. | +| `modal serve app.py` | `rp flash dev app.py` | Live session with temporary endpoints. There's no local HTTP server or dev URL. Call routes from the entrypoint. | +| `modal deploy app.py` | `rp flash deploy` | Deploys every app in the directory. | +| `modal deploy --env NAME` | `rp flash deploy --env NAME` | | +| `modal app list` | `rp flash app list`, `runpodctl serverless list` | | +| `modal app info NAME` | `rp flash app get NAME`, `rp flash env get ENV_NAME` | | +| `modal app stop NAME` | `rp flash undeploy --app NAME`, `rp flash app delete NAME` | `undeploy` removes one environment. `app delete` removes all of them. | +| `modal app rollback NAME` | Redeploy the earlier commit with `rp flash deploy` | | +| `modal app logs NAME` | Endpoint logs in the console, `rp logs POD_ID` for task Pods | Dev sessions stream worker output to the terminal. | +| `modal environment create NAME` | `rp flash env add NAME` | | +| `modal environment list` | `rp flash env list --app NAME` | | +| `modal volume create NAME` | `runpodctl network-volume create --name NAME --size GB --data-center-id DC` | | +| `modal volume list` | `runpodctl network-volume list` | | +| `modal volume put NAME LOCAL REMOTE` | `aws s3 cp LOCAL s3://VOLUME_ID/REMOTE --endpoint-url ... --region DC` | Needs an S3 API key. Add `--recursive` for a directory. | +| `modal volume get NAME REMOTE LOCAL` | `aws s3 cp s3://VOLUME_ID/REMOTE LOCAL --endpoint-url ... --region DC` | | +| `modal volume ls NAME` | `aws s3 ls s3://VOLUME_ID/ --endpoint-url ... --region DC` | | +| `modal volume delete NAME` | `runpodctl network-volume delete VOLUME_ID` | | +| `modal secret create NAME KEY=VALUE` | `rp secret add NAME` | One value per secret. | +| `modal secret list` | `rp secret list` | | +| `modal secret delete NAME` | `rp secret delete NAME` | | +| `modal shell` | `rp pod create`, then `rp ssh POD_ID` | Or `runpodctl pod create` and `runpodctl ssh info POD_ID`. | +| `modal container list` | `runpodctl serverless get ENDPOINT_ID --include-workers`, `runpodctl pod list` | | +| `modal endpoint create --model ...` | `runpodctl serverless create --hub-id ...` for a vLLM worker | Find the Hub ID with `runpodctl hub search vllm`. | +| `modal billing report` | `runpodctl billing serverless`, `runpodctl billing pods`, or `/v2/billing` in REST API v2 | | + +### SDK call map + +| Modal SDK | Runpod | REST route behind it | +| --- | --- | --- | +| `fn.remote(x)` | `fn.remote(x)`, `endpoint.run_sync({...})` | `POST https://api.runpod.ai/v2/ENDPOINT_ID/runsync` | +| `fn.spawn(x)` | `fn.spawn(x)`, `endpoint.run({...})` | `POST https://api.runpod.ai/v2/ENDPOINT_ID/run` | +| `call.get(timeout=0)` | `job.status()`, `run_request.status()` | `GET https://api.runpod.ai/v2/ENDPOINT_ID/status/JOB_ID` | +| `call.cancel()` | `job.cancel()` | `POST https://api.runpod.ai/v2/ENDPOINT_ID/cancel/JOB_ID` | +| `fn.remote_gen(x)` | `fn.stream(x)` | `GET https://api.runpod.ai/v2/ENDPOINT_ID/stream/JOB_ID` | +| `fn.get_current_stats()` | `endpoint.health()` | `GET https://api.runpod.ai/v2/ENDPOINT_ID/health` | +| `fn.update_autoscaler(...)` | `runpodctl serverless update ENDPOINT_ID --workers-min ... --workers-max ...` | `PATCH https://api.runpod.io/v2/serverless/ENDPOINT_ID` | +| `modal.Volume.from_name(...)` | `runpod.NetworkVolume(...)` | `POST https://api.runpod.io/v2/network-volumes` | +| `modal.Secret.from_name(...)` | `runpod.Secret(...)` | `POST https://api.runpod.io/v2/account/secrets` | +| `app.deploy()` | `rp flash deploy` | | + +### Configuration field map + +| Modal decorator argument | Runpod SDK argument | `runpodctl serverless create` flag | REST v2 field | +| --- | --- | --- | --- | +| `gpu` | `gpu` | `--gpu-id` | `gpu.pools`, `gpu.excludedTypes` | +| `gpu="TYPE:N"` | `gpu_count` | `--gpu-count` | `gpu.count` | +| `cpu`, `memory` | `cpu` | `--compute-type CPU --instance-id` | `cpu[].id`, `cpu[].vcpuCount` | +| `ephemeral_disk` | `container_disk_gb` | `--container-disk-in-gb` on the template | `disk` | +| `min_containers` | `workers=(min, max)` | `--workers-min` | `workers.min` | +| `max_containers` | `workers=(min, max)` | `--workers-max` | `workers.max` | +| `scaledown_window` | `idle_timeout` | `--idle-timeout` | `workers.idleTimeout` | +| `timeout` | `execution_timeout_ms` | `--execution-timeout` (seconds) | `timeout` (milliseconds) | +| `@modal.concurrent(max_inputs=)` | `max_concurrency` | Set in the handler (`concurrency_modifier`) | | +| `region` | `datacenter` | `--data-center-ids` | `dataCenterIds` | +| `volumes` | `mounts` | `--network-volume-id` | `mounts.network`, `mounts.global` | +| `secrets` | `env` with `runpod.Secret` | `--env` on the template | `env` | +| `image` | `image`, `dependencies`, `system_dependencies` | `--image` on the template | `image` | +| `enable_memory_snapshot` | `flashboot` | `--flash-boot` | `flashboot` | +| `name` | `name` | `--name` | `name` | + +### Environment variable map + +| Modal | Runpod | Where | +| --- | --- | --- | +| `MODAL_TOKEN_ID`, `MODAL_TOKEN_SECRET` | `RUNPOD_API_KEY` | Your machine and CI. | +| `~/.modal.toml` | `~/.runpod/config.toml` | Your machine. | +| `MODAL_ENVIRONMENT` | `--env` on `rp flash deploy`, or `env=` on `runpod.App` | Your machine and CI. | +| `MODAL_IS_REMOTE` | `runpod.is_local()` | In code. | +| `MODAL_REGION` | `RUNPOD_DC_ID` | Inside Pods, including task Pods. | +| `MODAL_TASK_ID` | `RUNPOD_POD_ID` | Inside Pods, including task Pods. | +| No counterpart | `RUNPOD_ENDPOINT_ID` | Inside Serverless workers. | +| No counterpart | `PORT`, `PORT_HEALTH`, `HEALTH_CHECK_PATH` | Inside load-balancing workers. | +| No counterpart | `RUNPOD_INIT_TIMEOUT` | Endpoint setting for how long a worker may take to initialize (default 420 seconds). | + +## Migrate with a coding agent + +A coding agent with the Runpod skills and MCP server can do the inventory, translate the code, and deploy with your approval. Install both first. + + + + The plugin also connects the hosted MCP server: + + ```bash + claude plugin marketplace add runpod/runpod-plugins-official + claude plugin install runpod@runpod + ``` + + + + Install the skills, then the hosted MCP server: + + ```bash + npx skills add runpod/runpod-plugins-official --skill '*' --agent AGENT_ID --global + npx @runpod/mcp-server@latest add --client CLIENT_ID + ``` + + + +The MCP server signs in with OAuth (**Sign in with Runpod**), so you don't need an API key to start. See [Agent skills](/get-started/agent-skills) and [MCP servers](/get-started/mcp-servers) for agent and client IDs. + +Then give the agent this prompt from the root of your repository: + +```text +Use the runpod-cloud-migrate skill to migrate this project from Modal to Runpod. + +1. Inventory first. Find every modal.App, @app.function, @app.cls, @app.server, web + endpoint decorator, schedule, Volume, Secret, Dict, Queue, Sandbox, and + CloudBucketMount in the code, and every caller that uses Function.from_name, + Cls.from_name, or a modal.run URL. If the modal CLI is authenticated, also run + modal app list, modal volume list, and modal secret list. Treat old names + (Stub, keep_warm, concurrency_limit, container_idle_timeout, + allow_concurrent_inputs, web_endpoint) as their current equivalents. +2. Show me a plan as a table: each Modal resource, its Runpod target (@app.queue, + @app.task, @app.api, a load-balancing endpoint, a custom handler image, a Pod), + the GPU mapping, the storage and secret mapping, and every feature with no direct + equivalent (schedules, Dict, Queue, map, batched, retries, proxy auth, custom + domains, sandboxes) with the workaround you propose. +3. Include a cost estimate for the Runpod side, with worker counts and idle + timeouts, and check live GPU stock in the data center you propose. +4. Wait for my approval before you create anything billable on Runpod (endpoints, + Pods, volumes) or create any secret. I'll enter secret values myself. +5. Migrate one workload first. Write the Runpod code next to the Modal code without + deleting it, run rp flash dev FILE --once, then deploy it. +6. Verify with a real request to the deployed endpoint and show me the request, the + response, and the cold and warm latency. +7. Never stop, delete, or change any Modal resource (apps, volumes, secrets) without + my explicit approval. +``` + +## Troubleshooting + +**`arguments must be json-serializable`, or a JSON `TypeError` when calling a function.** The function takes or returns a Python object that Modal pickled, such as a NumPy array, a DataFrame, or bytes. Convert to lists, base64 strings, or a URL to a file on a volume or in object storage. + +**A job is killed almost immediately.** A Modal `timeout` in seconds was copied into `execution_timeout_ms`. Multiply by 1,000. + +**`gpu and cpu are mutually exclusive`.** A Modal Function that set both `gpu=` and `cpu=` or `memory=` was translated argument by argument. Remove `cpu` from GPU resources. + +**`ImportError` when you start a dev session or deploy.** The SDK imports your module locally to discover resources, so a top-level `import torch` needs torch on your machine. Move the import into the function body. If you used `with image.imports():` on Modal, move those imports too. + +**A sibling module imports under `rp flash deploy` but not under `rp flash dev`.** Dev sessions send only the app's file. Inline the helper or install it as a package. + +**Reading `volume.path` raises an error, or files are missing.** The path was read at module level, where the volume isn't mounted, or a queue or API resource mounted the volume somewhere other than `/runpod-volume`. Read `.path` inside the function and change paths that pointed at the old Modal mount point. + +**Workers don't start, or the endpoint is throttled.** The network volume pins the endpoint to one data center, and the GPU you chose is scarce there. Allow more GPU types with a list, or move the volume to a data center with stock for the GPU you need. + +**`rp flash deploy` fails on a resource name, but the dev session worked.** Queue and API names need at least 3 characters, and dev sessions don't check. Rename the function or class, or pass `name=`. + +**The build artifact is larger than 1,500 MB.** Leave out packages the worker image already has with `--exclude`, add data files to `.runpodignore`, or move to a custom image. + +**Callers get `401`.** Clients that called a public `modal.run` URL send no credentials. Add `Authorization: Bearer` with a Runpod API key. If the SDK returns `401` while `rp login` succeeded, a stale `RUNPOD_API_KEY` in your shell is taking precedence. Unset it. + +**A load-balancing endpoint returns `no workers available` or `502`.** The first worker is still starting, or the server isn't listening where the load balancer checks. Retry with backoff during cold starts, serve `/ping`, and set `PORT` and `PORT_HEALTH` to the port your server uses. + +**`/status/JOB_ID` returns `404` for a job that finished.** Results are kept for 30 minutes after an async job. Fetch sooner, use a webhook, or write results to storage. + +**An endpoint that worked last week queues jobs without starting workers.** It received no requests for several days, so Runpod lowered its max workers. Raise max workers in the console or with `runpodctl serverless update ENDPOINT_ID --workers-max 3`. + +**`.remote()` on a queue function raises `TimeoutError` on a long job.** Depending on your SDK version, a blocking call on a queue resource can stop waiting after a few minutes while the job keeps running. Use `job = fn.spawn(...)` and `job.result(timeout=...)` for long jobs, or move the work to `@app.task`. + +If you find a gap or an error in this guide, open an issue in the [Runpod docs repository](https://github.com/runpod/docs/issues). + +## Next steps + + + + Apps, queue resources, tasks, API resources, and call styles. + + + + Every argument on `@app.queue`, `@app.api`, and `@app.task`. + + + + Serve your own HTTP server with custom paths. + + + + Persistent storage for models, datasets, and checkpoints. + + diff --git a/migrate/overview.mdx b/migrate/overview.mdx new file mode 100644 index 000000000..f1349623a --- /dev/null +++ b/migrate/overview.mdx @@ -0,0 +1,194 @@ +--- +title: "Migrate to Runpod" +sidebarTitle: "Overview" +description: "Move GPU workloads to Runpod from another cloud or AI platform, with a step-by-step guide for each one and a coding agent skill that does the work." +--- + +These guides take a workload that runs on another platform and land it on Runpod. Each one maps the other platform's concepts to Runpod's, lists what works differently, and walks through the move with before and after code, CLI commands, and API calls. Every guide was written against the other platform's current documentation. + + +Want an agent to do it? Install the [Runpod skills](/get-started/agent-skills) and ask your coding agent to migrate the project. See [Migrate with a coding agent](#migrate-with-a-coding-agent). + + +## Choose your guide + +### GPU clouds + + + + Instances, templates, and Vast Serverless to Pods and Serverless. + + + On-demand instances, filesystems, and 1-Click Clusters. + + + Kubernetes workloads, Slurm, and object storage. + + + Machines, Notebooks, Deployments, and GPU Droplets. + + + +### Serverless and inference platforms + + + + Modal apps, functions, classes, and volumes to the Runpod SDK. + + + Hosted models, Cog models, and deployments. + + + Truss models, Model APIs, and autoscaling settings. + + + Model APIs, the queue API, and fal Serverless apps. + + + Serverless inference, dedicated endpoints, and fine-tuned models. + + + Serverless inference, on-demand deployments, and custom models. + + + Inference Endpoints, custom handlers, and GPU Spaces. + + + +### Hyperscalers + + + + EC2 GPU instances, SageMaker, and Bedrock. + + + Compute Engine, Vertex AI, and Cloud Run. + + + GPU VMs, Azure Machine Learning, and Azure OpenAI. + + + +The hyperscaler guides include a Terraform path that uses the [Runpod Terraform provider](/integrations/terraform/overview), so you can run both providers in one configuration while you move. + +### Frameworks and API versions + + + + Run an existing Cog image on Runpod Serverless. + + + Point OpenAI client code at a vLLM endpoint on Runpod. + + + Move existing Runpod integrations to REST API v2. + + + Move a `runpod-flash` project to the Runpod SDK. + + + +If your platform isn't listed, the process below still applies. Match each workload to a Runpod product by its shape, then follow that product's docs. + +## Where workloads land + +The other platform's product name matters less than what the workload does. Sort each one by its shape. + +| Workload | Runpod product | Start here | +| --- | --- | --- | +| Calls a model someone else hosts. | [Public Endpoints](/public-endpoints/overview), when Runpod hosts that model. | [Quickstart](/public-endpoints/quickstart) | +| Takes a request, returns a response, and sits idle in between. | [Serverless](/serverless/overview) queue-based endpoint. | [Serverless quickstart](/serverless/quickstart) | +| Runs your own HTTP server with custom routes and low latency. | Serverless [load-balancing endpoint](/serverless/load-balancing/overview). | [Load balancing overview](/serverless/load-balancing/overview) | +| Serves an LLM behind an OpenAI-compatible API. | Serverless endpoint running the [vLLM worker](/serverless/vllm/overview). | [vLLM quickstart](/serverless/vllm/get-started) | +| Python functions you'd rather not containerize yourself. | The [Runpod SDK](/sdk/overview). | [SDK quickstart](/sdk/quickstart) | +| A job that runs to completion, such as training or a batch run. | A [Pod](/pods/overview), or one Pod per call from the Runpod SDK. | [Choose a Pod](/pods/choose-a-pod) | +| An interactive machine for notebooks, SSH, or an IDE. | A Pod. | [Connect to a Pod](/pods/connect-to-a-pod) | +| An always-on service with a stable address. | A Pod. | [Expose ports](/pods/configuration/expose-ports) | +| Multi-node training or HPC. | [Instant Clusters](/instant-clusters). | [Slurm clusters](/instant-clusters/slurm-clusters) | + +## How a migration goes + +Every guide follows the same eight steps, so you can run them in order or jump to the one you need. + + + + List what runs on the other platform and find the code that calls it. Consoles often hold resources that no file in your repo mentions. + + + Create an account, get an [API key](/get-started/credentials), and install the tools you'll use: [runpodctl](/runpodctl/overview), the [Runpod SDK](/sdk/overview), the [MCP server](/get-started/mcp-servers), or [Terraform](/integrations/terraform/overview). + + + Use the table above, then pick a GPU that fits the model. GPU stock varies by data center, so check what's available before you commit to a type. + + + Most container images run on Runpod unchanged if they're built for `linux/amd64`. Platform-specific contracts, such as a handler signature or a health route, need a small port. + + + Copy datasets and checkpoints to a [network volume](/storage/network-volumes). For models on Hugging Face, [cached models](/serverless/endpoints/model-caching) skip the copy. + + + Create the Pod or endpoint from the console, runpodctl, the REST API, the SDK, or Terraform. + + + Change the URL, the auth header, and the request shape in the code that calls the workload. Put the new values behind configuration so you can switch back. + + + Send real requests to both deployments and compare. Shift traffic, then stop what's still billing on the other platform. + + + +## Migrate with a coding agent + +The Runpod plugin for coding agents includes a migration skill, `runpod-cloud-migrate`. It scans your repository for the platforms it depends on, opens the matching playbook, and proposes a plan before it changes anything. It migrates one workload at a time and verifies each with a real request. + + + + + + The plugin installs the skills and connects the hosted MCP server: + + ```bash + claude plugin marketplace add runpod/runpod-plugins-official + claude plugin install runpod@runpod + ``` + + + Install the skills for your agent, then connect the hosted MCP server: + + ```bash + npx skills add runpod/runpod-plugins-official --skill '*' --agent AGENT_ID --global + npx @runpod/mcp-server@latest add --client CLIENT_ID + ``` + + Replace `AGENT_ID` and `CLIENT_ID` with your agent's ID, for example `cursor` or `codex`. + + + + The MCP server signs you in with your Runpod account, so you don't need an API key to start. See [Agent skills](/get-started/agent-skills) and [MCP servers](/get-started/mcp-servers) for details. + + + Open your project and paste this prompt. In Claude Code, you can run `/runpod:migrate-cloud` instead. + + ```text + Use the runpod-cloud-migrate skill to move this project to Runpod. + + 1. Inventory the repository first and show me which platforms it uses and where. + 2. For each workload, propose where it lands on Runpod, what changes in the code, + what data has to move, and anything that has no direct equivalent. + 3. Show me the plan and an hourly cost estimate. Wait for my approval before you + create anything on Runpod. + 4. Migrate one workload at a time, starting with the smallest. Verify each one with + a real request and show me the result. + 5. Keep the current deployment callable behind a config switch so I can roll back. + 6. Don't stop, change, or delete anything on the platform we're leaving unless I + name the resource and approve it. + ``` + + + The agent stops after the inventory and plan. Check the targets, the GPU choice, and the list of gaps, then approve it. Nothing billable is created before that point. + + + +## Get help + +If a guide is wrong or a step fails, [open an issue on GitHub](https://github.com/runpod/docs/issues). For large or committed workloads, [talk to the Runpod sales team](https://ecykq.share.hsforms.com/2MZdZATC3Rb62Dgci7knjbA) about capacity planning and pricing. diff --git a/migrate/paperspace.mdx b/migrate/paperspace.mdx new file mode 100644 index 000000000..6ea4a7fb1 --- /dev/null +++ b/migrate/paperspace.mdx @@ -0,0 +1,1333 @@ +--- +title: "Migrate from Paperspace and DigitalOcean GPU to Runpod" +sidebarTitle: "Paperspace and DigitalOcean" +description: "Move Paperspace Machines, Notebooks, Deployments, and Workflows, and DigitalOcean GPU Droplets and Inference, to Runpod." +--- + +This guide is for teams that run GPU workloads on [Paperspace](https://docs.digitalocean.com/products/paperspace/) or on DigitalOcean's own GPU products and want to run them on Runpod. Paperspace [Machines](https://docs.digitalocean.com/products/paperspace/machines/) and DigitalOcean [GPU Droplets](https://docs.digitalocean.com/products/gpu-droplets/) map to [Pods](/pods/overview), [Notebooks](https://docs.digitalocean.com/products/paperspace/notebooks/) map to Pods with JupyterLab, [Deployments](https://docs.digitalocean.com/products/paperspace/deployments/) map to [Serverless](/serverless/overview) endpoints, [Workflows](https://docs.digitalocean.com/products/paperspace/workflows/) map to the Runpod SDK or Serverless queue endpoints, and DigitalOcean [Inference](https://docs.digitalocean.com/products/inference/) maps to [Public Endpoints](/public-endpoints/overview) or a Serverless vLLM endpoint. By the end you'll have each workload running on Runpod, your data on Runpod storage, your clients and automation pointed at Runpod, and the old resources shut down. + + +You can hand most of this migration to a coding agent. See [Migrate with a coding agent](#migrate-with-a-coding-agent). + + + +**Product names, checked against DigitalOcean's docs on 2026-11-14.** Paperspace is part of DigitalOcean and still offers Notebooks, Deployments, Workflows, and Machines through the Paperspace console. "Gradient" still names those Paperspace products and their subscription plans. DigitalOcean's own AI products are GPU Droplets, Bare Metal GPUs, and Inference (serverless inference, dedicated inference, and Agent Platform). DigitalOcean's docs list no retirement date for the four Paperspace products. They do state that the legacy Gradient and Core API endpoints are no longer available (so the `gradient` CLI no longer works), that Windows templates are unavailable to users who joined after July 1, 2024, and that NVIDIA GRID machines are retired. + + +## How Paperspace and DigitalOcean map to Runpod + +| Paperspace or DigitalOcean concept | Runpod equivalent | Notes | +| --- | --- | --- | +| Paperspace Machine (Linux VM) | [Pod](/pods/overview) | A Pod is a container, not a VM. You bring a Docker image instead of an OS template. | +| DigitalOcean GPU Droplet (VM) | [Pod](/pods/overview) | Same shift from VM to container. 8-GPU plans map to a Pod with a GPU count of 8, subject to stock. | +| Paperspace Machine (Windows), desktop streaming | None | Pods run Linux containers. See [What has no direct equivalent](#what-has-no-direct-equivalent). | +| Paperspace Notebook | Pod from the official PyTorch template | JupyterLab on port 8888. Persistent files live in `/workspace`. | +| Paperspace Deployment (container, replicas, endpoint URL) | [Load-balancing endpoint](/serverless/load-balancing/overview) or [queue-based endpoint](/serverless/overview) | Load balancing keeps your HTTP server as is. Queue-based needs a handler function. | +| Paperspace Workflow (YAML jobs on Argo) | Runpod SDK [`@app.task`](/sdk/overview) or a queue-based endpoint | Runpod has no YAML pipeline engine. You express job order in Python or in your CI system. | +| DigitalOcean serverless inference | [Public Endpoints](/public-endpoints/overview) or [Serverless vLLM](/serverless/vllm/overview) | Both expose an OpenAI-compatible route. The model catalog differs. | +| DigitalOcean dedicated inference, 1-Click Models | [Serverless vLLM](/serverless/vllm/overview) or a Pod | You pick the Hugging Face model and the GPU. | +| DigitalOcean Bare Metal GPUs, multi-node contract GPU Droplets | [Instant Clusters](/instant-clusters) | Multi-node clusters for Slurm, PyTorch, and Ray. | +| DOKS GPU node pools | Serverless, Pods, or Instant Clusters | Runpod exposes no Kubernetes API. See [What has no direct equivalent](#what-has-no-direct-equivalent). | +| OS templates, custom templates, snapshots, Droplet images | Docker image plus a [template](/pods/templates/overview) | A template stores the image, ports, env, and disk sizes. | +| Notebook container, Deployment `image` | Docker image | Images must be `linux/amd64`. | +| Machine block storage, Droplet boot disk | Volume disk at `/workspace` or a [network volume](/storage/network-volumes) | Only the mounted path persists. The rest of the container filesystem resets when the Pod stops. | +| Droplet scratch disk | Container disk | Both are temporary. | +| Paperspace shared drive, Notebook `/storage`, DigitalOcean Network File Storage | [Network volume](/storage/network-volumes) | Shared by Pods and Serverless workers in one data center. | +| DigitalOcean Volumes (block storage) | [Network volume](/storage/network-volumes) | A file system mount, not a raw block device. | +| Paperspace datasets (versioned), DigitalOcean Spaces | Network volume with the [S3-compatible API](/storage/s3-api), or a [global volume](/storage/globalvolume/overview) | Runpod has no dataset versioning or bucket service. You can also keep reading from Spaces. | +| Public IP (static or dynamic), Droplet public IPv4 | HTTP proxy URL or a mapped TCP port | Runpod assigns the public IP and external port. See [Expose ports](/pods/configuration/expose-ports). | +| Paperspace private network, DigitalOcean VPC | [Global networking](/pods/networking) | Private Pod-to-Pod DNS at `POD_ID.runpod.internal`. | +| Paperspace secrets (`secret:NAME`) | [Secrets](/pods/templates/secrets) | Referenced as `{{ RUNPOD_SECRET_name }}` in environment variables. | +| Paperspace regions (NY2, CA1, AMS1), DigitalOcean regions | Data centers | List them with `runpodctl datacenter list`. | +| Paperspace hourly billing | Per-second billing | Pods bill per second for compute and storage. Serverless bills per second while workers run. | +| DigitalOcean GPU Droplet per-second billing | Per-second billing | A stopped Pod stops compute billing and keeps billing its volume disk. | +| Spot GPU Droplets, contract GPU Droplets | On-demand Pods | `runpodctl`, REST API v2, and the Terraform provider create on-demand Pods. | +| `digitalocean` Terraform provider | [Runpod Terraform provider](/integrations/terraform/overview) | See [Migrate your Terraform](#migrate-your-terraform). | +| `pspace` CLI, `doctl` | [`runpodctl`](/runpodctl/overview) | See the [CLI command map](#cli-command-map). | +| Paperspace API (`api.paperspace.com/v1`), DigitalOcean API (`api.digitalocean.com/v2`) | [REST API v2](/api-reference-v2/overview) (`api.runpod.io/v2`) | See the [API map](#api-map). | +| PyDo, Godo, DoTs | Runpod SDK (`pip install runpod`) | Includes the `rp` CLI and `runpod.Endpoint`. | +| Paperspace API key, DigitalOcean personal access token, model access key | One [Runpod API key](/get-started/credentials) | Sent as `Authorization: Bearer`. | +| Paperspace teams and projects, DigitalOcean teams and projects | [Organizations](/organizations/orgs-overview), [cost centers](/accounts-billing/cost-centers) | Roles and groups live on the organization. | +| Deployment logs and metrics, DigitalOcean Monitoring | [Serverless logs](/serverless/development/logs), `runpodctl pod logs`, REST v2 metrics routes | Endpoint metrics are at `/v2/serverless/ENDPOINT_ID/metrics/requests`. | + +## What's different + +### What works differently + +**VMs become containers.** A Paperspace Machine or a GPU Droplet is a virtual machine: you pick an OS image, get a boot disk that persists everything, and can install drivers, run `systemd` services, or run Docker yourself. A Pod is a single container that Runpod starts from your Docker image, with the GPU drivers provided by the host. Anything you installed by hand on the VM has to move into a Dockerfile. Runpod runs Docker for you, so you can't start your own Docker daemon or use Docker Compose inside a Pod. + +**Only `/workspace` persists.** On a VM the whole boot disk survives a shutdown. On a Pod, the container disk resets when the Pod stops, the volume disk at `/workspace` survives a stop and is deleted with the Pod, and a network volume outlives the Pod. Keep code, data, and checkpoints under the mounted path, and install system packages in the image. + +**Stopping and billing.** Paperspace bills machine compute only while the machine is powered on, and keeps billing storage and public IPs. DigitalOcean keeps billing a powered-off GPU Droplet until you destroy it. A stopped Pod stops billing compute and keeps billing its volume disk. A Serverless endpoint with zero active workers bills only while workers run. See [Pod pricing](/pods/pricing) and [Serverless pricing](/serverless/pricing). + +**Public access.** A Machine or Droplet gets a public IP address where every port you open is reachable. A Pod exposes only the ports you list. HTTP ports get a URL of the form `https://POD_ID-PORT.proxy.runpod.net`, which is public, has no built-in authentication, and closes connections after 100 seconds. TCP ports (such as SSH) get a public IP and a mapped external port that Runpod assigns. Pods support TCP and HTTP, not UDP. + +**Deployment endpoints.** A Paperspace Deployment serves your container at a `paperspacegradient.com` URL, optionally protected with basic authentication (username `paperspace`). A Runpod load-balancing endpoint serves your container at `https://ENDPOINT_ID.api.runpod.ai` and always requires `Authorization: Bearer YOUR_API_KEY`. A queue-based endpoint uses a different contract: you write a handler function and clients call `/run` or `/runsync` with an `{"input": {...}}` body. + +**Autoscaling signals.** Paperspace Deployments scale between `replicas` and `maxReplicas` on CPU, memory, or request duration, using the Kubernetes horizontal pod autoscaler. Runpod endpoints scale between a minimum and maximum worker count on request count (load balancing and queue-based) or queue delay (queue-based only). The minimum is 0 by default, so an endpoint scales to zero and the first request after idle pays a cold start. + +**Health checks.** Paperspace supports `liveness`, `readiness`, and `startup` probes with any 2xx or 3xx code as success. A load-balancing worker has one health path (`/ping` by default, configurable with `HEALTH_CHECK_PATH`) where `200` means healthy, `204` means initializing, and anything else means unhealthy. + +**Model APIs.** DigitalOcean serverless inference authenticates with a model access key against `https://inference.do-ai.run/v1` and includes commercial third-party models. Runpod Public Endpoints and vLLM endpoints authenticate with your Runpod API key and include the endpoint or model slug in the base URL. + +**One credential.** Paperspace and DigitalOcean use separate credentials (a Paperspace API key, a DigitalOcean personal access token, model access keys, and Spaces keys). Runpod uses one API key for the CLI, REST API, SDK, and endpoint requests, plus a separate S3 API key for the S3-compatible API. + +### What has no direct equivalent + +**On Runpod:** + +- **Windows machines and desktop streaming.** Pods run Linux containers. Keep Windows workloads where they are, or port them to Linux. For interactive work, use JupyterLab, SSH, or [an IDE connection](/pods/configuration/connect-to-ide). +- **A Kubernetes API.** DOKS GPU node pools have no Runpod counterpart you can point `kubectl` at. Move each workload to the product that fits its shape: a Deployment with a Service to a Serverless endpoint, a Job to `@app.task` or a queue-based endpoint, a StatefulSet or long-lived process to a Pod, and tightly coupled multi-node training to an [Instant Cluster](/instant-clusters). To keep a cluster-style scheduler, use the [SkyPilot](/integrations/skypilot) or [dstack](/integrations/dstack) integration. +- **VM snapshots and custom OS templates.** Rebuild the environment as a Docker image and store its settings in a template. +- **A workflow engine.** Gradient Workflows runs YAML job graphs with typed inputs and outputs. On Runpod, job order lives in Python (the SDK) or in your CI system. +- **Versioned datasets.** Paperspace datasets have immutable versions and tags. On a network volume, use versioned directory names, or keep datasets in an object store that supports versioning. +- **Object storage buckets with a CDN.** The S3-compatible API reads and writes files on a network volume. It isn't a bucket service with public URLs or a CDN. Keep Spaces for public assets. +- **Managed agents, knowledge bases, and commercial model routing.** Agent Platform, Knowledge Bases, the Inference Router, and third-party commercial models stay on DigitalOcean or move to another provider. +- **Raw block devices.** Network volumes mount as a file system. You can't partition or format them. +- **Multiple probe types.** Fold readiness and startup logic into the single health path. + +**On Paperspace and DigitalOcean:** + +- **Scale to zero for your own containers.** Paperspace `replicas` is the floor for a running Deployment, and you stop it by setting `enabled: false`. +- **A managed request queue.** Queue-based endpoints give you async jobs, status polling, streaming, retries, and webhooks without running a queue yourself. +- **Storage that isn't tied to a region.** A [global volume](/storage/globalvolume/overview) (beta) mounts in any data center. +- **Ephemeral compute per function call.** `@app.task` provisions a Pod per call and terminates it when the function returns. + +## Requirements + +- A [Runpod account](/accounts-billing/manage-accounts) with credits, and a [Runpod API key](/get-started/credentials). +- Access to your Paperspace team (a Paperspace API key for `pspace`) or your DigitalOcean team (a personal access token for `doctl`), or both. +- Docker on your machine, and a container registry you can push to (Docker Hub, GHCR, or DigitalOcean Container Registry). +- An SSH key pair for Pod access and file transfer. +- For Terraform users: Terraform 1.6 or later. + +## Step 1: Take inventory of what you run on Paperspace and DigitalOcean + +List every resource, record its configuration, and find the code that calls either platform. Use the names from each product's API, because you'll map them field by field in later steps. + +### Paperspace + +Install `pspace` and log in with a Paperspace API key, then list the resources it manages. The `--json` flag is available on every command. + +```bash +curl -fsSL https://paperspace.com/install.sh | sh +pspace login + +pspace machine list --json > paperspace-machines.json +pspace deployment list --json > paperspace-deployments.json +pspace project list --json > paperspace-projects.json +pspace shared-drive list --json > paperspace-shared-drives.json +pspace snapshot list --json > paperspace-snapshots.json +pspace template list --json > paperspace-templates.json +pspace startup-script list --json > paperspace-startup-scripts.json +pspace public-ip list --json > paperspace-public-ips.json +pspace secret list --global --json > paperspace-team-secrets.json +``` + +For each machine, record `machineType`, `region`, `diskSize`, `templateId`, `publicIpType`, `networkId`, `startupScriptId`, and the auto-shutdown settings. Secret values aren't needed yet, only the names. + +Export the full configuration of each Deployment. The response includes the spec (image, port, env, command, instance type, replicas, autoscaling, health checks, and integrations) and the endpoint URL. + +```bash +pspace deployment get DEPLOYMENT_ID --json > deployment-DEPLOYMENT_ID.json +``` + +The same data is available from the Paperspace API at `https://api.paperspace.com/v1`, for example `GET /deployments`, `GET /machines`, `GET /shared-drives`, and `GET /datasets`, with an `Authorization: Bearer` header. + +`pspace` has no commands for Notebooks, Workflows, or datasets. For those, use the Paperspace console: + +- **Notebooks:** For each Notebook, record the container image, the workspace repository URL, the machine type, and what you keep in `/notebooks`, `/storage`, and `/datasets`. +- **Workflows:** Workflow specs live in your repository under `.gradient/workflows`. Record each job's `instance-type`, image, `inputs`, `outputs`, `env`, and `needs`. +- **Datasets and models:** Record each dataset ID and its storage provider (Gradient-managed or your own S3-compatible bucket). + +### DigitalOcean + +Install `doctl`, authenticate with a personal access token, and list GPU resources. `doctl compute droplet list` returns only non-GPU Droplets unless you pass `--gpus`. + +```bash +doctl auth init + +doctl compute droplet list --gpus --output json > do-gpu-droplets.json +doctl compute volume list --output json > do-volumes.json +doctl compute snapshot list --output json > do-snapshots.json +doctl kubernetes cluster list --output json > do-doks-clusters.json +doctl kubernetes cluster node-pool list CLUSTER_ID --output json > do-node-pools.json +``` + +For each GPU Droplet, record the size slug (for example `gpu-h100x1-80gb`), the image slug (`gpu-h100x1-base`, `gpu-h100x8-base`, `gpu-amd-base`, a 1-Click Model, or your own snapshot), region, attached volume IDs, VPC, tags, and the user data script. For DOKS, record which node pools use `gpu-` size slugs and which workloads schedule onto them. + +List dedicated inference deployments with the DigitalOcean API: + +```bash +curl -s -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \ + "https://api.digitalocean.com/v2/dedicated-inferences" +``` + +For serverless inference, list your model access keys in the DigitalOcean Control Panel and note which model IDs your applications request. For Spaces, list buckets with any S3 client pointed at your region's endpoint, for example `aws s3 ls --endpoint-url https://nyc3.digitaloceanspaces.com`. + +### Find code that calls either platform + +Search your repositories for CLI invocations, hostnames, credentials, and Terraform resources: + +```bash +grep -rnE \ + "pspace |gradient (deployments|workflows|notebooks|datasets)|api\.paperspace\.com|paperspacegradient\.com|PAPERSPACE_API_KEY|\.gradient/workflows|doctl |api\.digitalocean\.com|digitaloceanspaces\.com|inference\.do-ai\.run|dedicated-inference\.do-infra\.ai|DIGITALOCEAN_(ACCESS_)?TOKEN|MODEL_ACCESS_KEY|from pydo|digitalocean/godo|digitalocean_(droplet|volume|volume_attachment|spaces_bucket|kubernetes_cluster|kubernetes_node_pool)" \ + --include="*.py" --include="*.ts" --include="*.js" --include="*.go" --include="*.sh" \ + --include="*.yaml" --include="*.yml" --include="*.json" --include="*.toml" --include="*.tf" \ + --include="Dockerfile*" --include="Makefile" . +``` + +Also check for hardcoded paths that only exist on the old platform: `/notebooks`, `/storage`, `/datasets`, `/inputs`, `/outputs`, `/home/paperspace`, and Droplet volume mount points under `/mnt`. + +## Step 2: Set up Runpod + +Create a [Runpod account](https://www.console.runpod.io), add credits, and create an [API key](/get-started/credentials). Then install the tools you'll use. + +Install `runpodctl` and configure it. `runpodctl doctor` sets your API key and an SSH key in one pass. + +```bash +curl -fsSL https://cli.runpod.net | sudo bash +runpodctl doctor +``` + +On macOS you can use `brew install runpod/runpodctl/runpodctl` instead. + +Install the Runpod SDK if you'll migrate Workflows, write Serverless handlers, or call endpoints from Python. It requires Python 3.10 to 3.14 and includes the `rp` CLI. + +```bash +pip install runpod +rp login +export RUNPOD_API_KEY="YOUR_API_KEY" +``` + +`rp login` stores your key in `~/.runpod/config.toml`. The `RUNPOD_API_KEY` variable is what the REST examples and the Terraform provider in this guide read. + +To let a coding agent manage Runpod for you, connect the [Runpod MCP server](/get-started/mcp-servers). For infrastructure as code, install the [Terraform provider](/integrations/terraform/overview). + +## Step 3: Choose where each workload lands + +Pick a target for each item in your inventory by its shape, not by the product it runs on today. + +| Workload on Paperspace or DigitalOcean | Runpod target | Why | +| --- | --- | --- | +| Machine or GPU Droplet you SSH into for development or training | [Pod](/pods/overview) | Long-lived, interactive, SSH and ports. | +| Notebook | Pod from the official PyTorch template | JupyterLab with `/workspace` persistence. | +| Machine or Droplet that hosts a web UI or API all day | Pod, or a [load-balancing endpoint](/serverless/load-balancing/overview) if traffic is bursty | A Pod is always on. An endpoint scales with traffic. | +| Deployment serving HTTP (FastAPI, Flask, TGI, vLLM) | [Load-balancing endpoint](/serverless/load-balancing/overview) | Keeps your HTTP paths. Requests are limited to 5.5 minutes and 30 MB. | +| Deployment with long or bursty requests, or batch inference | [Queue-based endpoint](/serverless/overview) | Queues requests, retries, and supports async jobs and webhooks. | +| Workflow job (train, evaluate, export) | [`@app.task`](/sdk/overview) | One Pod per call, terminated on return. | +| Workflow that fans out many short jobs | Queue-based endpoint (`@app.queue`) | Autoscaling workers pull from a queue. | +| Serverless inference with an open model that Runpod hosts | [Public Endpoint](/public-endpoints/overview) | Nothing to deploy. | +| Serverless or dedicated inference with another open model, 1-Click Model | [Serverless vLLM](/serverless/vllm/get-started) | OpenAI-compatible, your choice of Hugging Face model. | +| DOKS GPU workload | Serverless endpoint, Pod, or `@app.task` by shape | See [What has no direct equivalent](#what-has-no-direct-equivalent). | +| Bare Metal GPUs, multi-node training | [Instant Cluster](/instant-clusters) | Multi-node with Slurm, PyTorch, or Ray. | +| Windows machine | Stays where it is | Pods are Linux containers. | + +### Map your GPUs + +Choose a GPU by what the workload needs (VRAM first, then generation), then confirm stock. The Runpod IDs below come from the [GPU types reference](/references/gpu-types). Pods take a GPU type ID. Serverless endpoints take a GPU pool ID. + +| Paperspace machine type | VRAM | Closest Runpod GPU type ID | Serverless pool | +| --- | --- | --- | --- | +| GPU+ (M4000), P4000, RTX4000 | 8 GB | `NVIDIA RTX A4000` (16 GB) | `AMPERE_16` | +| P5000, RTX5000, A4000 | 16 GB | `NVIDIA RTX A4000` | `AMPERE_16` | +| V100 | 16 GB | `Tesla V100-PCIE-16GB` or `NVIDIA RTX A4000` | `AMPERE_16` | +| P6000, A5000 | 24 GB | `NVIDIA RTX A5000` | `AMPERE_24` | +| V100-32G | 32 GB | `Tesla V100-SXM2-32GB` or `NVIDIA RTX 5000 Ada Generation` | `AMPERE_48` | +| A100 | 40 GB | `NVIDIA A40` (48 GB) or `NVIDIA RTX A6000` (48 GB) | `AMPERE_48` | +| A6000 | 48 GB | `NVIDIA RTX A6000` | `AMPERE_48` | +| A100-80G | 80 GB | `NVIDIA A100 80GB PCIe` or `NVIDIA A100-SXM4-80GB` | `AMPERE_80` | +| H100 | 80 GB | `NVIDIA H100 80GB HBM3` or `NVIDIA H100 PCIe` | `ADA_80_PRO` | + +| DigitalOcean GPU Droplet size slug | VRAM | Closest Runpod GPU type ID | Serverless pool | +| --- | --- | --- | --- | +| `gpu-4000adax1-20gb` | 20 GB | `NVIDIA RTX 4000 Ada Generation` | `AMPERE_16` | +| `gpu-6000adax1-48gb` | 48 GB | `NVIDIA RTX 6000 Ada Generation` | `ADA_48_PRO` | +| `gpu-l40sx1-48gb` | 48 GB | `NVIDIA L40S` | `ADA_48_PRO` | +| `gpu-h100x1-80gb`, `gpu-h100x8-640gb` | 80 GB each | `NVIDIA H100 80GB HBM3` | `ADA_80_PRO` | +| `gpu-h200x1-141gb`, `gpu-h200x8-1128gb` | 141 GB each | `NVIDIA H200` | `HOPPER_141` | +| `gpu-mi300x1-192gb`, `gpu-mi300x8-1536gb` | 192 GB each | `AMD Instinct MI300X OAM` | Check the catalog | +| B300 plans | 288 GB each | `NVIDIA B300 SXM6 AC` | Check the catalog | +| MI325X, MI350X, MI355X plans | 256 to 288 GB each | Check the catalog | Check the catalog | + +GPU stock varies by data center and changes through the day, so check what's available before you commit to a type: + +```bash +runpodctl gpu list +runpodctl datacenter list +``` + +For AMD GPU Droplets, check both that an AMD GPU type has stock and that your software stack (ROCm) has a matching image. The official PyTorch templates include one ROCm variant. If you move from AMD to NVIDIA GPUs, rebuild on a CUDA base image. + +## Step 4: Move your images and code + +Every Runpod target runs a `linux/amd64` Docker image. What you do here depends on whether the source is already a container. + +### Machines and GPU Droplets: turn the VM into an image + +Write a Dockerfile that reproduces what you installed on the VM. Start from a Runpod PyTorch image so you get CUDA, SSH, and JupyterLab conventions that the platform already understands. + +```dockerfile Dockerfile +FROM runpod/pytorch:1.0.2-cu1281-torch280-ubuntu2404 + +# System packages you installed with apt on the Machine or Droplet +RUN apt-get update && apt-get install -y --no-install-recommends \ + ffmpeg git-lfs rsync \ + && rm -rf /var/lib/apt/lists/* + +# Python packages from the VM environment (pip freeze > requirements.txt) +COPY requirements.txt /tmp/requirements.txt +RUN pip install --no-cache-dir --break-system-packages -r /tmp/requirements.txt + +# Application code. Data and checkpoints go on /workspace at runtime, not in the image. +WORKDIR /app +COPY . /app +``` + +The Ubuntu 24.04 base image marks the system Python as externally managed, which is why the install uses `--break-system-packages`. Build for `linux/amd64` and push to a registry Runpod can pull from: + +```bash +docker buildx build --platform linux/amd64 -t YOUR_REGISTRY/trainer:v1 --push . +``` + +If you build on an Apple silicon Mac, the `--platform` flag is required. Without it the image is `arm64` and the Pod fails to start. + +Translate the rest of the VM setup as follows: + +- **GPU drivers and CUDA.** Don't install NVIDIA drivers in the image. The host provides them. Your image supplies the CUDA user-space libraries, and you set a CUDA floor when you create the Pod (`--min-cuda-version 12.8`) so it lands on a host whose driver supports your image. +- **Startup scripts and user data.** A Paperspace startup script or Droplet `user_data` (cloud-init) becomes either Dockerfile steps (for installs) or the container start command (for launching your process). +- **Docker and Docker Compose on the VM.** If the Droplet ran `docker run --gpus all your-image`, that image is your Pod image, and the `docker run` flags become template fields: `-p` becomes `ports`, `-e` becomes `env`, `-v` becomes a volume mount, and the trailing command becomes the start command. For a Compose stack, run one Pod per service and connect them with [global networking](/pods/networking), or combine the services into one image with a process supervisor. +- **`systemd` services.** A container has one main process. Run your service in the foreground as the start command. +- **The `paperspace` user.** Paperspace Linux machines provision a `paperspace` user with a home directory at `/home/paperspace`. Runpod's SSH examples connect as `root`, and persistent data lives in `/workspace`. Update hardcoded paths. +- **ML-in-a-Box and the AI/ML-ready image.** Both are VM images with a preinstalled stack. The closest starting point is a `runpod/pytorch` image plus your own `requirements.txt`. + +For a private registry, save credentials once and reference them when you create Pods, templates, or endpoints: + +```bash +printenv REGISTRY_TOKEN | runpodctl registry create --name "my-registry" --username "my-user" --password-stdin +runpodctl registry list +``` + +### Notebooks: reuse the container or switch to the PyTorch template + +A Paperspace Notebook already runs in a container (for example `paperspace/gradient-base`), with your files in `/notebooks`. You have two options: + +- **Use the official PyTorch template** (`runpod-torch-v280`), which starts JupyterLab on port 8888 and mounts `/workspace`. Reinstall your Python packages from a `requirements.txt` kept in `/workspace`. This is the shortest path. +- **Build your own image** from `runpod/pytorch` with your packages baked in, as in the Dockerfile above, if you want the environment to survive Pod deletion. + +Either way, change notebook code that reads `/notebooks`, `/storage`, or `/datasets` to read from `/workspace`. + +### Deployments: keep the image, adjust the health check and port + +A Deployment image already serves HTTP, so it can run as a load-balancing worker with configuration changes only. The worker must do two things: listen on the port named by the `PORT` environment variable, and answer a health check. + +If your app already has a readiness route, point Runpod at it with `HEALTH_CHECK_PATH` instead of changing code. If it has none, add a `/ping` route. This FastAPI app is the Paperspace health check example adapted for Runpod: it returns `204` while the model loads and `200` when it's ready. + + + +```python Before: Paperspace Deployment +from fastapi import FastAPI, Response, status + +app = FastAPI() +model_loaded = False + + +@app.on_event("startup") +async def model_load(): + global model_loaded + # Download and load the model here + model_loaded = True + + +@app.get("/liveness/", status_code=200) +def liveness_check(): + return "Liveness check succeeded." + + +@app.get("/readiness/", status_code=200) +def readiness_check(response: Response): + if not model_loaded: + response.status_code = status.HTTP_503_SERVICE_UNAVAILABLE + return "Readiness check failed." + return "Readiness check succeeded." + + +@app.post("/predict/") +def predict(body: dict): + return {"prediction": f"result for {body['prompt']}"} +``` + +```python After: Runpod load-balancing worker +import os + +import uvicorn +from fastapi import FastAPI, Response + +app = FastAPI() +model_loaded = False + + +@app.on_event("startup") +async def model_load(): + global model_loaded + # Download and load the model here + model_loaded = True + + +@app.get("/ping") +def ping(): + # 200 = healthy, 204 = still initializing, anything else = unhealthy + if not model_loaded: + return Response(status_code=204) + return {"status": "healthy"} + + +@app.post("/predict/") +def predict(body: dict): + return {"prediction": f"result for {body['prompt']}"} + + +if __name__ == "__main__": + uvicorn.run(app, host="0.0.0.0", port=int(os.getenv("PORT", "80"))) +``` + + + +The request and response bodies don't change. What changes is the health route, the port source, and (in [Step 7](#step-7-update-the-code-that-calls-paperspace-and-digitalocean)) the URL and auth header your clients send. + +To use a queue-based endpoint instead, replace the HTTP server with a handler. Clients then send `{"input": {...}}` to `/run` or `/runsync` instead of calling your paths. + +```python handler.py +import runpod + +# Load the model once, outside the handler, so warm workers reuse it +MODEL = None + + +def load_model(): + global MODEL + if MODEL is None: + MODEL = lambda prompt: f"result for {prompt}" # Replace with your model load + return MODEL + + +def handler(job): + prompt = job["input"]["prompt"] + model = load_model() + return {"prediction": model(prompt)} + + +runpod.serverless.start({"handler": handler}) +``` + +The image for a queue-based worker needs `pip install runpod` and a start command of `python -u handler.py`. See [Handler functions](/serverless/workers/handler-functions) for streaming and async handlers. + +### Workflows: rewrite the job graph in Python + +A Workflow spec declares jobs, their order (`needs`), and data passed through `/inputs` and `/outputs`. With the Runpod SDK, each job becomes a function decorated with `@app.task`, the order becomes ordinary Python, and shared data moves through a network volume mounted on each task. + + + +```yaml Before: .gradient/workflows/train.yaml +defaults: + env: + HF_TOKEN: secret:hf_token + resources: + instance-type: A100-80G + +jobs: + train: + outputs: + model: + type: dataset + with: + ref: my-model + uses: container@v1 + with: + image: my-org/trainer:v1 + args: + - bash + - -c + - python train.py --steps 1000 --out /outputs/model + + evaluate: + needs: + - train + inputs: + model: train.outputs.model + uses: container@v1 + with: + image: my-org/trainer:v1 + args: + - bash + - -c + - python evaluate.py --model /inputs/model +``` + +```python After: main.py +import subprocess + +import runpod +from runpod import App, NetworkVolume, Secret + +app = App("train-pipeline") +artifacts = NetworkVolume("pipeline-artifacts", size=100) + + +@app.task( + gpu="A100", + image="my-org/trainer:v1", + mounts={"/artifacts": artifacts}, + env={"HF_TOKEN": Secret("hf_token")}, +) +def train(steps: int = 1000): + out_dir = artifacts.path / "model" + subprocess.run( + ["python", "train.py", "--steps", str(steps), "--out", str(out_dir)], + check=True, + ) + return {"model_dir": "model"} + + +@app.task( + gpu="A100", + image="my-org/trainer:v1", + mounts={"/artifacts": artifacts}, + env={"HF_TOKEN": Secret("hf_token")}, +) +def evaluate(model_dir: str): + subprocess.run( + ["python", "evaluate.py", "--model", str(artifacts.path / model_dir)], + check=True, + ) + return {"evaluated": model_dir} + + +@runpod.local_entrypoint +def main(): + trained = train.remote(steps=1000) + print(evaluate.remote(trained["model_dir"])) +``` + + + +Each task call provisions a Pod, runs the function, and terminates the Pod when it returns. The `needs` relationship is now the order of the two `.remote()` calls, and `/outputs/model` and `/inputs/model` are one directory on a shared network volume. Create the secret with `rp secret add hf_token` before the first run, and check the short GPU names the SDK accepts in [SDK GPU types](/sdk/configuration/gpu-types). + +Workflows that a GitHub push triggers through the Gradient GitHub App need a new trigger. Run the pipeline from your CI system with `rp flash dev main.py --once`, which runs the entrypoint once and exits. + +### Runtime environment variables + +Review code that reads platform-injected variables or fixed paths. See the [Environment variable map](#environment-variable-map). + +## Step 5: Move your data and models + +First decide where each data set lives on Runpod: + +| Source | Runpod destination | +| --- | --- | +| Machine disk, Droplet boot disk, files in `/notebooks` | Volume disk at `/workspace` for one Pod, or a network volume if more than one Pod or endpoint needs it | +| Shared drive, Notebook `/storage`, DigitalOcean Volumes, Network File Storage | Network volume | +| Paperspace datasets, Spaces buckets | Network volume, or leave in place and read over the S3 API | +| Model weights from Hugging Face | [Cached models](/serverless/endpoints/model-caching) on Serverless, or download into a network volume | +| Model weights you trained | Network volume, or a [global volume](/storage/globalvolume/overview) if workers in several data centers read them | +| Droplet scratch disk | Container disk (don't migrate, it's temporary on both sides) | + + +A network volume lives in one data center, and every Pod or worker that mounts it runs there. Choose a data center that has stock for the GPU you picked in Step 3, and prefer one that supports the [S3-compatible API](/storage/s3-api#datacenter-availability) so you can transfer files without a running Pod. + + +Create the network volume: + +```bash +runpodctl network-volume create --name "migrated-data" --size 500 --data-center-id "US-KS-2" +runpodctl network-volume list +``` + +Size is in GB (10 to 4096) and can grow later but not shrink. + +### From a Machine, Droplet, or DigitalOcean Volume: push with the S3-compatible API + +The source VM can push straight to the network volume, with no Pod running. Create an S3 API key on the [Credentials page](https://console.runpod.io/user/credentials) (it's separate from your Runpod API key), then run the AWS CLI on the Machine or Droplet: + +```bash +aws configure # enter the S3 API key's access key and secret + +aws s3 sync /mnt/volume_nyc2_01/datasets/ s3://NETWORK_VOLUME_ID/datasets/ \ + --region US-KS-2 \ + --endpoint-url https://s3api-us-ks-2.runpod.io/ +``` + +The bucket name is the network volume ID, and object keys are file paths: `s3://NETWORK_VOLUME_ID/datasets/a.bin` appears at `/workspace/datasets/a.bin` on a Pod and `/runpod-volume/datasets/a.bin` on a Serverless worker. `aws s3 sync` can struggle with more than about 10,000 files, so for directories with many small files, create a tar archive first and copy that with `aws s3 cp`. + +### From a Machine or Droplet: push over SSH with rsync + +If the data center you chose doesn't support the S3-compatible API, or you want `rsync` semantics, start a Pod with the network volume attached and push over SSH. Register the source VM's public key with Runpod before you create the Pod, because keys are injected at creation. + +```bash +# On your workstation: register the VM's public key, then create the transfer Pod +runpodctl ssh add-key --key-file ./source-vm-id_ed25519.pub +runpodctl pod create --name "transfer" --compute-type cpu \ + --image "runpod/pytorch:1.0.2-cu1281-torch280-ubuntu2404" \ + --network-volume-id NETWORK_VOLUME_ID --data-center-ids "US-KS-2" --ports "22/tcp" +runpodctl ssh info POD_ID + +# On the Machine or Droplet: push the data +rsync -avz --progress -e "ssh -p POD_SSH_PORT" /home/paperspace/data/ root@POD_IP:/workspace/data/ +``` + +`runpodctl ssh info` prints the host, port, and key to use. `rsync` needs the direct SSH connection, which is why the Pod exposes `22/tcp` and uses an official Runpod image that starts an SSH server. Install `rsync` on the Pod first (`apt update && apt install -y rsync`). If `runpodctl ssh info` reports no public IP, recreate the transfer Pod as a GPU Pod on Secure Cloud. Delete the transfer Pod when the copy finishes. + +### From a Notebook or a dataset + +Open a terminal in the Notebook, where `/notebooks`, `/storage`, and mounted datasets under `/datasets` are all visible, and push to the network volume with the S3-compatible API: + +```bash +pip install awscli +aws configure # enter the S3 API key's access key and secret + +tar -cf /tmp/notebooks.tar -C /notebooks . +aws s3 cp /tmp/notebooks.tar s3://NETWORK_VOLUME_ID/notebooks.tar \ + --region US-KS-2 --endpoint-url https://s3api-us-ks-2.runpod.io/ + +aws s3 sync /storage/ s3://NETWORK_VOLUME_ID/storage/ \ + --region US-KS-2 --endpoint-url https://s3api-us-ks-2.runpod.io/ +``` + +For notebooks tracked in Git, cloning the repository into `/workspace` on the Pod is simpler than copying files. + +### From Spaces + +Spaces is S3-compatible, so you can copy from it with the AWS CLI from inside a Pod that has the network volume mounted. Use a named profile for the Spaces keys so they don't collide with other credentials. + +```bash +aws configure --profile spaces # enter your Spaces access key and secret + +aws s3 sync s3://my-space/models/ /workspace/models/ \ + --profile spaces --endpoint-url https://nyc3.digitaloceanspaces.com +``` + +Replace `nyc3` with your bucket's region. Running this inside a Pod writes straight to the volume's file system. You can also leave data in Spaces and keep reading it from Runpod with the same S3 client code, since only the compute moved. + +### Models + +For Hugging Face models on Serverless, attach the model to the endpoint instead of copying weights. Runpod stages it on the host before the worker starts, and it appears under `/runpod-volume/huggingface-cache/hub/`. This replaces a Paperspace Deployment `git-lfs` integration that clones a Hugging Face URL into `/models`. + +```bash +runpodctl serverless create --template-id TEMPLATE_ID --gpu-id AMPERE_80 \ + --model-reference https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct:main +``` + +Point your model loading code at the cache path, or load by model ID and let the Hugging Face libraries resolve it. See [Cached models](/serverless/endpoints/model-caching). + +## Step 6: Deploy on Runpod + +### Deploy a Pod (from a Machine or GPU Droplet) + +Create the Pod from your image with the GPU, storage, and ports from your inventory. This is the equivalent of `pspace machine create` or `doctl compute droplet create`. + + + +```bash Before: Paperspace +pspace machine create \ + --name trainer \ + --machine-type A100-80G \ + --template-id TEMPLATE_ID \ + --region ny2 \ + --disk-size 500 \ + --public-ip-type dynamic +``` + +```bash Before: DigitalOcean +doctl compute droplet create trainer \ + --size gpu-h100x1-80gb \ + --image gpu-h100x1-base \ + --region nyc2 \ + --ssh-keys SSH_KEY_ID \ + --volumes VOLUME_ID \ + --user-data-file ./setup.sh +``` + +```bash After: runpodctl +runpodctl pod create \ + --name trainer \ + --image YOUR_REGISTRY/trainer:v1 \ + --gpu-id "NVIDIA H100 80GB HBM3" \ + --gpu-count 1 \ + --container-disk-in-gb 50 \ + --network-volume-id NETWORK_VOLUME_ID \ + --data-center-ids "US-KS-2" \ + --ports "8888/http,22/tcp" \ + --min-cuda-version 12.8 \ + --env '{"WANDB_PROJECT":"trainer"}' +``` + +```bash After: REST API v2 +curl -s -X POST https://api.runpod.io/v2/pods \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "trainer", + "image": "YOUR_REGISTRY/trainer:v1", + "gpu": {"id": "NVIDIA H100 80GB HBM3", "count": 1, "minCudaVersion": "12.8"}, + "disk": 50, + "ports": ["8888/http", "22/tcp"], + "env": {"WANDB_PROJECT": "trainer"}, + "dataCenterIds": ["US-KS-2"], + "mounts": {"network": [{"volumeId": "NETWORK_VOLUME_ID", "path": "/workspace"}]}, + "startSsh": true + }' +``` + + + +The Pod's data center must match the network volume's. To use a volume disk instead of a network volume, replace `--network-volume-id` with `--volume-in-gb 500` (REST: `"mounts": {"persistent": {"size": 500, "path": "/workspace"}}`). A volume disk is tied to the Pod's host and is deleted with the Pod, so use a network volume for data you can't recreate. In the console, the same flow is **Pods**, then **Deploy**. + +Wait for the Pod and connect: + +```bash +runpodctl pod get POD_ID +runpodctl ssh info POD_ID +``` + +`runpodctl pod get` reports `runtimeStatus: running` when the container is up. HTTP ports are at `https://POD_ID-8888.proxy.runpod.net`. + +### Deploy a notebook Pod (from a Notebook) + +Create a Pod from the official PyTorch template and set `JUPYTER_PASSWORD`, which the template uses as the JupyterLab token. JupyterLab starts only when that variable is set. + +```bash +runpodctl pod create \ + --name notebook \ + --template-id runpod-torch-v280 \ + --gpu-id "NVIDIA RTX A5000" \ + --volume-in-gb 100 \ + --env '{"JUPYTER_PASSWORD":"CHOOSE_A_TOKEN"}' +``` + +Open `https://POD_ID-8888.proxy.runpod.net` and enter the token. Deploying the same template from the console sets the token for you. Stop the Pod when you finish a session (`runpodctl pod stop POD_ID`) and start it again later (`runpodctl pod start POD_ID`): `/workspace` is kept, and packages installed outside `/workspace` are not. + +### Deploy a load-balancing endpoint (from a Deployment) + +Create the endpoint with REST API v2. The endpoint type is set at creation and can't change later. `runpodctl serverless create` creates queue-based endpoints, so use REST, Terraform, or the console for load balancing. + + + +```yaml Before: Paperspace deployment spec +apiVersion: v1 +name: sentiment-api +image: my-org/sentiment-api:v3 +enabled: true +env: + - name: MODEL_ID + value: distilbert-base-uncased-finetuned-sst-2-english + - name: HF_TOKEN + value: secret:hf_token +resources: + ports: + - 8000 + replicas: 1 + machineType: A4000 + autoscaling: + enabled: true + maxReplicas: 3 + metrics: + - metric: requestDuration + summary: average + value: 0.5 +healthChecks: + readiness: + path: /readiness/ + port: 8000 +``` + +```bash After: REST API v2 +curl -s -X POST https://api.runpod.io/v2/serverless \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "sentiment-api", + "image": "my-org/sentiment-api:v3", + "type": "LOAD_BALANCER", + "gpu": {"pools": ["AMPERE_16"], "count": 1}, + "disk": 20, + "ports": ["8000/http"], + "env": { + "PORT": "8000", + "PORT_HEALTH": "8000", + "HEALTH_CHECK_PATH": "/ping", + "MODEL_ID": "distilbert-base-uncased-finetuned-sst-2-english", + "HF_TOKEN": "{{ RUNPOD_SECRET_hf_token }}" + }, + "workers": {"min": 0, "max": 3, "idleTimeout": 30}, + "scaling": {"type": "REQUEST_COUNT", "requestCount": 4}, + "timeout": 600000 + }' +``` + + + +The response includes `requestUrls.base` (`https://ENDPOINT_ID.api.runpod.ai`) and `requestUrls.health`. Set both `PORT` and `PORT_HEALTH` to your server's port and list the same port in `ports`: each defaults to 80 on its own, and a mismatch leaves the worker unhealthy. `{{ RUNPOD_SECRET_hf_token }}` reads a Runpod secret named `hf_token`, which you create in the console or with `POST /v2/account/secrets`. Load-balancing endpoints scale on `REQUEST_COUNT` only. `workers.min` of 0 scales to zero. Set it to 1 or more to keep warm workers, which is the closest match to Paperspace `replicas`. In the console, the same flow is **Serverless**, then **New Endpoint**. + +### Deploy a queue-based endpoint (from a Deployment or a fan-out Workflow) + +Create a Serverless template for the handler image, then an endpoint from it. A Serverless template binds to one endpoint. + +```bash +runpodctl template create --name "sentiment-queue" --image "my-org/sentiment-handler:v1" \ + --serverless --container-disk-in-gb 20 \ + --env '{"MODEL_ID":"distilbert-base-uncased-finetuned-sst-2-english"}' + +runpodctl serverless create --name "sentiment-queue" --template-id TEMPLATE_ID \ + --gpu-id AMPERE_16 --workers-min 0 --workers-max 3 \ + --scale-by delay --scale-threshold 4 --idle-timeout 30 +``` + +`--scale-by delay` adds workers when requests wait longer than the threshold in seconds. `--scale-by requests` scales on pending request count. You can also deploy a handler straight from a repository with the [GitHub integration](/serverless/workers/github-integration), which builds the image for you. + +### Deploy a vLLM endpoint (from serverless inference, dedicated inference, or a 1-Click Model) + +Deploy the vLLM worker from the Hub with the Hugging Face model you served on DigitalOcean: + +```bash +runpodctl hub search vllm +runpodctl serverless create --hub-id HUB_ID --name "my-llm" \ + --env MODEL_NAME=mistralai/Mistral-7B-Instruct-v0.3 +``` + +The Hub release supplies the GPU and disk defaults, and you can override the GPU with `--gpu-id`. For gated models, set a Hugging Face token as described in [Get started with vLLM](/serverless/vllm/get-started). If Runpod already hosts the model as a Public Endpoint, there's nothing to deploy: go to [Step 7](#step-7-update-the-code-that-calls-paperspace-and-digitalocean). + +### Deploy an SDK app (from a Workflow) + +Run the pipeline from Step 4 in a dev session, then deploy any `@app.queue` or `@app.api` resources in the app as production endpoints: + +```bash +rp flash dev main.py --once +rp flash deploy +``` + +Tasks don't need a deploy step. Each call provisions its own Pod. A task started with `.spawn()` keeps running and billing until it finishes if your script exits without waiting on it. + +## Step 7: Update the code that calls Paperspace and DigitalOcean + +### Deployment clients + +For a load-balancing endpoint, change the base URL and the auth header. Paths and bodies stay the same. + + + +```python Before: Paperspace Deployment +import requests + +response = requests.post( + "https://DEPLOYMENT_HOST.paperspacegradient.com/predict/", + auth=("paperspace", "ENDPOINT_PASSWORD"), # Basic auth for a protected endpoint + json={"prompt": "I love this"}, + timeout=60, +) +print(response.json()) +``` + +```python After: Runpod load-balancing endpoint +import os + +import requests + +response = requests.post( + "https://ENDPOINT_ID.api.runpod.ai/predict/", + headers={"Authorization": f"Bearer {os.environ['RUNPOD_API_KEY']}"}, + json={"prompt": "I love this"}, + timeout=120, +) +print(response.json()) +``` + + + +Raise client timeouts and add retries for the first request after idle, because a cold start has to pull the image and load the model. If no worker is available within 2 minutes the request fails, and Runpod's docs recommend at least 3 retries with 5 to 10 second delays. + +For a queue-based endpoint, wrap the body in `input` and choose sync or async: + +```python client.py +import runpod + +endpoint = runpod.Endpoint("ENDPOINT_ID") + +# Synchronous: wait for the result +print(endpoint.run_sync({"prompt": "I love this"})) + +# Asynchronous: submit, then poll +job = endpoint.run({"prompt": "I love this"}) +print(job.status()) +print(job.output()) +``` + +The equivalent HTTP calls are `POST https://api.runpod.ai/v2/ENDPOINT_ID/runsync` and `POST https://api.runpod.ai/v2/ENDPOINT_ID/run` with `Authorization: Bearer YOUR_API_KEY`, then `GET /status/JOB_ID`. Results come back under `output`. To get a callback instead of polling, add `"webhook": "https://your-app.example.com/hook"` next to `input`. See [Send requests](/serverless/endpoints/send-requests). + +### OpenAI-compatible clients + +If you call DigitalOcean serverless inference with the OpenAI SDK, swap the base URL, the key, and the model name. + + + +```python Before: DigitalOcean serverless inference +import os + +from openai import OpenAI + +client = OpenAI( + base_url="https://inference.do-ai.run/v1", + api_key=os.getenv("MODEL_ACCESS_KEY"), +) + +resp = client.chat.completions.create( + model="llama3.3-70b-instruct", + messages=[{"role": "user", "content": "What is the capital of Portugal?"}], +) +print(resp.choices[0].message.content) +``` + +```python After: Runpod Serverless vLLM +import os + +from openai import OpenAI + +client = OpenAI( + base_url="https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1", + api_key=os.getenv("RUNPOD_API_KEY"), +) + +resp = client.chat.completions.create( + model="mistralai/Mistral-7B-Instruct-v0.3", # The model your endpoint serves + messages=[{"role": "user", "content": "What is the capital of Portugal?"}], +) +print(resp.choices[0].message.content) +``` + +```python After: Runpod Public Endpoint +import os + +from openai import OpenAI + +client = OpenAI( + base_url="https://api.runpod.ai/v2/qwen3-32b-awq/openai/v1", + api_key=os.getenv("RUNPOD_API_KEY"), +) + +resp = client.chat.completions.create( + model="Qwen/Qwen3-32B-AWQ", + messages=[{"role": "user", "content": "What is the capital of Portugal?"}], +) +print(resp.choices[0].message.content) +``` + + + +On Runpod the endpoint ID or Public Endpoint slug is part of the base URL, so each model has its own base URL. vLLM endpoints support `/chat/completions`, `/completions`, and `/models`. If your code uses DigitalOcean's `/v1/responses`, `/v1/messages`, `/v1/embeddings`, image, audio, or video routes, check [OpenAI compatibility](/serverless/vllm/openai-compatibility) and the [Public Endpoints reference](/public-endpoints/reference) for what's available before you cut over. Calls made through PyDo (`client.chat.completions.create` on a `pydo.Client`) need to move to the OpenAI SDK or `runpod.Endpoint`. + +Dedicated inference clients change the same way: replace the `do-infra.ai` endpoint URL and its access token with the vLLM base URL and your Runpod API key. A private dedicated inference endpoint that you reach over a VPC with a CA certificate becomes a public HTTPS URL authenticated by API key. 1-Click Models serve the Hugging Face TGI API, so clients that call TGI-specific routes need to switch to the OpenAI-compatible routes. + +### Control-plane automation + +Replace scripts that call `pspace`, `doctl`, the Paperspace API, or the DigitalOcean API with `runpodctl` or REST API v2. The full maps are in the [Reference](#reference) section. The pattern is the same throughout: + + + +```bash Before: Paperspace API +curl -X PATCH -H "Authorization: Bearer $PAPERSPACE_API_KEY" \ + "https://api.paperspace.com/v1/machines/MACHINE_ID/stop" +``` + +```bash After: REST API v2 +curl -X POST -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"action": "stop"}' \ + "https://api.runpod.io/v2/pods/POD_ID/action" +``` + + + +A Paperspace auto-shutdown timer has no setting to copy across. Pods come with `runpodctl` and a Pod-scoped API key preinstalled, so a job can stop its own Pod when it finishes: end your script with `runpodctl pod stop $RUNPOD_POD_ID`. + +## Step 8: Verify, cut over, and shut down + + + + Call each new endpoint from the same network your production clients use, with production-shaped inputs. For a load-balancing endpoint: + + ```bash + curl -s https://ENDPOINT_ID.api.runpod.ai/ping -H "Authorization: Bearer $RUNPOD_API_KEY" + curl -s -X POST https://ENDPOINT_ID.api.runpod.ai/predict/ \ + -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" \ + -d '{"prompt": "I love this"}' + ``` + + For a queue-based endpoint, check `https://api.runpod.ai/v2/ENDPOINT_ID/health` and send a `/runsync` request. For a Pod, open the proxy URL and run `nvidia-smi` over SSH. + + + Send the same inputs to the old and new deployments and compare results. Measure a cold request (after the endpoint has scaled to zero) and a warm one separately. If cold starts are too slow for your traffic, set a minimum worker count above 0, use [cached models](/serverless/endpoints/model-caching), or move weights from download-at-start to a network volume. + + + Keep the Paperspace or DigitalOcean deployment live. Route a small share of traffic to Runpod, watch error rates and [logs](/serverless/development/logs), then increase the share. Keep the old deployment until you've run a full traffic cycle on Runpod. + + + Verify file counts and checksums on the network volume before you delete any source disk, shared drive, dataset, or bucket. Paperspace snapshots are deleted when their machine is deactivated, so copy anything you need first. + + + Powering off a machine stops compute billing but storage, public IPs, snapshots, and templates keep billing until you delete them. + + ```bash + pspace deployment delete DEPLOYMENT_ID + pspace machine stop MACHINE_ID + pspace machine delete MACHINE_ID + pspace shared-drive delete SHARED_DRIVE_ID + pspace snapshot delete SNAPSHOT_ID + pspace template delete TEMPLATE_ID + pspace public-ip release PUBLIC_IP + ``` + + Deleting a machine (Paperspace calls this deactivating) permanently removes the machine, its files, and its snapshots, and the machine must be off first. Delete Notebooks, Workflows, and datasets in the Paperspace console, and review your Gradient subscription plan there. Run `pspace public-ip release --help` and `pspace snapshot delete --help` for the exact arguments. + + + A powered-off GPU Droplet still bills. Destroy it, and remove the volumes and snapshots that the destroy leaves behind. + + ```bash + doctl compute droplet delete DROPLET_ID + doctl compute volume delete VOLUME_ID + doctl compute snapshot list + doctl kubernetes cluster node-pool delete CLUSTER_ID POOL_ID + + curl -X DELETE -H "Authorization: Bearer $DIGITALOCEAN_TOKEN" \ + "https://api.digitalocean.com/v2/dedicated-inferences/DEDICATED_INFERENCE_ID" + ``` + + Droplet backups stay until their retention period ends. Delete Spaces buckets and model access keys in the Control Panel once nothing reads them. Revoke the Paperspace API keys and DigitalOcean tokens you created for the migration. + + + Delete transfer Pods and stop development Pods you aren't using (`runpodctl pod list --all`). Keep endpoint minimum workers at 0 unless you need warm capacity, and set an idle timeout that fits your traffic. Review spend with `runpodctl billing pods` and `runpodctl billing serverless`, and turn on [low balance notifications](/accounts-billing/billing#low-balance-notifications): Pods stop when your balance reaches zero. + + + +## Migrate your Terraform + +DigitalOcean resources are managed with the `digitalocean/digitalocean` provider. The Runpod provider (`runpod/runpod`, version 2.x) is built on REST API v2 and reads `RUNPOD_API_KEY` from the environment. Paperspace resources aren't covered by the `digitalocean` provider, so recreate them from the inventory in Step 1. + +| `digitalocean` resource | Runpod resource | Notes | +| --- | --- | --- | +| `digitalocean_droplet` (GPU size) | `runpod_pod` | `size` becomes `gpu_type_id` and `gpu_count`. `image` becomes a Docker image. `region` becomes `data_center_ids`. | +| `digitalocean_droplet` `user_data` | `runpod_pod` `args`, or steps in your Dockerfile | Installs belong in the image. | +| `digitalocean_droplet` `ssh_keys` | `runpod_account_ssh_keys`, `start_ssh = true` | Keys are registered on the account. | +| `digitalocean_volume` | `runpod_network_volume` | `region` becomes `data_center`. `size` is in GB on both. | +| `digitalocean_volume_attachment` | `network_volumes` on `runpod_pod` | The mount is declared on the Pod. | +| `digitalocean_spaces_bucket` | `runpod_network_volume` (with the S3-compatible API) or `runpod_global_volume` | Or keep the bucket and the `digitalocean` provider for it. | +| `digitalocean_container_registry` | `runpod_registry` | Stores pull credentials. It doesn't host images. | +| `digitalocean_kubernetes_node_pool` (GPU size) | `runpod_serverless_endpoint`, `runpod_pod`, or `runpod_cluster` | Depends on the workload. | +| `digitalocean_vpc`, `digitalocean_firewall` | `global_networking = true` on `runpod_pod`, and the `ports` list | Only listed ports are exposed. | + +The following example replaces a GPU Droplet with an attached volume. + + + +```hcl Before: DigitalOcean +terraform { + required_providers { + digitalocean = { + source = "digitalocean/digitalocean" + version = "~> 2.0" + } + } +} + +provider "digitalocean" {} + +resource "digitalocean_volume" "data" { + region = "nyc2" + name = "training-data" + size = 500 + initial_filesystem_type = "ext4" +} + +resource "digitalocean_droplet" "trainer" { + name = "trainer" + image = "gpu-h100x1-base" + size = "gpu-h100x1-80gb" + region = "nyc2" + ssh_keys = [var.ssh_key_fingerprint] + user_data = file("${path.module}/setup.sh") +} + +resource "digitalocean_volume_attachment" "data" { + droplet_id = digitalocean_droplet.trainer.id + volume_id = digitalocean_volume.data.id +} + +variable "ssh_key_fingerprint" { + type = string +} +``` + +```hcl After: Runpod +terraform { + required_version = ">= 1.6" + + required_providers { + runpod = { + source = "runpod/runpod" + version = "~> 2.0" + } + } +} + +provider "runpod" {} + +resource "runpod_network_volume" "data" { + name = "training-data" + size = 500 + data_center = "US-KS-2" +} + +resource "runpod_pod" "trainer" { + name = "trainer" + image = "YOUR_REGISTRY/trainer:v1" + gpu_type_id = "NVIDIA H100 80GB HBM3" + gpu_count = 1 + cloud = "SECURE" + disk = 50 + min_cuda_version = "12.8" + data_center_ids = [runpod_network_volume.data.data_center] + ports = ["8888/http", "22/tcp"] + start_ssh = true + + network_volumes = [{ + volume_id = runpod_network_volume.data.id + path = "/workspace" + }] + + env = { + WANDB_PROJECT = "trainer" + } +} + +output "pod_id" { + value = runpod_pod.trainer.id +} +``` + + + +The `setup.sh` user data script has no direct counterpart: its install steps move into the Docker image. Changing `gpu_type_id`, `gpu_count`, `cloud`, or `data_center_ids` replaces the Pod, so keep data on the network volume. For a Deployment or DOKS inference service, use `runpod_serverless_endpoint` with `endpoint_type = "LOAD_BALANCER"` or `"QUEUE"`, `gpu_pools`, `scaling_type`, `scaling_value`, `workers_min`, and `workers_max`. For every argument, see the [provider reference](https://registry.terraform.io/providers/runpod/runpod/latest/docs). + +Migrate in two applies: create the Runpod resources alongside the DigitalOcean ones, cut over, then remove the `digitalocean_` resources and run `terraform apply` again to destroy them. + +## Reference + +### CLI command map + +| `pspace` | `doctl` | `runpodctl` | +| --- | --- | --- | +| `pspace login` | `doctl auth init` | `runpodctl doctor` or `runpodctl config --apiKey YOUR_API_KEY` | +| `pspace machine list` | `doctl compute droplet list --gpus` | `runpodctl pod list --all` | +| `pspace machine get MACHINE_ID` | `doctl compute droplet get DROPLET_ID` | `runpodctl pod get POD_ID` | +| `pspace machine create` | `doctl compute droplet create` | `runpodctl pod create` | +| `pspace machine start`, `stop`, `restart` | `doctl compute droplet-action` commands | `runpodctl pod start`, `stop`, `restart` | +| `pspace machine update` | | `runpodctl pod update` | +| `pspace machine delete` | `doctl compute droplet delete` | `runpodctl pod delete` | +| | `doctl compute ssh DROPLET_ID` | `runpodctl ssh info POD_ID` (prints the command to run) | +| | `doctl compute size list` | `runpodctl gpu list` | +| | `doctl compute region list` | `runpodctl datacenter list` | +| `pspace template list` | `doctl compute image list` | `runpodctl template list` | +| `pspace template create` | | `runpodctl template create` | +| `pspace shared-drive create`, `list`, `delete` | `doctl compute volume create`, `list`, `delete` | `runpodctl network-volume create`, `list`, `delete` | +| `pspace deployment up`, `pspace up` | | `runpodctl serverless create` (queue-based), REST API v2 (load balancing) | +| `pspace deployment list`, `get` | | `runpodctl serverless list`, `get` | +| `pspace deployment delete` | | `runpodctl serverless delete` | +| `pspace secret create`, `list`, `delete` | | `rp secret add`, `list`, `delete` | +| `pspace init` | | `rp flash init` | +| `pspace startup-script create` | `--user-data-file` | `--docker-args` on `runpodctl pod create`, or the image | +| `pspace public-ip`, `pspace snapshot`, `pspace autoscaling-group` | `doctl compute snapshot` | No counterpart (see [What's different](#whats-different)) | + +Run `runpodctl COMMAND --help` for current flags. + +### API map + +| Paperspace API (`https://api.paperspace.com/v1`) | DigitalOcean API (`https://api.digitalocean.com/v2`) | Runpod REST API v2 (`https://api.runpod.io/v2`) | +| --- | --- | --- | +| `GET /machines` | `GET /droplets` | `GET /pods` | +| `POST /machines` | `POST /droplets` | `POST /pods` | +| `GET /machines/MACHINE_ID` | `GET /droplets/DROPLET_ID` | `GET /pods/POD_ID` | +| `PATCH /machines/MACHINE_ID/start`, `/stop`, `/restart` | `POST /droplets/DROPLET_ID/actions` | `POST /pods/POD_ID/action` with `{"action": "start"}`, `"stop"`, or `"restart"` | +| `PUT /machines/MACHINE_ID` | | `PATCH /pods/POD_ID` | +| `DELETE /machines/MACHINE_ID` | `DELETE /droplets/DROPLET_ID` | `DELETE /pods/POD_ID` | +| `POST /deployments` (upsert) | `POST /dedicated-inferences` | `POST /serverless` | +| `GET /deployments`, `GET /deployments/DEPLOYMENT_ID` | `GET /dedicated-inferences` | `GET /serverless`, `GET /serverless/ENDPOINT_ID` | +| `DELETE /deployments/DEPLOYMENT_ID` | `DELETE /dedicated-inferences/ID` | `DELETE /serverless/ENDPOINT_ID` | +| `GET /deployments/DEPLOYMENT_ID/logs` | | `GET /serverless/ENDPOINT_ID/workers/WORKER_ID/logs` | +| `GET /deployments/DEPLOYMENT_ID/metrics` | | `GET /serverless/ENDPOINT_ID/metrics/requests` | +| `POST /shared-drives`, `GET /shared-drives` | `POST /volumes`, `GET /volumes` | `POST /network-volumes`, `GET /network-volumes` | +| `GET /machine-availability` | `GET /sizes` | `GET /catalog/gpus` | +| | `GET /regions` | `GET /catalog/datacenters` | +| `GET /auth/session` | `GET /account` | `GET /account` | + +All three APIs authenticate with `Authorization: Bearer`. Requests to deployed endpoints go to a different host than the management API: `https://api.runpod.ai/v2/ENDPOINT_ID` for queue-based endpoints and `https://ENDPOINT_ID.api.runpod.ai` for load balancing. + +### Configuration field map + +| Paperspace Deployment spec | Runpod endpoint (REST API v2 `POST /v2/serverless`) | Notes | +| --- | --- | --- | +| `name` | `name` | | +| `image` | `image` | | +| `containerRegistry` | `registry` | A registry credential ID from `runpodctl registry list`. | +| `command` | `entrypoint`, `cmd` | Exec-form arrays. | +| `resources.ports` (`port` in `v0alpha0` specs) | `ports` plus `PORT` and `PORT_HEALTH` in `env` | Load balancing only. | +| `env` (list of `name` and `value`) | `env` (object) | | +| `secret:NAME` values | `{{ RUNPOD_SECRET_name }}` | | +| `resources.machineType` (`resources.instanceType` in `v0alpha0` specs) | `gpu.pools`, `gpu.count` | A pool ID, not a machine name. | +| `resources.replicas` | `workers.min` | Runpod's default is 0. | +| `resources.autoscaling.maxReplicas` | `workers.max` | | +| `resources.autoscaling.metrics` | `scaling.type`, `scaling.requestCount` or `scaling.queueDelay` | No CPU or memory metric. | +| `enabled: false` | `workers.max` of 0, or delete the endpoint | | +| `healthChecks.readiness.path` | `HEALTH_CHECK_PATH` in `env` | One path. `200` healthy, `204` initializing. | +| `healthChecks.liveness`, `startup` | None | Fold into the health path. | +| `basicAuthKey` | None | Bearer API key is always required. | +| `region` | `dataCenterIds` | | +| `integrations` (`git-lfs` Hugging Face URL), `models` in `v0alpha0` specs | Cached model (`--model-reference`) | Path becomes `/runpod-volume/huggingface-cache/hub/`. | +| `integrations` (`s3`) | Network volume in `mounts.network`, or S3 client code | No bucket mount. | + +| Machine or Droplet field | `runpodctl pod create` flag | REST API v2 `POST /v2/pods` field | +| --- | --- | --- | +| `name` | `--name` | `name` | +| `machineType`, Droplet `size` | `--gpu-id`, `--gpu-count` | `gpu.id`, `gpu.count` | +| `templateId`, Droplet `image` | `--image` or `--template-id` | `image` or `templateId` | +| `region` | `--data-center-ids` | `dataCenterIds` | +| `diskSize`, boot disk | `--volume-in-gb` or `--network-volume-id`, plus `--container-disk-in-gb` | `mounts.persistent` or `mounts.network`, plus `disk` | +| Droplet `volumes` | `--network-volume-id` | `mounts.network` | +| `publicIpType`, open ports | `--ports` | `ports` | +| `networkId`, `vpc_uuid` | `--global-networking` | `globalNetworking` | +| `startupScriptId`, `user_data` | `--docker-args` | `cmd`, `entrypoint` | +| Droplet `ssh_keys` | `--ssh` (on by default), keys from `runpodctl ssh add-key` | `startSsh` | +| Droplet `tags` | None | None | +| `autoShutdownTimeout` | None | None | + +### Environment variable map + +| On Paperspace or DigitalOcean | On Runpod | Notes | +| --- | --- | --- | +| Deployment port from the spec | `PORT`, `PORT_HEALTH` | Load-balancing workers. Both default to 80. | +| Readiness probe path from the spec | `HEALTH_CHECK_PATH` | Defaults to `/ping`. | +| `/notebooks`, `/storage`, `/home/paperspace` | `/workspace` | Pod volume mount. | +| `/inputs/NAME`, `/outputs/NAME`, `/datasets` | A path on a mounted network volume | You choose the layout. | +| Droplet volume mount under `/mnt` | `/workspace` (Pods), `/runpod-volume` (Serverless) | | +| Machine ID from the Paperspace API | `RUNPOD_POD_ID` | | +| Droplet region | `RUNPOD_DC_ID` | | +| Droplet public IPv4 | `RUNPOD_PUBLIC_IP`, `RUNPOD_TCP_PORT_22` | The external SSH port is mapped. | +| `nvidia-smi` GPU count | `RUNPOD_GPU_COUNT` | | +| `PAPERSPACE_API_KEY`, `DIGITALOCEAN_TOKEN`, `MODEL_ACCESS_KEY` in your own env | `RUNPOD_API_KEY` | Pods also receive a Pod-scoped `RUNPOD_API_KEY`. | + +### Status map + +| Paperspace machine state | Runpod Pod status | Notes | +| --- | --- | --- | +| Provisioning | `PROVISIONING` | | +| Starting up | `STARTING` (`runtimeStatus: initializing`) | Includes the image pull. | +| On/Ready | `RUNNING` (`runtimeStatus: running`) | Running doesn't mean your port is listening yet. | +| Restarting | `STARTING` | | +| Shutting Down, Off | `EXITED` (`runtimeStatus: stopped`) | Volume disk is kept and billed. | +| Inaccessible | `ERROR` | Check `runpodctl pod logs POD_ID`. | +| Deactivated | `TERMINATED` | | + +Queue-based jobs report `IN_QUEUE`, `IN_PROGRESS`, `COMPLETED`, `FAILED`, `CANCELLED`, or `TIMED_OUT`. See [Job states](/serverless/endpoints/job-states). + +## Migrate with a coding agent + +A coding agent can run the inventory, propose a plan, and perform the migration with the Runpod skills and MCP server. + + + + + + The plugin also connects the hosted MCP server. + + ```bash + claude plugin marketplace add runpod/runpod-plugins-official + claude plugin install runpod@runpod + ``` + + + Install the skills, then add the hosted MCP server. This works for Codex, Cursor, and other agents. + + ```bash + npx skills add runpod/runpod-plugins-official --skill '*' --agent AGENT_ID --global + npx @runpod/mcp-server@latest add --client CLIENT_ID + ``` + + + + The MCP server signs in with OAuth ("Sign in with Runpod"), so you don't need an API key to start. See [Agent skills](/get-started/agent-skills) and [MCP servers](/get-started/mcp-servers). + + + Run your agent in the repository that contains your application code and infrastructure files, and paste this prompt: + + ```text + Migrate this project from Paperspace and DigitalOcean GPU to Runpod. Use the + runpod-cloud-migrate skill and its Paperspace and DigitalOcean reference. + + 1. Inventory first, read-only. Scan this repo for pspace, gradient, and doctl + commands, api.paperspace.com, paperspacegradient.com, .gradient/workflows, + api.digitalocean.com, inference.do-ai.run, digitaloceanspaces.com, pydo, and + digitalocean_* Terraform resources. If pspace or doctl is authenticated, list + Machines, Deployments, shared drives, GPU Droplets (doctl compute droplet list + --gpus), volumes, and DOKS GPU node pools. Ask me about Notebooks, Workflows, + and datasets, which only the Paperspace console lists. + 2. Show me a plan: for each workload, the Runpod target (Pod, load-balancing + endpoint, queue-based endpoint, Public Endpoint, vLLM endpoint, @app.task, or + Instant Cluster), the GPU type or pool, the storage, and what has no + equivalent (Windows machines, desktop streaming, the Kubernetes API, VM + snapshots, versioned datasets). Include a cost estimate from live Runpod + pricing and check live GPU stock. + 3. Wait for my approval before you create anything billable on Runpod. + 4. Migrate one workload end to end: build a linux/amd64 image, move the data it + needs to a network volume, deploy, and update the client code. + 5. Verify with a real request from outside Runpod and show me the response. + 6. Never stop or delete Paperspace or DigitalOcean resources, and never delete + source data, without my explicit approval. Don't print secret values. + ``` + + + +## Troubleshooting + +**The Pod starts but files I saved are gone after a stop.** On a VM the whole disk persists. On a Pod only the volume mount does. Move the files to `/workspace`, and install system packages in the image instead of at runtime. + +**JupyterLab returns a 502 on port 8888.** The official PyTorch template starts JupyterLab only when `JUPYTER_PASSWORD` is set. Pods created from the console get it automatically. With `runpodctl pod create`, pass `--env '{"JUPYTER_PASSWORD":"CHOOSE_A_TOKEN"}'`. See [Troubleshooting 502 errors](/pods/troubleshooting/troubleshooting-502-errors). + +**The container fails with a CUDA or driver error.** A VM image carries its own driver. A Pod uses the host's, and hosts differ. Set a CUDA floor that matches your image (`--min-cuda-version` on `runpodctl`, `gpu.minCudaVersion` in REST, `min_cuda_version` in Terraform), and remove any driver installation from your Dockerfile. + +**`exec format error` when the Pod or worker starts.** The image was built for `arm64`. Rebuild with `docker buildx build --platform linux/amd64`. + +**The load-balancing endpoint returns 502 and workers never become healthy.** The server isn't listening where Runpod is checking. Set `PORT` and `PORT_HEALTH` to the port your server binds, list that port in `ports`, bind to `0.0.0.0`, and make sure the health path returns `200`. A readiness route that returns `503` while loading keeps the worker unhealthy: return `204` instead. + +**Requests that worked on a Deployment now time out.** Load-balancing endpoints cap each request at 5.5 minutes and 30 MB, and the Pod HTTP proxy closes connections after 100 seconds. Move long requests to a queue-based endpoint and poll or use a webhook, or expose a TCP port on a Pod. + +**Clients get `401` after the cutover.** Paperspace protected endpoints use basic authentication with the username `paperspace`. Runpod endpoints require `Authorization: Bearer YOUR_API_KEY`. For OpenAI-compatible clients, replace the model access key with your Runpod API key. + +**The first request after idle is slow or fails.** The endpoint scaled to zero and is cold starting. Retry with a delay on the client, raise `workers.min`, enable FlashBoot, or use cached models so weights don't download on every cold start. + +**`runpodctl pod create` fails for lack of capacity.** The GPU type has no stock in the data center your network volume pins you to. Check `runpodctl gpu list`, try another GPU type with the same VRAM, or create the volume in a data center that has stock for that GPU. + +**`aws s3 sync` to the network volume stalls or errors.** The S3-compatible API handles `cp`, `ls`, `mv`, and `rm` well, and `sync` can have trouble past about 10,000 files. Archive small files into a tar first. Also check that `--region` is the data center ID, that the endpoint URL matches it, and that no old `AWS_ACCESS_KEY_ID` environment variable is overriding the Runpod S3 API key. + +## Next steps + + + + Size GPU, memory, and storage for a workload. + + + Serve your own HTTP server with autoscaling workers. + + + Share persistent storage across Pods and endpoints. + + + Manage Runpod infrastructure as code. + + diff --git a/migrate/replicate.mdx b/migrate/replicate.mdx new file mode 100644 index 000000000..e7fcbe235 --- /dev/null +++ b/migrate/replicate.mdx @@ -0,0 +1,1286 @@ +--- +title: "Migrate from Replicate to Runpod" +sidebarTitle: "Replicate" +description: "Move Replicate models, Cog predictors, deployments, and client code to Runpod Public Endpoints and Serverless." +--- + +This guide is for teams that call models on [Replicate](https://replicate.com/docs) or host their own [Cog](https://cog.run) models there. Calls to [official models](https://replicate.com/docs/topics/models/official-models) map to Runpod [Public Endpoints](/public-endpoints/overview) where Runpod hosts the same model. Your own Cog models and [deployments](https://replicate.com/docs/topics/deployments) map to Runpod [Serverless](/serverless/overview) queue-based endpoints, and [trainings](https://replicate.com/docs/get-started/fine-tune-with-flux) map to Pods or `@app.task` runs in the Runpod SDK. At the end you'll have each workload running on Runpod, your client code calling it, and Replicate scaled down. + + + You can hand most of this migration to a coding agent. See [Migrate with a coding agent](#migrate-with-a-coding-agent). + + +## How Replicate maps to Runpod + +| Replicate concept | Runpod equivalent | Notes | +| --- | --- | --- | +| Official model (always on, billed per output) | [Public Endpoint](/public-endpoints/overview) | Only for models in the [Public Endpoints model reference](/public-endpoints/reference). Billed per output. | +| Community or public model (shared hardware, billed for run time) | Public Endpoint, a [Hub](/hub/overview) repo you deploy, or your own Serverless endpoint | On Runpod you run your own copy of anything that isn't a Public Endpoint. | +| Private model | Serverless [queue-based endpoint](/serverless/endpoints/overview) | Scales to zero by default. | +| Deployment (`min_instances`, `max_instances`, `hardware`) | Serverless endpoint (active workers, max workers, GPU pools) | See [Endpoint settings](/serverless/endpoints/endpoint-configurations). | +| Model version (64-character ID) | Container image tag or digest on a Serverless template | You version by image. Endpoints built from GitHub keep a build history you can roll back. | +| Cog (`cog.yaml`, `predict.py`, `cog push`) | `Dockerfile` + `handler.py`, or a `runpod.App` with `@app.queue` | See [Step 4](#step-4-move-your-images-and-code). | +| `r8.im` registry | Any registry Runpod can pull from (Docker Hub, GHCR, ECR, and others) | Private registries use [container registry credentials](/get-started/credentials#container-registry-auth). | +| Prediction | Job | `POST /run` or `/runsync` on `https://api.runpod.ai/v2/ENDPOINT_ID`. | +| Prediction input (`{"input": {...}}`) | Job input (`{"input": {...}}`) | Same envelope. Field names inside `input` are whatever your handler reads. | +| Output files on `replicate.delivery` | A URL you produce (your own S3-compatible bucket) or base64 in the response | Public Endpoints return hosted URLs. Your own endpoints return what your handler returns. | +| Training (`trainings.create`, destination model) | A [Pod](/pods/overview) or an `@app.task` function that writes weights to a [network volume](/storage/network-volumes) | You own the training script and the weights. | +| Hardware SKU (`gpu-t4`, `gpu-l40s`, `gpu-a100-large`, `gpu-h100`) | GPU pool (`AMPERE_16`, `ADA_48_PRO`, `AMPERE_80`, `ADA_80_PRO`) | See the [GPU mapping](#gpu-mapping). | +| Cold boot | Cold start, reduced by FlashBoot, [cached models](/serverless/endpoints/model-caching), and active workers | | +| Secret inputs (`cog.Secret`) | Endpoint environment variables that reference [Runpod secrets](/pods/templates/secrets) | Secrets are set on the endpoint, not sent per request. | +| Regions | [Data centers](/serverless/endpoints/endpoint-configurations#data-centers) you can allow or restrict per endpoint | Network volumes pin an endpoint to one data center. | +| Billing: run time on shared hardware, or all online time for private models and deployments | Per-second billing while workers run (cold start, execution, idle timeout) | See [Serverless pricing](/serverless/pricing). Public Endpoints bill per output. | +| HTTP API (`https://api.replicate.com/v1`) | Job API (`https://api.runpod.ai/v2/ENDPOINT_ID`) and [REST API v2](/api-reference-v2/overview) (`https://api.runpod.io/v2`) for managing resources | Two hosts: one to run jobs, one to manage infrastructure. | +| Client libraries (`replicate` for Python and Node.js, Go, Swift) | `runpod` (Python), `runpod-sdk` (JavaScript), Go SDK, or plain HTTP | See the [operation reference](/serverless/endpoints/operation-reference). | +| `REPLICATE_API_TOKEN` (`r8_...`) | `RUNPOD_API_KEY` | Both are sent as `Authorization: Bearer`. | +| Organizations | [Organizations](/organizations/orgs-overview) | | +| Deployments API for automation (`POST /v1/deployments`) | REST API v2, runpodctl, or the [Terraform provider](/integrations/terraform/overview) (`runpod/runpod`, built on REST API v2) | The provider manages `runpod_serverless_endpoint`, `runpod_template`, `runpod_network_volume`, and more. | +| Deployment metrics dashboard, prediction logs | Endpoint **Metrics** and **Logs** tabs, worker log streaming | See [Logs](/serverless/development/logs) and [Job states and metrics](/serverless/endpoints/job-states). | +| Replicate MCP server and agent skills | Runpod [MCP server](/get-started/mcp-servers) and [agent skills](/get-started/agent-skills) | | + +## What's different + +### What works differently + +**You run your own copy of community models.** On Replicate, any public model can be called by name on shared hardware. On Runpod, a model is either a Public Endpoint that Runpod hosts, or an endpoint in your account that you deploy from a Hub repo, a container image, or a `runpod.App`. + +**Billing follows workers, not predictions.** For Replicate public models you pay for the time a prediction runs. A Runpod Serverless endpoint bills per second while a worker is running, which includes cold start, execution, and the idle timeout after a job. With active workers set to 0, you pay nothing while no worker is running. This is closest to how Replicate bills private models and deployments. + +**Inputs are validated by your code.** Cog generates an input schema from the type annotations on `predict()` and rejects bad input before your code runs. A Runpod handler receives `job["input"]` as a plain dictionary. Validate it yourself, for example with the [validator utility](/serverless/development/validation). With `@app.queue` in the Runpod SDK, the keys of `input` are passed to your function as keyword arguments. + +**File outputs are yours to store.** Replicate uploads any `Path` your predictor returns and gives the client a `replicate.delivery` URL that expires after one hour. A Runpod handler returns JSON. Upload files to your own S3-compatible bucket and return the URL, or return base64 for small files. Response payloads are limited to 10 MB for `/run` and 20 MB for `/runsync`. + +**File inputs arrive as strings.** Cog downloads a file input and hands your code a local `Path`. On Runpod, your handler receives whatever the client sent (usually a URL) and downloads it itself. The Replicate client libraries also upload local files for you. With Runpod, host the file somewhere your worker can reach and send its URL. + +**Results are kept for a shorter time.** Replicate keeps API prediction data for one hour. Runpod keeps a job's result for 30 minutes after an async (`/run`) job completes and 1 minute after a sync (`/runsync`) job completes. Fetch results promptly, or use a webhook. + +**Sync waits are longer.** `Prefer: wait` holds a Replicate request open for up to 60 seconds. `/runsync` waits 90 seconds by default, and you can set `?wait=` from 1000 to 300000 milliseconds. + +**Timeouts are set per endpoint and per job.** Replicate predictions time out after 30 minutes unless you arrange otherwise, and `Cancel-After` sets a per-prediction deadline. Runpod has an execution timeout per endpoint (600 seconds by default in the console, up to 7 days) that you can override per request with `policy.executionTimeout`, plus a job TTL. + +**Streaming is polled.** Replicate streams over server-sent events from `urls.stream`. On Runpod, a handler that `yield`s exposes chunks at `GET /stream/JOB_ID`, which you call repeatedly until the job finishes. For LLMs served through an OpenAI-compatible route, `stream=True` works as it does with any OpenAI-compatible server. + +**Webhooks fire on completion.** Replicate can send `start`, `output`, `logs`, and `completed` events and signs each request. Runpod calls your webhook when the job finishes and retries up to 2 more times, 10 seconds apart, if your server doesn't return `200`. + +### What has no direct equivalent + +| On Replicate | On Runpod | Workaround | +| --- | --- | --- | +| A public catalog where anyone can call any community model by `owner/name` | Public Endpoints for a set of hosted models, and the Hub for repos you deploy into your own account | Deploy the model yourself (Hub, container, or `runpod.App`). | +| Hosted output files (`replicate.delivery`) for your own models | Not provided for your own endpoints | Use your own bucket with `rp_upload`, or return base64. | +| Signed webhooks (`webhook-id`, `webhook-timestamp`, `webhook-signature`) | Runpod's webhook docs don't describe a signature | Put an unguessable token in the webhook URL path, and confirm the job with `GET /status/JOB_ID` before you trust the payload. | +| `webhook_events_filter` (`start`, `output`, `logs`) | Completion webhook only | Use `/stream` or progress updates with `/status` for intermediate state. | +| Managed fine-tuning into a destination model with one API call | No managed trainer | Run the training script on a Pod or with `@app.task`, then serve the weights from a volume. | +| Schema generated from `predict()` annotations, shown on the model page | No generated schema for handlers | Document your inputs, and validate in the handler. | +| Per-request secret inputs that are redacted in logs | Secrets are endpoint environment variables | Move credentials from request inputs to endpoint env vars. | + +| On Runpod | On Replicate | Notes | +| --- | --- | --- | +| Scale to zero for private endpoints, with per-second billing only while workers run | Private models and deployments bill for setup, idle, and active time (fast-booting fine-tunes are the exception) | Set active workers to 0 on Runpod. | +| Any container image, any base, any server | Cog is the packaging format | You can still run a Cog image on Runpod. | +| [Load balancing endpoints](/serverless/load-balancing/overview) with your own HTTP routes | Models are called through the predictions API | Useful for custom REST APIs and WebSockets. | +| Jobs that run up to 7 days | 30 minute default prediction timeout | Set `policy.executionTimeout` and `policy.ttl`. | +| Persistent [network volumes](/storage/network-volumes) shared by workers and Pods | Cog models typically carry weights in the image or fetch them in `setup()` | | + +## Requirements + +- Access to your Replicate account or organization, and an API token (`REPLICATE_API_TOKEN`). +- A [Runpod account](/accounts-billing/manage-accounts) with credits, and a [Runpod API key](/get-started/credentials#api-keys). +- [Docker](https://docs.docker.com/get-started/get-docker/) if you plan to build worker images, and an account with a container registry such as Docker Hub. +- An S3-compatible bucket if your models return files. +- Python 3.10 to 3.14 for the Runpod SDK. +- The source code for each Cog model you host (`cog.yaml`, `predict.py`, and weights). + +## Step 1: Take inventory of what you run on Replicate + +List what you have on Replicate before you create anything on Runpod. Replicate's HTTP API covers everything you need, and the commands below use `curl` and `jq`. + +Confirm which account or organization the token belongs to: + +```bash +curl -s -H "Authorization: Bearer $REPLICATE_API_TOKEN" \ + https://api.replicate.com/v1/account +``` + +List your deployments with their hardware and scaling settings: + +```bash +curl -s -H "Authorization: Bearer $REPLICATE_API_TOKEN" \ + https://api.replicate.com/v1/deployments \ + | jq '.results[] | {owner, name, model: .current_release.model, version: .current_release.version, config: .current_release.configuration}' +``` + +Each result includes `current_release.configuration` with `hardware`, `min_instances`, and `max_instances`. These become the GPU pool, active workers, and max workers on Runpod. Results are paginated, so follow the `next` URL until it's `null`. + +List recent predictions to see which models and versions your code calls in practice: + +```bash +curl -s -H "Authorization: Bearer $REPLICATE_API_TOKEN" \ + "https://api.replicate.com/v1/predictions" \ + | jq -r '.results[] | [.model, .version, .status, .metrics.predict_time] | @tsv' \ + | sort | uniq -c | sort -rn +``` + +This returns 100 predictions per page, most recent first. Use the `predict_time` values to size timeouts and to pick a GPU later. + +List the trainings you've run, and the hardware SKUs Replicate offers so you can read the `hardware` values: + +```bash +curl -s -H "Authorization: Bearer $REPLICATE_API_TOKEN" https://api.replicate.com/v1/trainings \ + | jq '.results[] | {id, model, version, status, created_at}' + +curl -s -H "Authorization: Bearer $REPLICATE_API_TOKEN" https://api.replicate.com/v1/hardware +``` + +The models API lists public models, so find your own models (including private ones) on your Replicate dashboard. For each model you own, fetch its details and its input schema: + +```bash +curl -s -H "Authorization: Bearer $REPLICATE_API_TOKEN" \ + https://api.replicate.com/v1/models/OWNER/MODEL_NAME \ + | jq '{visibility, latest: .latest_version.id, schema: .latest_version.openapi_schema.components.schemas.Input}' +``` + +The `Input` schema lists every input with its type, default, and constraints. Keep it: it's the contract your Runpod handler has to honor. + +Then find every place your code touches Replicate: + +```bash +grep -rnE "import replicate|from replicate|require\(.replicate.\)|from .replicate.|api\.replicate\.com|REPLICATE_API_TOKEN|replicate\.delivery|r8\.im/|from cog import|cog (push|build|predict|login)" \ + --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=.venv . +``` + +Sort each hit into client calls (`replicate.run`, `predictions.create`, webhooks handlers), model code (`cog.yaml`, `predict.py`), and CI (`cog push` in pipelines). Also note anywhere you store or allow-list `replicate.delivery` URLs. + +For each workload, write down the model identifier, whether it's an official model, a community model, or your own, the hardware SKU, min and max instances, typical and worst-case run time, input and output types (especially files), and whether callers use sync mode, polling, streaming, or webhooks. + +## Step 2: Set up Runpod + +Create a [Runpod account](/accounts-billing/manage-accounts), add credits, and create an [API key](/get-started/credentials#api-keys). Then install the tools you need: + +```bash +# runpodctl: manage endpoints, templates, volumes, and Pods from the terminal +curl -fsSL https://cli.runpod.net | sudo bash +runpodctl doctor + +# The Runpod SDK: endpoint client, worker SDK, and the rp CLI +pip install runpod +rp login + +export RUNPOD_API_KEY="YOUR_API_KEY" +``` + +`runpodctl doctor` walks you through API key and SSH key setup. `rp login` stores credentials in `~/.runpod/config.toml`, and the SDK also reads `RUNPOD_API_KEY`. On macOS you can install the CLI with `brew install runpod/runpodctl/runpodctl`. See the [runpodctl overview](/runpodctl/overview) and the [SDK overview](/sdk/overview). + +If you use a coding agent, also connect the [Runpod MCP server](/get-started/mcp-servers). + +## Step 3: Choose where each workload lands + +| What you run on Replicate | Where it lands on Runpod | Why | +| --- | --- | --- | +| Calls to an official or popular public model that Runpod also hosts | [Public Endpoint](/public-endpoints/overview) | Nothing to deploy. You change the URL, the input fields, and how you read the output. | +| Calls to an open-source model that isn't a Public Endpoint (for example, speech-to-text) | A [Hub](/hub/overview) repo deployed to your account | A maintained worker with a handler already written. | +| An LLM you call for chat or completions | Public Endpoint with an OpenAI-compatible route, or a [vLLM worker](/serverless/vllm/overview) | Both expose `/openai/v1`. | +| Your own Cog model, and you want the fastest move | Serverless queue-based endpoint running your Cog image through [cog-worker](/tutorials/migrations/cog/overview) | No code changes. Adds an HTTP hop inside the worker. | +| Your own Cog model, long term | Serverless queue-based endpoint with a native handler | Removes the Cog server from the request path and gives you full control of the image. | +| Your own model, and you don't want to write a Dockerfile | `runpod.App` with `@app.queue` | Python-only, deployed with `rp flash deploy`. | +| A model that needs custom HTTP routes or WebSockets | [Load balancing endpoint](/serverless/load-balancing/overview) | Your own HTTP server instead of the job queue. | +| Trainings and fine-tunes | A [Pod](/pods/overview), or `@app.task` in the Runpod SDK | Long-running batch work that writes weights to a volume. | +| Multi-node training | [Instant Cluster](/instant-clusters) | For jobs that need more than one machine. | + +### Public Endpoint model map + +These are the model families Runpod hosts as Public Endpoints, with the endpoint ID you put in the URL. Check the [model reference](/public-endpoints/reference) for the full list, since models are added and retired. + +| Model family | Runpod Public Endpoint ID | Output field | +| --- | --- | --- | +| FLUX.1 schnell | [`black-forest-labs-flux-1-schnell`](/public-endpoints/models/flux-schnell) | `output.image_url` | +| FLUX.1 dev | [`black-forest-labs-flux-1-dev`](/public-endpoints/models/flux-dev) | `output.image_url` | +| FLUX.1 Kontext dev | [`black-forest-labs-flux-1-kontext-dev`](/public-endpoints/models/flux-kontext-dev) | `output.image_url` | +| Qwen Image | [`qwen-image-t2i`](/public-endpoints/models/qwen-image) | See the model page. | +| Seedream 4.0 | [`seedream-v4-t2i`](/public-endpoints/models/seedream-4-t2i) | See the model page. | +| Nano Banana 2 (edit) | [`google-nano-banana-2-edit`](/public-endpoints/models/nano-banana-2-edit) | See the model page. | +| Wan 2.5 (image-to-video) | [`wan-2-5`](/public-endpoints/models/wan-2-5) | `output.result` | +| Wan 2.6 (text-to-video) | [`wan-2-6-t2v`](/public-endpoints/models/wan-2-6-t2v) | See the model page. | +| Wan 2.2 (image-to-video, 720p) | [`wan-2-2-i2v-720`](/public-endpoints/models/wan-2-2-i2v) | See the model page. | +| Kling v2.6 Motion Control | [`kling-v2-6-std-motion-control`](/public-endpoints/models/kling-v2-6-motion-control) | See the model page. | +| Kling Video O1 (reference-to-video) | [`kling-video-o1-r2v`](/public-endpoints/models/kling-video-o1-r2v) | `output.result` | +| Seedance 2.5 (text-to-video) | [`tdq07exynr7zux`](/public-endpoints/models/seedance-2-5-t2v) | `output.result` | +| Seedance 2.5 (image-to-video) | [`16jsvx5ncmm24x`](/public-endpoints/models/seedance-2-5-i2v) | `output.result` | +| Sora 2 (image-to-video) | [`sora-2-i2v`](/public-endpoints/models/sora-2) | See the model page. | +| Qwen3 32B | [`qwen3-32b-awq`](/public-endpoints/models/qwen3-32b) | `output[0].choices[0].tokens`, or OpenAI-compatible | +| IBM Granite 4.0 | [`granite-4-0-h-small`](/public-endpoints/models/granite-4) | See the model page. | +| Moonshot Kimi | [`moonshot-kimi`](/public-endpoints/models/moonshot-kimi) | OpenAI-compatible | +| Chatterbox Turbo (text-to-speech) | [`chatterbox-turbo`](/public-endpoints/models/chatterbox-turbo) | `output.audio_url` | +| Minimax Speech 02 HD | [`minimax-speech-02-hd`](/public-endpoints/models/minimax-speech) | See the model page. | + + + The same model family can ship in different variants on each platform, and input field names differ (for example, a Replicate model may take `aspect_ratio` where the Runpod endpoint takes `width` and `height`). Compare the input schema on the Replicate model page with the parameters on the Runpod model page before you switch. + + +If the model you call isn't in the reference (Whisper and other speech-to-text models, for example), search the Hub for a worker you can deploy: + +```bash +runpodctl hub search whisper --type SERVERLESS +runpodctl hub get OWNER/REPO_NAME +``` + +If there's no hosted model and no Hub repo, and the model has open weights, deploy it yourself using the handler or `runpod.App` path in [Step 4](#step-4-move-your-images-and-code). Closed-weights models are only available on Runpod if they appear in the model reference. + +### GPU mapping + +Serverless endpoints select hardware by GPU pool. Map each Replicate hardware SKU to a pool with the same or more VRAM, then test: the fit depends on your model's memory use, not on the SKU name. + +| Replicate hardware SKU | GPU on Replicate | Runpod GPU pool | GPUs in the pool | +| --- | --- | --- | --- | +| `gpu-t4` | 1x T4, 16 GB | `AMPERE_16` | A4000, A4500, RTX 4000, RTX 2000 (16 GB) | +| `gpu-t4` (if you need headroom) | 1x T4, 16 GB | `AMPERE_24` or `ADA_24` | L4, A5000, 3090, or 4090 (24 GB) | +| `gpu-l40s` | 1x L40S, 48 GB | `ADA_48_PRO` | L40, L40S, 6000 Ada (48 GB) | +| `gpu-a100-large` | 1x A100, 80 GB | `AMPERE_80` | A100 (80 GB) | +| `gpu-h100` | 1x H100, 80 GB | `ADA_80_PRO` | H100 (80 GB) | +| `gpu-h200` | 1x H200 | `HOPPER_141` | H200 (141 GB) | +| `gpu-l40s-2x`, `gpu-a100-large-2x`, and other multi-GPU SKUs | 2 to 8 GPUs | The same pool with GPUs per worker set to the count | `--gpu-count` in runpodctl, `gpu.count` in REST v2, `gpu_count` in the SDK | +| `cpu`, `cpu-small` | CPU only | A CPU endpoint | `--compute-type CPU`. See [CPU types](/references/cpu-types). | + +Run `GET https://api.replicate.com/v1/hardware` to see the SKUs your account can use. On Runpod, list available GPU types with `runpodctl gpu list` and availability by data center with `runpodctl datacenter list`, and see [GPU types](/references/gpu-types) for every device. Stock varies by data center and over time, so give an endpoint two or three pools in priority order when your model fits on more than one. + +## Step 4: Move your images and code + +Pick the path that matches each workload from Step 3. + +### Path 1: Call a Public Endpoint instead of a Replicate model + +There's no image or code to move. Skip to [Step 7](#step-7-update-the-code-that-calls-replicate) for the client changes. + +### Path 2a: Run your existing Cog image with cog-worker + +[cog-worker](https://github.com/runpod-workers/cog-worker) wraps a Cog model in a Runpod Serverless worker. You build an image that names the Replicate model and version, push it to your registry, and create an endpoint from it: + +```bash +git clone https://github.com/runpod-workers/cog-worker.git +cd cog-worker/ + +docker build --platform linux/amd64 \ + --tag DOCKER_USERNAME/my-cog-worker:v1 \ + --build-arg COG_REPO=REPLICATE_OWNER \ + --build-arg COG_MODEL=MODEL_NAME \ + --build-arg COG_VERSION=VERSION_ID \ + . + +docker push DOCKER_USERNAME/my-cog-worker:v1 +``` + +`COG_REPO`, `COG_MODEL`, and `COG_VERSION` are the owner, model name, and 64-character version ID from the model's **Versions** tab on Replicate. The full walkthrough is in [Migrate a Cog model](/tutorials/migrations/cog/overview). + + + This path keeps the FastAPI server that Cog uses to run the model inside the worker, so each job passes through an extra layer before it reaches your model, which can add a delay. Use it to move quickly, then port to a native handler. Send one test request to confirm how the worker maps `input` to your predictor's arguments and how it returns file outputs before you move traffic. + + +### Path 2b: Port predict.py to a Runpod handler + +This is the long-term path. The translation is mechanical: + +| Cog | Runpod handler | +| --- | --- | +| `cog.yaml` `build` section | `Dockerfile` | +| `predict: "predict.py:Predictor"` | `CMD ["python", "-u", "/handler.py"]` | +| `Predictor.setup()` | Module-level code that runs once when the worker starts | +| `Predictor.predict(**inputs)` | `handler(job)`, reading `job["input"]` | +| `Input(default=..., ge=..., le=..., choices=...)` | A validation schema in the handler | +| Return `Path` | Upload the file and return a URL, or return base64 | +| Return `Iterator[str]` and `yield` | A generator handler that `yield`s | +| Raise an exception | Raise an exception (the job becomes `FAILED`), or return `{"error": "..."}` for a custom error message | +| `cog predict -i prompt="..."` | `python handler.py --test_input '{"input": {"prompt": "..."}}'` | +| `cog push r8.im/OWNER/MODEL` | `docker build` and `docker push` to your registry | + +Here's a complete Cog model before the port. It loads a text-to-image pipeline in `setup()` and returns a PNG: + +```yaml cog.yaml +build: + gpu: true + python_version: "3.11" + python_requirements: requirements.txt +predict: "predict.py:Predictor" +``` + +```python predict.py +import random + +import torch +from cog import BasePredictor, Input, Path +from diffusers import AutoPipelineForText2Image + + +class Predictor(BasePredictor): + def setup(self): + """Load the model into memory once, when the container starts.""" + self.pipe = AutoPipelineForText2Image.from_pretrained( + "stabilityai/sdxl-turbo", + torch_dtype=torch.float16, + variant="fp16", + ).to("cuda") + + def predict( + self, + prompt: str = Input(description="Text prompt"), + num_inference_steps: int = Input(description="Denoising steps", default=1, ge=1, le=8), + seed: int = Input(description="Random seed. Use -1 for a random seed.", default=-1), + ) -> Path: + if seed < 0: + seed = random.randint(0, 2**31 - 1) + generator = torch.Generator("cuda").manual_seed(seed) + image = self.pipe( + prompt=prompt, + num_inference_steps=num_inference_steps, + guidance_scale=0.0, + generator=generator, + ).images[0] + out = "/tmp/out.png" + image.save(out) + return Path(out) +``` + + + Current Cog documentation uses `run: "run.py:Runner"`, `BaseRunner`, and a `run()` method. `BasePredictor` and `predict()` are deprecated there but still work, and Replicate's own guides still show them. The port below is the same for both: `setup()` becomes module-level code, and `predict()` or `run()` becomes the handler. + + +Here's the same model as a Runpod worker. The handler validates input, generates the image, and returns a URL if a bucket is configured or base64 if not: + +```python handler.py +import base64 +import os +import random + +import runpod +import torch +from diffusers import AutoPipelineForText2Image +from runpod.serverless.utils import rp_upload +from runpod.serverless.utils.rp_validator import validate + +# Runs once per worker, when the container starts (Cog's setup()). +pipe = AutoPipelineForText2Image.from_pretrained( + "stabilityai/sdxl-turbo", + torch_dtype=torch.float16, + variant="fp16", +).to("cuda") + +# Replaces the Input(...) annotations on predict(). +INPUT_SCHEMA = { + "prompt": {"type": str, "required": True}, + "num_inference_steps": { + "type": int, + "required": False, + "default": 1, + "constraints": lambda steps: 1 <= steps <= 8, + }, + "seed": {"type": int, "required": False, "default": -1}, +} + + +def handler(job): + """Runs once per job (Cog's predict()).""" + validated = validate(job["input"], INPUT_SCHEMA) + if "errors" in validated: + return {"error": validated["errors"]} + args = validated["validated_input"] + + seed = args["seed"] if args["seed"] >= 0 else random.randint(0, 2**31 - 1) + generator = torch.Generator("cuda").manual_seed(seed) + image = pipe( + prompt=args["prompt"], + num_inference_steps=args["num_inference_steps"], + guidance_scale=0.0, + generator=generator, + ).images[0] + + out = f"/tmp/{job['id']}.png" + image.save(out) + + try: + if os.environ.get("BUCKET_ENDPOINT_URL"): + # Uploads to your S3-compatible bucket and returns the object's URL. + return {"image_url": rp_upload.upload_image(job["id"], out), "seed": seed} + + # No bucket configured: return the file inline. Keep this for small files only. + with open(out, "rb") as f: + return {"image_base64": base64.b64encode(f.read()).decode("utf-8"), "seed": seed} + finally: + os.remove(out) + + +runpod.serverless.start({"handler": handler}) +``` + +What matters in this file: + +- The model loads at module level, so it loads once per worker and stays in memory between jobs. Anything inside `handler()` runs on every job. +- `validate()` returns either `errors` or `validated_input` with defaults filled in. Returning `{"error": ...}` sends your message back to the caller instead of running the model. See [Validate inputs](/serverless/development/validation). +- `rp_upload.upload_image()` reads `BUCKET_ENDPOINT_URL`, `BUCKET_ACCESS_KEY_ID`, and `BUCKET_SECRET_ACCESS_KEY` from the worker's environment. `BUCKET_ENDPOINT_URL` must include your bucket name. See [S3 bucket configuration](/serverless/development/environment-variables#s3-bucket-configuration). +- The returned dictionary becomes the job's `output`. There's no fixed output shape, so decide on one and keep it stable for your clients. + +The `Dockerfile` replaces the `build` section of `cog.yaml`: + +```dockerfile Dockerfile +FROM runpod/pytorch:2.8.0-py3.11-cuda12.8.1-cudnn-devel-ubuntu22.04 + +WORKDIR / + +COPY requirements.txt . +RUN pip install --no-cache-dir -r requirements.txt + +# Bake the weights into the image so cold starts don't download them. +RUN python -c "import torch; from diffusers import AutoPipelineForText2Image; AutoPipelineForText2Image.from_pretrained('stabilityai/sdxl-turbo', torch_dtype=torch.float16, variant='fp16')" + +COPY handler.py /handler.py + +CMD ["python", "-u", "/handler.py"] +``` + +```txt requirements.txt +runpod +diffusers +transformers +accelerate +``` + +The base image already includes PyTorch and CUDA, so `requirements.txt` only lists what the handler adds. Pin the versions you run on Replicate so outputs match. If your model is on Hugging Face, you can leave the weights out of the image and use [cached models](/serverless/endpoints/model-caching) instead. + +Test the handler on a machine with a GPU before you build: + +```bash +python handler.py --test_input '{"input": {"prompt": "a lighthouse at dawn", "num_inference_steps": 2}}' +``` + +Then build for `linux/amd64` and push: + +```bash +docker build --platform linux/amd64 -t DOCKER_USERNAME/sdxl-turbo-worker:v1 . +docker push DOCKER_USERNAME/sdxl-turbo-worker:v1 +``` + +The `--platform linux/amd64` flag is required, including on Apple Silicon. Use a version tag or digest, not `:latest`, so you know which image each worker runs. See [Create a Dockerfile](/serverless/workers/create-dockerfile) and [Deploy workers from Docker Hub](/serverless/workers/deploy). + +#### File inputs + +If your predictor takes a `Path` input, the handler receives a string. Download it at the start of the handler: + +```python +import requests + + +def download(url: str, dest: str) -> str: + with requests.get(url, stream=True, timeout=60) as response: + response.raise_for_status() + with open(dest, "wb") as f: + for chunk in response.iter_content(chunk_size=1 << 20): + f.write(chunk) + return dest + + +def handler(job): + image_path = download(job["input"]["image"], f"/tmp/{job['id']}-input") + ... +``` + +Clients that passed a local file handle to `replicate.run()` relied on the Replicate client to upload it. Change those callers to upload the file to your bucket first and send the URL. Request payloads are limited to 10 MB for `/run` and 20 MB for `/runsync`, so base64 inputs only suit small files. + +#### Streaming output + +A Cog predictor that returns `Iterator[str]` becomes a generator handler: + +```python handler.py +import runpod + + +def handler(job): + for token in generate_tokens(job["input"]["prompt"]): + yield token + + +runpod.serverless.start({ + "handler": handler, + "return_aggregate_stream": True, +}) +``` + +Each `yield` is available from `/stream/JOB_ID`. `return_aggregate_stream` also makes the collected output available from `/status` and `/runsync`. See [Streaming handlers](/serverless/workers/handler-functions#streaming-handlers). + +#### Concurrency + +Cog's `@cog.concurrent(max=N)` on an async `run()` (or `concurrency.max` in older `cog.yaml` files) maps to a [concurrent handler](/serverless/workers/concurrent-handler), where one worker processes several jobs at once. + +### Path 3: Port to a runpod.App + +If you'd rather not maintain a Dockerfile, define the model as a function in the Runpod SDK. The SDK packages your code and dependencies and runs them on Runpod's runtime images: + +```python main.py +import runpod + +app = runpod.App("sdxl-turbo") + +sdxl_turbo = runpod.Model("stabilityai/sdxl-turbo") + +_pipe = None # Loaded once per worker and reused across jobs. + + +@app.queue( + gpu=runpod.GpuGroup.ADA_24, + workers=(0, 3), + idle_timeout=60, + dependencies=["diffusers", "transformers", "accelerate"], + model=sdxl_turbo, +) +def generate(prompt: str, num_inference_steps: int = 1, seed: int = -1) -> dict: + import base64 + import io + import random + + import torch + from diffusers import AutoPipelineForText2Image + + global _pipe + if _pipe is None: + _pipe = AutoPipelineForText2Image.from_pretrained( + str(sdxl_turbo.path), + torch_dtype=torch.float16, + variant="fp16", + ).to("cuda") + + if not 1 <= num_inference_steps <= 8: + raise ValueError("num_inference_steps must be between 1 and 8") + if seed < 0: + seed = random.randint(0, 2**31 - 1) + + image = _pipe( + prompt=prompt, + num_inference_steps=num_inference_steps, + guidance_scale=0.0, + generator=torch.Generator("cuda").manual_seed(seed), + ).images[0] + + buffer = io.BytesIO() + image.save(buffer, format="PNG") + return {"image_base64": base64.b64encode(buffer.getvalue()).decode("utf-8"), "seed": seed} + + +@runpod.local_entrypoint +def main(): + result = generate.remote(prompt="a lighthouse at dawn", num_inference_steps=2) + print(result["seed"], len(result["image_base64"])) +``` + +How this lines up with Cog: + +- `@app.queue(...)` replaces `cog.yaml`: `gpu` is the hardware, `workers` is the scaling range, and `dependencies` is the package list. `torch` is already on the built-in GPU images. +- The function's parameters are the inputs. A request's `input` keys are passed as keyword arguments, so `{"input": {"prompt": "..."}}` calls `generate(prompt="...")`. +- Imports go inside the function so your local machine doesn't need them, and the pipeline is cached in a module-level variable so it loads once per worker. +- `model=sdxl_turbo` stages the Hugging Face weights on the host before the worker starts, and `sdxl_turbo.path` points to them inside the worker. +- This example returns base64. For larger files, upload to your bucket inside the function and return the URL. Pass bucket credentials with `env` and `runpod.Secret`. + +Run it against real hardware with `rp flash dev main.py`. See the [SDK quickstart](/sdk/quickstart) and [resource parameters](/sdk/configuration/parameters). + +### Environment variables and secrets + +Cog models take secrets as `cog.Secret` inputs on each prediction. On Runpod, set them once as environment variables on the endpoint and read them with `os.environ`. Store the value as a [Runpod secret](/get-started/credentials#secrets) and reference it as `{{ RUNPOD_SECRET_secret_name }}` in the variable's value, or use `runpod.Secret("secret-name")` in a `runpod.App`. Remove secret fields from your request inputs. + +## Step 5: Move your data and models + +Cog models carry their weights in the image or fetch them in `setup()`, so there's usually no separate storage to copy. What you need to move is model weights and any files your callers depend on. + +**Weights in your Cog image.** If your weights are in the model repository or downloaded during the Cog build, they're already in your source. Bake them into the worker image as shown in Step 4, or put them on a network volume. + +**Weights from Hugging Face.** Use [cached models](/serverless/endpoints/model-caching). Runpod schedules workers on hosts that already have the model and makes it available under `/runpod-volume/huggingface-cache/hub/`. Set it in the console's **Model** field, with `--model-reference` in runpodctl, or with `model=runpod.Model(...)` in the SDK. + +**Fine-tuned weights created on Replicate.** A Replicate training adds a new version to its destination model. Replicate's fine-tuning guide describes a **Download weights** option for trained models, and the training object (`GET /v1/trainings/TRAINING_ID`) has an `output` field. Download the weights while you still have access, then put them where your Runpod worker can read them. + +**Large or shared weights.** Create a network volume and copy files to it through the [S3-compatible API](/storage/s3-api): + +```bash +runpodctl network-volume create --name model-weights --size 100 --data-center-id EU-RO-1 + +aws s3 cp --region EU-RO-1 \ + --endpoint-url https://s3api-eu-ro-1.runpod.io/ \ + ./my-lora.safetensors s3://NETWORK_VOLUME_ID/loras/my-lora.safetensors +``` + +The S3-compatible API uses a separate S3 API key and is available in the data centers listed on that page. On Serverless workers the volume is mounted at `/runpod-volume`, so this file is at `/runpod-volume/loras/my-lora.safetensors`. A network volume lives in one data center, and an endpoint that mounts it runs its workers there. Choose a data center that has the GPU pool you need (`runpodctl datacenter list`). + +**Output files your users still reference.** Files on `replicate.delivery` from API predictions are deleted after an hour, so anything you need should already be in your own storage. Update allow-lists and any code that expects `replicate.delivery` hosts. + +## Step 6: Deploy on Runpod + +### Public Endpoints + +There's nothing to deploy. Test the model in the [Hub playground](https://console.runpod.io/hub?tabSelected=public_endpoints), then go to Step 7. + +### Hub repo + +Deploy a Hub repo to a Serverless endpoint in your account: + +```bash +runpodctl hub search whisper --type SERVERLESS +runpodctl serverless create --hub-id HUB_ID --name "speech-to-text" +``` + +GPU settings, container disk, and default environment variables come from the Hub release. Override the GPU with `--gpu-id` and environment variables with `--env KEY=VALUE`. In the console, open the [Hub](https://console.runpod.io/hub), select the repo, and click **Deploy**. + +### Serverless endpoint from your image + +A Replicate deployment is one API call with a model, version, hardware, and instance range: + +```bash +curl -s -X POST https://api.replicate.com/v1/deployments \ + -H "Authorization: Bearer $REPLICATE_API_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "sdxl-turbo", + "model": "acme/sdxl-turbo", + "version": "VERSION_ID", + "hardware": "gpu-l40s", + "min_instances": 0, + "max_instances": 3 + }' +``` + +On Runpod, create a Serverless template that holds the image and environment, then an endpoint that holds the hardware and scaling: + +```bash +runpodctl template create \ + --name "sdxl-turbo-worker" \ + --image "docker.io/DOCKER_USERNAME/sdxl-turbo-worker:v1" \ + --serverless \ + --container-disk-in-gb 30 \ + --env '{"BUCKET_ENDPOINT_URL":"https://YOUR_BUCKET.s3.us-west-2.amazonaws.com","BUCKET_ACCESS_KEY_ID":"{{ RUNPOD_SECRET_bucket_access_key_id }}","BUCKET_SECRET_ACCESS_KEY":"{{ RUNPOD_SECRET_bucket_secret_access_key }}"}' + +runpodctl serverless create \ + --name "sdxl-turbo" \ + --template-id TEMPLATE_ID \ + --gpu-id ADA_48_PRO \ + --workers-min 0 \ + --workers-max 3 \ + --idle-timeout 5 \ + --execution-timeout 600 \ + --flash-boot \ + --min-cuda-version 12.8 +``` + +`min_instances` becomes `--workers-min` (active workers) and `max_instances` becomes `--workers-max`. `--gpu-id` takes a pool ID or a GPU type ID. Use the template ID that `template create` returns. For a private registry, create credentials with `runpodctl registry create` and pass `--registry-auth-id` to `template create`. Check `runpodctl serverless create --help` for the current flags, or see the [runpodctl serverless reference](/runpodctl/reference/runpodctl-serverless). + +The same endpoint through [REST API v2](/api-reference-v2/serverless/create-a-serverless-endpoint), without a separate template: + +```bash +curl -s -X POST https://api.runpod.io/v2/serverless \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "sdxl-turbo", + "image": "docker.io/DOCKER_USERNAME/sdxl-turbo-worker:v1", + "type": "QUEUE", + "disk": 30, + "gpu": {"pools": ["ADA_48_PRO", "AMPERE_48"], "count": 1, "minCudaVersion": "12.8"}, + "workers": {"min": 0, "max": 3, "idleTimeout": 5}, + "scaling": {"type": "QUEUE_DELAY", "queueDelay": 4}, + "flashboot": "FLASHBOOT", + "timeout": 600000 + }' +``` + +`type` and `scaling` are required. REST v2 defaults differ from the console: `flashboot` defaults to `OFF`, `timeout` to 300000 ms, and `workers.idleTimeout` to 10 seconds, so set them explicitly. The response includes the endpoint ID you use in request URLs. + +In the console, go to [Serverless](https://www.console.runpod.io/serverless), click **New Endpoint**, and import from a Docker registry or a [GitHub repository](/serverless/workers/github-integration). The GitHub integration builds the image for you on each push, which replaces `cog push` in CI. + +### Scaling and cold starts + +| Replicate deployment setting | Runpod endpoint setting | Notes | +| --- | --- | --- | +| `min_instances` | Active workers (`--workers-min`, `workers.min`) | Active workers are always on and are billed continuously. | +| `max_instances` | Max workers (`--workers-max`, `workers.max`) | Caps concurrency and spend. | +| `hardware` | GPU pools (`--gpu-id`, `gpu.pools`) and GPUs per worker | List up to three in priority order. | +| `version` | Template image tag | Update the template's image to roll out a new version. | +| Autoscaling by demand | Queue delay or request count scaling | See [auto-scaling type](/serverless/endpoints/endpoint-configurations#auto-scaling-type). | +| Setup timeout | `RUNPOD_INIT_TIMEOUT` environment variable | 420 seconds by default. Raise it for models that take longer to load. | +| Always-on official models | Public Endpoints, or active workers on your own endpoint | | + +To reduce cold starts without paying for an always-on worker, enable FlashBoot, use cached models or bake weights into the image, keep the image small, and raise the idle timeout so a worker stays warm between bursts. To remove cold starts for latency-sensitive traffic, set active workers to 1 or more. + +### runpod.App + +Deploy the app from Step 4: + +```bash +rp flash deploy +rp flash app list +``` + +`rp flash deploy` builds your code and dependencies into an artifact, creates the endpoints, and prints a link to each one. Scaling lives in the decorator (`workers=(0, 3)`, `idle_timeout=60`), so changing it is a code change followed by another `rp flash deploy`. See [Deploy an app](/sdk/apps/deploy). + +### Trainings + +Replicate trainings call a trainer model's `train()` function and push the result to a destination model. On Runpod, run your own training script and write the weights to a network volume that your serving endpoint also mounts. + +With a Pod: + +```bash +runpodctl network-volume create --name ft-weights --size 50 --data-center-id EU-RO-1 + +runpodctl pod create --name ft-train \ + --template-id runpod-torch-v280 \ + --gpu-id "NVIDIA GeForce RTX 4090" \ + --data-center-ids EU-RO-1 \ + --network-volume-id NETWORK_VOLUME_ID + +# Connect over SSH, run your training script, and write outputs under /workspace. + +runpodctl pod delete POD_ID +``` + +The volume is mounted at `/workspace` on the Pod and at `/runpod-volume` on Serverless workers, so a file the Pod writes to `/workspace/outputs/lora` is at `/runpod-volume/outputs/lora` in your handler. Delete the Pod when training finishes. The volume keeps the weights. + +With the Runpod SDK, `@app.task` provisions a Pod for each call and terminates it when the function returns: + +```python main.py +import runpod + +app = runpod.App("finetune") + +weights = runpod.NetworkVolume("ft-weights", size=50) + + +@app.task(gpu="4090", mounts={"/weights": weights}, dependencies=["peft", "datasets", "transformers"]) +def train(dataset_url: str, steps: int = 1000) -> dict: + out_dir = weights.path / "outputs" / "lora" + out_dir.mkdir(parents=True, exist_ok=True) + # Download dataset_url, run your training loop, and save the adapter to out_dir. + return {"weights": "outputs/lora"} + + +@app.queue(gpu="4090", mounts={"/runpod-volume": weights}) +def generate(prompt: str) -> dict: + adapter_dir = weights.path / "outputs" / "lora" + # Load the base model and the adapter from adapter_dir once per worker, then generate. + return {"adapter": str(adapter_dir), "prompt": prompt} + + +@runpod.local_entrypoint +def main(): + print(train.remote(dataset_url="https://example.com/data.zip", steps=500)) +``` + +Both functions mount the same volume, so `generate` reads what `train` wrote, and the SDK places both in the volume's data center. See [Storage](/sdk/configuration/storage). + +## Step 7: Update the code that calls Replicate + +### Run a model and wait for the result + + + + Before, with the Replicate client: + + ```python + import replicate + + output = replicate.run( + "black-forest-labs/flux-schnell", + input={"prompt": "an iguana on the beach, pointillism"}, + ) + + with open("output.png", "wb") as f: + f.write(output[0].read()) + ``` + + After, against a Runpod Public Endpoint: + + ```python + import os + + import requests + + response = requests.post( + "https://api.runpod.ai/v2/black-forest-labs-flux-1-schnell/runsync", + headers={"Authorization": f"Bearer {os.environ['RUNPOD_API_KEY']}"}, + json={"input": {"prompt": "an iguana on the beach, pointillism", "width": 1024, "height": 1024}}, + timeout=120, + ) + response.raise_for_status() + result = response.json() + + image = requests.get(result["output"]["image_url"], timeout=60) + with open("output.png", "wb") as f: + f.write(image.content) + ``` + + After, against your own endpoint with the Runpod SDK: + + ```python + import base64 + import os + + import runpod + + runpod.api_key = os.environ["RUNPOD_API_KEY"] + endpoint = runpod.Endpoint("ENDPOINT_ID") + + output = endpoint.run_sync( + {"prompt": "an iguana on the beach, pointillism", "num_inference_steps": 2}, + timeout=120, + ) + + with open("output.png", "wb") as f: + f.write(base64.b64decode(output["image_base64"])) + ``` + + + + Before, with the Replicate client: + + ```javascript + import { writeFile } from "node:fs/promises"; + import Replicate from "replicate"; + + const replicate = new Replicate(); + + const [output] = await replicate.run("black-forest-labs/flux-schnell", { + input: { prompt: "an iguana on the beach, pointillism" }, + }); + + await writeFile("output.png", output); + ``` + + After, against a Runpod Public Endpoint: + + ```javascript + import { writeFile } from "node:fs/promises"; + + const response = await fetch( + "https://api.runpod.ai/v2/black-forest-labs-flux-1-schnell/runsync", + { + method: "POST", + headers: { + Authorization: `Bearer ${process.env.RUNPOD_API_KEY}`, + "Content-Type": "application/json", + }, + body: JSON.stringify({ + input: { prompt: "an iguana on the beach, pointillism", width: 1024, height: 1024 }, + }), + } + ); + const result = await response.json(); + + const image = await fetch(result.output.image_url); + await writeFile("output.png", Buffer.from(await image.arrayBuffer())); + ``` + + After, against your own endpoint with the `runpod-sdk` package: + + ```javascript + import { writeFile } from "node:fs/promises"; + import runpodSdk from "runpod-sdk"; + + const runpod = runpodSdk(process.env.RUNPOD_API_KEY); + const endpoint = runpod.endpoint("ENDPOINT_ID"); + + const result = await endpoint.runSync({ + input: { prompt: "an iguana on the beach, pointillism", num_inference_steps: 2 }, + timeout: 120000, + }); + + await writeFile("output.png", Buffer.from(result.output.image_base64, "base64")); + ``` + + + + Before: + + ```bash + curl -s -X POST https://api.replicate.com/v1/models/black-forest-labs/flux-schnell/predictions \ + -H "Authorization: Bearer $REPLICATE_API_TOKEN" \ + -H "Content-Type: application/json" \ + -H "Prefer: wait" \ + -d '{"input": {"prompt": "an iguana on the beach, pointillism"}}' + ``` + + After: + + ```bash + curl -s -X POST https://api.runpod.ai/v2/black-forest-labs-flux-1-schnell/runsync \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"input": {"prompt": "an iguana on the beach, pointillism", "width": 1024, "height": 1024}}' + ``` + + + +What changes in every caller: + +- The model is in the URL path (`/v2/ENDPOINT_ID/...`), not in a `version` field or a `/models/OWNER/NAME/` path. +- The Python `runpod.Endpoint` client returns the job's `output` directly. Over HTTP, the response is a job object and your data is under `output`. +- There's no `FileOutput` object. You get the URL or base64 string your endpoint returns. Public Endpoint output URLs expire after 7 days, so download what you need to keep. +- A model error doesn't raise `ModelError`. Check for `status` of `FAILED` and read `error` in the job object. + +### Create a job and poll + +Before, with `predictions.create` and polling: + +```python +import replicate + +prediction = replicate.predictions.create( + model="black-forest-labs/flux-schnell", + input={"prompt": "painting of a cat"}, +) +prediction.wait() +print(prediction.status, prediction.output) +``` + +After, with `/run` and `/status`: + +```python +import os +import time + +import requests + +BASE_URL = "https://api.runpod.ai/v2/ENDPOINT_ID" +HEADERS = {"Authorization": f"Bearer {os.environ['RUNPOD_API_KEY']}"} + +job = requests.post(f"{BASE_URL}/run", headers=HEADERS, json={"input": {"prompt": "painting of a cat"}}, timeout=30).json() + +interval = 1 +while True: + status = requests.get(f"{BASE_URL}/status/{job['id']}", headers=HEADERS, timeout=30).json() + if status["status"] in ("COMPLETED", "FAILED", "CANCELLED", "TIMED_OUT"): + break + time.sleep(interval) + interval = min(interval * 2, 30) + +print(status["status"], status.get("output")) +``` + +Or with the SDK, where `endpoint.run()` returns a job handle: + +```python +import runpod + +endpoint = runpod.Endpoint("ENDPOINT_ID") + +run_request = endpoint.run({"prompt": "painting of a cat"}) +print(run_request.status()) +print(run_request.output(timeout=120)) +``` + +Fetch the result within 30 minutes of completion. To cancel, send `POST /cancel/JOB_ID` in place of `prediction.cancel()`. If `/run` times out on the network, look the job up with `/status` before you resubmit, because a retry creates a second job that's billed separately. + +### Webhooks + +Before: + +```python +prediction = replicate.predictions.create( + model="black-forest-labs/flux-schnell", + input={"prompt": "painting of a cat"}, + webhook="https://example.com/webhooks/replicate", + webhook_events_filter=["completed"], +) +``` + +After, `webhook` is a top-level field next to `input`: + +```bash +curl -s -X POST https://api.runpod.ai/v2/ENDPOINT_ID/run \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "input": {"prompt": "painting of a cat"}, + "webhook": "https://example.com/webhooks/runpod/UNGUESSABLE_TOKEN" + }' +``` + +Runpod posts the job result to your URL when the job finishes. Return `200`. If the call fails, Runpod retries up to 2 more times with a 10-second delay. Update your receiver for the new payload: `status` is `COMPLETED` or `FAILED` instead of `succeeded` or `failed`, and there are no `start`, `output`, or `logs` events. Replace Replicate's signature verification (`validateWebhook`, or the `webhook-signature` HMAC check) with your own check, such as a token in the URL path plus a `GET /status/JOB_ID` call to confirm the job. + +### Streaming + +Before: + +```python +import replicate + +for event in replicate.stream( + "meta/meta-llama-3-70b-instruct", + input={"prompt": "Write a haiku about llamas."}, +): + print(str(event), end="") +``` + +After, for a Public Endpoint LLM or a vLLM worker, use the OpenAI-compatible route: + +```python +import os + +from openai import OpenAI + +client = OpenAI( + api_key=os.environ["RUNPOD_API_KEY"], + base_url="https://api.runpod.ai/v2/qwen3-32b-awq/openai/v1", +) + +stream = client.chat.completions.create( + model="Qwen/Qwen3-32B-AWQ", + messages=[{"role": "user", "content": "Write a haiku about llamas."}], + stream=True, +) +for chunk in stream: + if chunk.choices and chunk.choices[0].delta.content: + print(chunk.choices[0].delta.content, end="") +``` + +After, for your own generator handler, submit with `/run` and read `/stream`: + +```python +import runpod + +endpoint = runpod.Endpoint("ENDPOINT_ID") + +run_request = endpoint.run({"prompt": "Write a haiku about llamas."}) +for chunk in run_request.stream(): + print(chunk, end="") +``` + +For your own vLLM endpoint, the base URL is `https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1`. See [OpenAI compatibility](/serverless/vllm/openai-compatibility). + +### Deployments + +Calls to `POST /v1/deployments/OWNER/NAME/predictions` (or `replicate.deployments.predictions.create`) become calls to your endpoint's `/run` or `/runsync`. The request body keeps its `input` object, and `webhook` stays a top-level field. + +### Calls from a runpod.App + +If the caller is Python and the target is a deployed `runpod.App`, you can call the function by name instead of by endpoint ID: + +```python +import runpod + +generate = runpod.Queue(app="sdxl-turbo", name="generate") +result = generate.remote(prompt="an iguana on the beach, pointillism") +``` + +### Control-plane automation + +Scripts that manage Replicate resources map to runpodctl or REST v2. See the [API map](#api-map) below. The most common change is CI: replace `cog login` and `cog push r8.im/OWNER/MODEL` with `docker build`, `docker push`, and an update of the template's image (`runpodctl template update TEMPLATE_ID --image NEW_IMAGE`), or connect the repository through the [GitHub integration](/serverless/workers/github-integration). + +## Step 8: Verify, cut over, and shut down + + + + Call the Runpod endpoint from the same environment your application runs in, with a production-shaped input: + + ```bash + curl -s -X POST "https://api.runpod.ai/v2/ENDPOINT_ID/runsync?wait=120000" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"input": {"prompt": "a lighthouse at dawn", "num_inference_steps": 2, "seed": 42}}' + ``` + + The first request includes a cold start. Check that `status` is `COMPLETED`, and note `delayTime` (queue time, including cold start) and `executionTime`, both in milliseconds. + + + + Run the same inputs with fixed seeds on both platforms. Compare outputs, compare `executionTime` with Replicate's `metrics.predict_time`, and measure cold and warm latency separately. Differences in GPU, library versions, and model variant all change results, so pin versions before you conclude anything. + + + + Send an invalid input and confirm you get the error shape your client expects. Send a job that exceeds your execution timeout and confirm it ends in `TIMED_OUT`. Trigger a webhook and confirm your receiver handles the Runpod payload. + + + + Put the provider behind a flag in your client and send a small share of traffic to Runpod. Watch the endpoint's **Metrics** tab for delay time, execution time, and failures, and [worker logs](/serverless/development/logs) for errors. Increase the share as the numbers hold, and raise max workers as traffic moves. + + + + Once Replicate receives no traffic, scale each deployment to zero, then delete it. Replicate only deletes deployments that have been offline and unused for at least 15 minutes: + + ```bash + curl -s -X PATCH https://api.replicate.com/v1/deployments/OWNER/DEPLOYMENT_NAME \ + -H "Authorization: Bearer $REPLICATE_API_TOKEN" \ + -H "Content-Type: application/json" \ + -d '{"min_instances": 0, "max_instances": 0}' + + # At least 15 minutes later: + curl -s -X DELETE https://api.replicate.com/v1/deployments/OWNER/DEPLOYMENT_NAME \ + -H "Authorization: Bearer $REPLICATE_API_TOKEN" + ``` + + If Replicate rejects `max_instances` of 0 for your deployment, set the instance counts in the deployment's **Settings** tab instead. Private models bill while their instances are online, so confirm on your Replicate dashboard that nothing is still running. Download any fine-tuned weights you want to keep before you delete models, then disable the API tokens your application used at `replicate.com/account/api-tokens`. + + + + Keep active workers at 0 unless you need warm capacity, and set max workers to the concurrency you intend to pay for. Set an execution timeout so a stuck job can't run indefinitely. Runpod also reduces max workers on endpoints that receive no requests for several days. Review your account's [spending limit](/accounts-billing/billing#spending-limits), and delete network volumes you no longer need, since they bill for storage until deleted. + + + +## Reference + +### API map + +| Replicate | Runpod | +| --- | --- | +| `POST /v1/predictions` (with `version`) | `POST https://api.runpod.ai/v2/ENDPOINT_ID/run` | +| `POST /v1/models/OWNER/NAME/predictions` | `POST https://api.runpod.ai/v2/ENDPOINT_ID/run` (Public Endpoint ID or your own) | +| `POST /v1/deployments/OWNER/NAME/predictions` | `POST https://api.runpod.ai/v2/ENDPOINT_ID/run` | +| `Prefer: wait` or `Prefer: wait=N` | `POST .../runsync`, optionally with `?wait=MILLISECONDS` | +| `GET /v1/predictions/PREDICTION_ID` | `GET .../status/JOB_ID` | +| `POST /v1/predictions/PREDICTION_ID/cancel` | `POST .../cancel/JOB_ID` | +| `GET urls.stream` (server-sent events) | `GET .../stream/JOB_ID`, or `/openai/v1` with `stream=True` for LLMs | +| `Cancel-After` header | `policy.executionTimeout` and `policy.ttl` in the request body (milliseconds) | +| `webhook`, `webhook_events_filter` | `webhook` (completion only) | +| `POST /v1/deployments` | `POST https://api.runpod.io/v2/serverless`, or `runpodctl serverless create` | +| `GET /v1/deployments`, `GET /v1/deployments/OWNER/NAME` | `GET https://api.runpod.io/v2/serverless`, `GET .../v2/serverless/ENDPOINT_ID`, or `runpodctl serverless list` and `get` | +| `PATCH /v1/deployments/OWNER/NAME` | `PATCH https://api.runpod.io/v2/serverless/ENDPOINT_ID`, or `runpodctl serverless update` | +| `DELETE /v1/deployments/OWNER/NAME` | `DELETE https://api.runpod.io/v2/serverless/ENDPOINT_ID`, or `runpodctl serverless delete` | +| `GET /v1/hardware` | `GET https://api.runpod.io/v2/catalog/gpus`, or `runpodctl gpu list` | +| `POST /v1/models/OWNER/NAME/versions/VERSION_ID/trainings` | `runpodctl pod create`, or an `@app.task` call | +| `GET /v1/models`, `GET /v1/collections` | [Public Endpoints model reference](/public-endpoints/reference), `runpodctl hub list` and `search` | +| `GET /v1/account` | `runpodctl user` | + +### CLI command map + +| Cog or Replicate | Runpod | +| --- | --- | +| `cog init` | `rp flash init my-app` (SDK app), or write `handler.py` and a `Dockerfile` | +| `cog predict -i key=value` | `python handler.py --test_input '{"input": {"key": "value"}}'`, or `rp flash dev main.py` | +| `cog build -t IMAGE` | `docker build --platform linux/amd64 -t IMAGE .` | +| `cog login` | `docker login` for your registry, and `rp login` or `runpodctl doctor` for Runpod | +| `cog push r8.im/OWNER/MODEL` | `docker push IMAGE`, then `runpodctl template update TEMPLATE_ID --image IMAGE`. Or `rp flash deploy`. | +| `cog serve` | `python handler.py --rp_serve_api` (see [Local testing](/serverless/development/local-testing)) | + +### Configuration field map + +| Replicate or Cog field | Runpod field | +| --- | --- | +| Deployment `name` | Endpoint `name` (`--name`) | +| Deployment `model` and `version` | Template `image` (`--image`), or `image` in REST v2 | +| Deployment `hardware` | `--gpu-id`, REST v2 `gpu.pools`, SDK `gpu=` | +| Multi-GPU SKU (`-2x`, `-4x`, `-8x`) | `--gpu-count`, REST v2 `gpu.count`, SDK `gpu_count=` | +| Deployment `min_instances` | `--workers-min`, REST v2 `workers.min`, SDK `workers=(min, max)` | +| Deployment `max_instances` | `--workers-max`, REST v2 `workers.max`, SDK `workers=(min, max)` | +| `cog.yaml` `build.gpu` | GPU endpoint (the default compute type) | +| `cog.yaml` `build.python_version` | Base image in the `Dockerfile`, or `rp flash deploy --python-version` | +| `cog.yaml` `build.python_requirements`, `python_packages` | `requirements.txt` and `pip install` in the `Dockerfile`, or SDK `dependencies=[...]` | +| `cog.yaml` `build.system_packages` | `apt-get install` in the `Dockerfile`, or SDK `system_dependencies=[...]` | +| `cog.yaml` `build.cuda` | Base image CUDA version, plus `--min-cuda-version` on the endpoint | +| `cog.yaml` `build.run` | `RUN` lines in the `Dockerfile` | +| `cog.yaml` `predict` or `run` | `CMD` in the `Dockerfile` | +| `cog.yaml` `concurrency.max`, `@cog.concurrent(max=N)` | Concurrent handler, or SDK `max_concurrency=` | +| `cog.Secret` input | Endpoint environment variable referencing a Runpod secret | + +### Environment variable map + +| Replicate or Cog | Runpod | Notes | +| --- | --- | --- | +| `REPLICATE_API_TOKEN` (client) | `RUNPOD_API_KEY` (client) | | +| `COG_SETUP_TIMEOUT` | `RUNPOD_INIT_TIMEOUT` | Seconds a worker can take to initialize. Default 420. | +| `COG_MAX_CONCURRENCY` | Concurrency setting in your handler | See [concurrent handlers](/serverless/workers/concurrent-handler). | +| `COG_WEIGHTS` | Your own variable, such as `MODEL_PATH` | Set on the endpoint and read in the handler. | +| Not applicable | `BUCKET_ENDPOINT_URL`, `BUCKET_ACCESS_KEY_ID`, `BUCKET_SECRET_ACCESS_KEY` | Read by `rp_upload`. | +| Not applicable | `RUNPOD_ENDPOINT_ID` | Set by Runpod inside Serverless workers. | + +### Status map + +| Replicate prediction status | Runpod job status | Notes | +| --- | --- | --- | +| `starting` | `IN_QUEUE` | Waiting for a worker, including a worker that's cold starting. | +| `processing` | `IN_PROGRESS` | A worker has picked up the job. | +| `succeeded` | `COMPLETED` | | +| `failed` | `FAILED` | Set when the handler raises an exception. | +| `canceled` (by the user) | `CANCELLED` | Note the spelling. | +| `canceled` (deadline passed after starting) | `TIMED_OUT` | Execution timeout reached. | +| `aborted` (deadline passed before starting) | `TIMED_OUT` | The job expired before a worker picked it up. A job whose TTL has expired returns 404 from `/status`. | + +## Migrate with a coding agent + +A coding agent with the Runpod skills and MCP server can do the inventory, write the handler, deploy the endpoint, and update your client code. Install both first. + + + + ```bash + # The plugin also connects the hosted MCP server + claude plugin marketplace add runpod/runpod-plugins-official + claude plugin install runpod@runpod + ``` + + + + ```bash + # Skills, then the hosted MCP server (Codex, Cursor, and others) + npx skills add runpod/runpod-plugins-official --skill '*' --agent AGENT_ID --global + npx @runpod/mcp-server@latest add --client CLIENT_ID + ``` + + + +The MCP server signs in with OAuth ("Sign in with Runpod"), so you don't need an API key to start. See [Agent skills](/get-started/agent-skills) and [MCP servers](/get-started/mcp-servers) for agent and client IDs. + +Then give your agent this prompt from the root of the repository that uses Replicate: + +```text +Use the runpod-cloud-migrate skill to migrate this project from Replicate to Runpod. + +1. Inventory first, and change nothing yet. Scan the codebase for Replicate usage + (replicate.run, predictions.create, deployments, trainings, webhooks, streaming, + api.replicate.com, REPLICATE_API_TOKEN, replicate.delivery URLs, cog.yaml, predict.py, + cog push in CI). If REPLICATE_API_TOKEN is set, list my deployments with their hardware, + min_instances, and max_instances through the Replicate HTTP API. +2. For each workload, choose a Runpod target: a Public Endpoint if Runpod hosts the model, + a Hub repo, a Serverless queue-based endpoint with a native handler ported from + predict.py, or a runpod.App with @app.queue. Map each Replicate hardware SKU to a Runpod + GPU pool by VRAM. +3. Show me the plan as a table: workload, Replicate setup, Runpod target, GPU pool, worker + range, how file inputs and outputs will be handled, and an estimated cost from Runpod's + published pricing. List every difference that needs my decision (output file storage, + webhook verification, result retention, secret inputs). +4. Wait for my approval before creating anything billable on Runpod. +5. Migrate one workload end to end: write the handler and Dockerfile (or the runpod.App), + deploy with active workers at 0, and update the client code behind a flag. +6. Verify with a real request, and show me the output, delayTime, and executionTime next to + the same request on Replicate. +7. Never delete or scale down anything on Replicate without my explicit approval. +``` + +## Troubleshooting + +**The request returns 400 or the handler reports missing input.** The body must be `{"input": {...}}`. Remove Replicate-only fields such as `version` and `stream` from the top level. With a `runpod.App`, the keys inside `input` must match the function's parameter names. + +**The job stays `IN_QUEUE`.** No worker is available. Check that max workers is above 0, that the GPU pool is available in the allowed data centers (`runpodctl datacenter list`), and that the image pulls. Add a second GPU pool, or remove a data center restriction. An endpoint that received no requests for several days may have had its max workers reduced, so raise it again. + +**The worker starts and then exits, or never becomes ready.** The image was built for the wrong architecture, the handler isn't started, or the model takes too long to load. Build with `--platform linux/amd64`, make sure the container runs `python -u /handler.py` and that the file calls `runpod.serverless.start`, and raise `RUNPOD_INIT_TIMEOUT` for slow model loads. Read the worker logs in the endpoint's **Logs** tab. + +**Every request is slow.** The model is loading inside `handler()`. Move it to module level so it loads once per worker, as `setup()` did in Cog. + +**The job ends in `TIMED_OUT` although the model works.** The job ran past the endpoint's execution timeout, or it expired in the queue. Raise the execution timeout on the endpoint or set `policy.executionTimeout` on the request. Replicate allowed 30 minutes by default, and the Runpod console default is 10 minutes. + +**`/status` returns 404, or the output is missing.** The result expired. Results are kept for 30 minutes after an async job completes and 1 minute after a sync job. Fetch sooner, use a webhook, or write outputs to your own bucket. + +**The response fails with a payload size error, or the client receives a truncated result.** The output is over the 10 MB (`/run`) or 20 MB (`/runsync`) limit, usually because of base64 files. Upload to a bucket and return the URL. + +**`rp_upload` fails or returns an unusable URL.** Check that `BUCKET_ENDPOINT_URL`, `BUCKET_ACCESS_KEY_ID`, and `BUCKET_SECRET_ACCESS_KEY` are set on the endpoint, and that `BUCKET_ENDPOINT_URL` includes the bucket name. Workers read environment variables at start, so new values apply to new workers. + +**CUDA or driver errors on some workers only.** Workers in a pool can land on hosts with different driver versions. Set a minimum CUDA version on the endpoint (`--min-cuda-version 12.8`, or `gpu.minCudaVersion` in REST v2) that matches your image. + +**Client code breaks on status checks.** Runpod statuses are uppercase and spelled differently: `COMPLETED`, not `succeeded`, and `CANCELLED` with two Ls. Update comparisons using the [status map](#status-map). + +## Next steps + + + + Write standard, streaming, and concurrent handlers. + + + + Tune workers, timeouts, FlashBoot, and GPU priority. + + + + See every hosted model, its parameters, and its pricing. + + + + Build apps with `@app.queue`, `@app.task`, and `@app.api`. + + diff --git a/migrate/together-ai.mdx b/migrate/together-ai.mdx new file mode 100644 index 000000000..7677c9205 --- /dev/null +++ b/migrate/together-ai.mdx @@ -0,0 +1,1183 @@ +--- +title: "Migrate from Together AI to Runpod" +sidebarTitle: "Together AI" +description: "Move Together AI serverless models, dedicated endpoints, fine-tunes, batch jobs, and GPU clusters to Runpod." +--- + +This guide is for teams that call [Together AI](https://docs.together.ai/intro) from production code and want to run the same workloads on Runpod. Together's [serverless models](https://docs.together.ai/docs/serverless/overview) map to Runpod [Public Endpoints](/public-endpoints/overview) where Runpod hosts the same model family, and to your own [vLLM endpoint](/serverless/vllm/overview) on Runpod Serverless for every other open model. [Dedicated model inference](https://docs.together.ai/docs/dedicated-endpoints/overview) and [dedicated container inference](https://docs.together.ai/docs/dedicated-container-inference) map to Serverless endpoints, [fine-tuning](https://docs.together.ai/docs/fine-tuning/overview) maps to a Pod, the [batch API](https://docs.together.ai/docs/inference/batch/overview) maps to Serverless batch jobs, and [GPU clusters](https://docs.together.ai/docs/gpu-clusters-overview) map to Instant Clusters. At the end, your application calls a Runpod base URL with a Runpod API key, and nothing is left billing on Together. + +Together is an API-first inference company and Runpod is a GPU cloud, so most of this migration is a change of operating model: you stop choosing a model string from a catalog and start running a model on workers you size and scale. The guide is explicit about where that costs you something. + + +You can hand most of this migration to a coding agent. See [Migrate with a coding agent](#migrate-with-a-coding-agent). + + +## How Together AI maps to Runpod + +| Together AI | Runpod equivalent | Notes | +| --- | --- | --- | +| [Serverless models](https://docs.together.ai/docs/serverless/models) (chat, image, video, audio), called by model string | [Public Endpoints](/public-endpoints/reference) for the model families Runpod hosts. A [vLLM endpoint](/serverless/vllm/overview) on Serverless for any other open LLM. | Together's serverless catalog is much larger than the Public Endpoints catalog. For most models, the answer is your own endpoint. | +| [Dedicated model inference](https://docs.together.ai/docs/dedicated-endpoints/concepts) (endpoint, deployment, replica) | Serverless endpoint running the vLLM worker, with [active workers](/serverless/endpoints/endpoint-configurations#active-workers) set to keep it warm | A Together replica is a Runpod worker. `minReplicas` and `maxReplicas` become active workers and max workers. | +| Deployment profile and config (`cr_...`), instance type (`1xnvidia-h100-80gb`) | GPU pool or GPU type on the endpoint, plus vLLM [environment variables](/serverless/vllm/environment-variables) | Together publishes certified model and hardware pairings. On Runpod you pick the GPU and the engine settings. | +| Autoscaling on `inflight_requests`, latency, or utilization metrics | Autoscaling on [queue delay or request count](/serverless/endpoints/endpoint-configurations#auto-scaling-type) | Runpod has two scaling signals. | +| Inactivity timeout (stop the deployment after idle minutes) | Scale to zero by default, with an [idle timeout](/serverless/endpoints/endpoint-configurations#idle-timeout) in seconds | A Runpod endpoint restarts on the next request. A stopped Together deployment waits for you to restart it. | +| [Dedicated container inference](https://docs.together.ai/docs/dedicated-container-inference) (Jig CLI, Sprocket SDK, queue API) | Serverless [queue-based endpoint](/serverless/endpoints/send-requests) with a [handler function](/serverless/workers/handler-functions), or a [load balancing endpoint](/serverless/load-balancing/overview) for your own HTTP server | Sprocket's `setup()` and `predict()` become module-level loading and a `handler(job)` function. | +| Model registry (`ml_...`), `tg models upload` | Hugging Face plus [cached models](/serverless/endpoints/model-caching), or a [network volume](/storage/network-volumes) | Runpod serves any model vLLM supports, not only fine-tunes of a supported base model. | +| [Fine-tuning](https://docs.together.ai/docs/fine-tuning/overview) jobs (LoRA and full) | A training script on a [Pod](/pods/overview), the [fine-tuning Pod](/fine-tune) (Axolotl), or `@app.task` in the [Runpod SDK](/sdk/overview) | Runpod gives you the GPU and the environment. You own the training loop and its hyperparameters. | +| [Batch API](https://docs.together.ai/docs/inference/batch/overview) (JSONL file, 24-hour window) | Serverless [batch jobs](/serverless/batch-jobs) on a queue-based endpoint, or `/run` with a [webhook](/serverless/endpoints/send-requests#webhook-notifications) | Runpod batches take a JSON array of requests, not an uploaded file. | +| [GPU clusters](https://docs.together.ai/docs/gpu-clusters-overview) (Kubernetes or Slurm, reserved or on-demand) | [Instant Clusters](/instant-clusters) (Slurm, PyTorch, Ray) | Instant Clusters are on-demand multi-node clusters. Kubernetes is not one of the managed cluster types. | +| Cluster shared volumes | [Network volumes](/storage/network-volumes) | A network volume is pinned to one data center and mounts into every node of a cluster in that data center. | +| Jig volumes (versioned, uploaded files) | Network volume, or weights baked into the image | Mounts at `/runpod-volume` on Serverless workers. | +| Jig secrets | [Secrets](/pods/templates/secrets) and endpoint environment variables | | +| [Code sandbox](https://docs.together.ai/docs/together-code-sandbox) | A CPU or GPU [Pod](/pods/overview) | See [What has no direct equivalent](#what-has-no-direct-equivalent). | +| Regions (placement profiles, `us-api.together.ai`) | [Data centers](/serverless/endpoints/endpoint-configurations#data-centers) on the endpoint | You choose data centers on each endpoint you create. | +| Per-token, per-megapixel, per-second serverless billing | Public Endpoints bill per token or per output. Serverless bills [per second of worker time](/serverless/pricing). | The unit changes for anything you host yourself. See [What works differently](#what-works-differently). | +| Per-minute billing per ready replica | Per-second billing from worker start to worker stop | Runpod bills model load time and the idle timeout window. Together doesn't bill provisioning or cold-start time. | +| Reserved capacity, provisioned throughput | Active workers, and custom pricing through sales | | +| Prepaid credits, cut-off at zero balance | Prepaid credits, plus a default [spend limit](/accounts-billing/billing#spending-limits) per hour | | +| `tg` CLI (`together[cli]`) | [`runpodctl`](/runpodctl/overview), and the `rp` CLI for apps written with the Runpod SDK | | +| `together` Python SDK, `together-ai` TypeScript SDK | The OpenAI SDKs against a Runpod base URL, the [Runpod SDK](/sdk/overview) (`pip install runpod`), and [`@runpod/ai-sdk-provider`](/public-endpoints/ai-sdk) for the Vercel AI SDK | | +| Management API at `https://api.together.ai` | [REST API v2](/api-reference-v2/overview) at `https://api.runpod.io/v2` | | +| Infrastructure as code | The Runpod [Terraform provider](/integrations/terraform/overview), built on REST API v2 | | +| `TOGETHER_API_KEY`, scoped to a project | `RUNPOD_API_KEY`, with [per-endpoint restrictions](/get-started/credentials) available on a key | | +| Organizations and projects | [Organizations](/organizations/orgs-overview) and teams | | +| Endpoint events, dashboards, Grafana | Endpoint and worker [logs](/serverless/development/logs), and request, worker, and cold-start metrics in the console and REST API v2 | | + +## What's different + +### What works differently + +**You deploy the model before you call it.** On Together serverless, a model string is enough. On Runpod, unless the model is a Public Endpoint, you create a Serverless endpoint first and call its endpoint ID. Plan for one endpoint per model you use. + +**The base URL carries the endpoint.** Together uses one base URL (`https://api.together.ai/v1`, or `https://api-inference.together.ai/v1` for dedicated endpoints) and selects the model with the `model` field. A Runpod vLLM endpoint has its own base URL, `https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1`, and `model` must be the Hugging Face ID you deployed or the name you set with `OPENAI_SERVED_MODEL_NAME_OVERRIDE`. Code that switches models by changing one string needs a small routing table from model to base URL. + +**Billing moves from tokens to worker seconds.** A self-hosted endpoint bills for every second a worker runs: loading the model, processing requests, and waiting out the idle timeout. Token counts don't appear on the bill. Whether that works in your favor depends on utilization: a worker that batches many concurrent requests spreads its cost across all of them, and a worker that wakes up for one short request pays for the cold start too. Measure with your own traffic before you commit. Public Endpoints keep usage-based pricing. + +**Cold starts are yours to manage.** Together serverless models are always loaded. A Runpod endpoint with zero active workers scales to zero, and the first request after that waits for a worker to start and load the weights. A synchronous request that waits on a cold worker can exceed the gateway's time limit and fail with a 524. You control this with active workers (always warm, always billed), the idle timeout, [cached models](/serverless/endpoints/model-caching), and FlashBoot. See [Step 6](#step-6-deploy-on-runpod). + +**Model features are engine settings.** Together's catalog marks each model for function calling and structured outputs. On a vLLM endpoint you turn tool calling on with `ENABLE_AUTO_TOOL_CHOICE` and `TOOL_CALL_PARSER`, set a reasoning parser with `REASONING_PARSER`, and set context length with `MAX_MODEL_LEN`. The right values depend on the model. See [Configure vLLM to work with your model](/serverless/vllm/configuration). + +**You pick the quantization.** Together's catalog lists the quantization each serverless model runs at. On Runpod, you get whatever precision the Hugging Face repository you deploy contains, so outputs can differ slightly from Together's for the same model family. Compare outputs on your own evaluation set. + +**Rate limits are per endpoint.** Together applies dynamic limits and may return `429` or `503` under load. Runpod applies documented [per-endpoint limits](/serverless/endpoints/send-requests#rate-limits) to each operation, and real throughput is bounded by your max workers and each worker's concurrency (`MAX_CONCURRENCY` on the vLLM worker). + +**Async jobs replace long HTTP calls.** Runpod queue-based endpoints have a native job API (`/run`, `/status`, `/stream`, `/cancel`) with webhooks, alongside the OpenAI-compatible route. Together's video API and container queue API follow a similar submit-and-poll pattern, so those translate directly. + +### What has no direct equivalent + +| On Together AI | On Runpod | +| --- | --- | +| A serverless catalog of 100+ models callable with no deployment | Public Endpoints cover a smaller set. Deploy other open models yourself on Serverless. Proprietary models that Together resells (for example, image and video models from third-party providers) are only available on Runpod if they're listed as a [Public Endpoint](/public-endpoints/reference). | +| Certified deployment profiles (model, quantization, hardware, and optimizations chosen for you) | No equivalent. Use the [vLLM GPU sizing guidance](/serverless/vllm/configuration#gpu-selection) and test. | +| Traffic splitting, A/B tests, shadow experiments, and metric-gated rollouts on one endpoint | No built-in equivalent. Run two endpoints and split traffic in your own gateway or client. Changing an endpoint's image or environment rolls workers to a new release. | +| `/v1/embeddings` and `/v1/rerank` on dedicated endpoints | The vLLM worker documents `/chat/completions`, `/completions`, and `/models`. For embeddings or reranking, run an embedding server on a [load balancing endpoint](/serverless/load-balancing/overview), or look for an embedding worker in the [Hub](/hub/overview). | +| OpenAI Responses API and Anthropic Messages API | Not documented for the vLLM worker. Use chat completions. | +| Managed fine-tuning jobs with a dataset upload and a job ID | No managed job API. Run training on a Pod. See [Fine-tunes](#fine-tunes). | +| Automatic prompt caching with a discounted cached-token price | No token pricing on your own endpoint. vLLM prefix caching is a setting (`ENABLE_PREFIX_CACHING`) that affects speed, not billing. | +| Geo-specific inference through `us-api.together.ai` | Set data centers on your own endpoint. | +| Kubernetes GPU clusters with `kubectl` access | Instant Clusters offer Slurm, PyTorch, and Ray. To keep Kubernetes, run your own distribution on cluster nodes or Pods. | +| Reserved clusters with a fixed term, and preemptible cluster nodes | Instant Clusters are on-demand. Stock varies by data center, and no capacity is guaranteed. Talk to sales about larger or longer commitments. | +| Code sandbox snapshots and sandboxes | No managed equivalent. Use a Pod and remove it when you're done. | + +| On Runpod | On Together AI | +| --- | --- | +| Any Hugging Face model that vLLM supports, including architectures Together doesn't serve | Dedicated uploads must be fine-tuned variants of a supported base model. | +| Any container as a worker without a sales-gated product | Dedicated container inference requires access through Together's sales team. | +| A wide range of GPU sizes, from 16 GB cards upward, for small models | Dedicated model inference lists H100-class hardware and larger. | +| Scale to zero and automatic restart on the next request | A stopped deployment must be restarted by raising its replica bounds. | + +## Requirements + +- A [Runpod account](/accounts-billing/manage-accounts) with credits and a [Runpod API key](/get-started/credentials). +- Access to your Together AI projects, with the Together CLI 3.0 or later (`uv tool install "together[cli]" --upgrade`) and `TOGETHER_API_KEY` set for each project you're migrating. +- A [Hugging Face](https://huggingface.co) account and access token if you use gated models or need a place to store fine-tuned weights. +- Docker, if you run custom containers (dedicated container inference). +- An evaluation set of real prompts and expected outputs. You'll use it in [Step 8](#step-8-verify-cut-over-and-shut-down) to compare the two platforms. + +## Step 1: Take inventory of what you run on Together AI + +Together API keys are scoped to a project, so repeat this step for each project. Start by confirming which project the CLI is pointed at. + +```bash +tg whoami +tg switch PROJECT_ID # move to another project +``` + +### List billable resources + +Run the following commands and save the JSON. Each one covers a different Together product. + +```bash +# Dedicated model inference (v2): endpoints, then the deployments under each one +tg endpoints ls --json > together-endpoints.json +tg endpoints get ENDPOINT_ID --json + +# Models and adapters you uploaded or fine-tuned +tg models list --json > together-models.json + +# Fine-tuning jobs and their checkpoints +tg fine-tuning list --json > together-finetunes.json +tg fine-tuning list-checkpoints FT_ID + +# Batch jobs and uploaded files +tg batches list --json +tg files list --json + +# GPU clusters and shared volumes +tg clusters list --json > together-clusters.json +tg clusters storage list --json + +# Dedicated container inference deployments +tg beta jig list +``` + +For each dedicated deployment, record the model, the config (hardware and GPU count), `minReplicas` and `maxReplicas`, the scaling metric and target, the inactivity timeout, the placement regions, and the endpoint string (`PROJECT_SLUG/ENDPOINT_NAME`) your code passes as `model`. + +Endpoints created on Together's older v1 API don't appear in CLI 3.0. List them with the v1 REST API. + +```bash +curl -s "https://api.together.ai/v1/endpoints?type=dedicated&mine=true" \ + -H "Authorization: Bearer $TOGETHER_API_KEY" +``` + +For each dedicated container deployment, keep the `pyproject.toml` that Jig deploys from. The `[tool.jig.image]` and `[tool.jig.deploy]` tables hold the start command, GPU type and count, CPU, memory, storage, port, replica bounds, environment variables, and volume mounts. + +For each cluster, record the GPU type, GPU count, cluster type (`KUBERNETES` or `SLURM`), region, billing type, driver and CUDA versions, and the shared volumes attached. + +### Find serverless usage + +Serverless models aren't resources you can list, so the inventory comes from your code and your bill. Together's cost analytics page in the console breaks spend down by project and API key. In your codebase, search for the SDKs, hostnames, and environment variables. + +```bash +grep -rnE \ + "from together import|import together|together-ai|TOGETHER_API_KEY|TOGETHER_PROJECT_ID|api(-inference)?\.together\.(ai|xyz)|us-api\.together\.ai|tg (endpoints|fine-tuning|ft|batches|files|clusters|models|beta jig)|sprocket|\[tool\.jig" \ + --include="*.py" --include="*.ts" --include="*.tsx" --include="*.js" --include="*.mjs" \ + --include="*.go" --include="*.sh" --include="*.toml" --include="*.yaml" --include="*.yml" \ + --include="*.json" --include="*.env*" --include="Dockerfile*" . +``` + +The pattern also matches the `api.together.xyz` hostname in case older code still uses it. Also check frameworks that wrap Together behind their own provider name, such as LangChain, LiteLLM, and the Vercel AI SDK. For every call site, record the model string, the API (chat, completions, images, video, audio, rerank, batch), and the features in use: streaming, tools, `response_format`, image inputs, and `logprobs`. + +## Step 2: Set up Runpod + +Create an API key in the [Runpod console](https://console.runpod.io/user/settings) and export it. See [Manage API keys](/get-started/credentials) for restricted and read-only keys. + +```bash +export RUNPOD_API_KEY="YOUR_API_KEY" +``` + +Install `runpodctl` to create and manage resources from your terminal, and the Runpod SDK if you'll call endpoints from Python or write your own workers. + +```bash +# runpodctl (or: brew install runpod/runpodctl/runpodctl) +curl -fsSL https://cli.runpod.net | sudo bash +runpodctl doctor + +# Runpod SDK 2.x for Python 3.10 to 3.14, which includes the rp CLI +pip install runpod +``` + +`runpodctl doctor` walks through first-time setup of your API key and SSH key. For agent-driven work, connect the [Runpod MCP server](/get-started/mcp-servers). If you manage infrastructure as code, the [Terraform provider](/integrations/terraform/overview) (`runpod/runpod`, `~> 2.0`) covers endpoints, templates, network volumes, Pods, and clusters. + +## Step 3: Choose where each workload lands + +Use this table for every item in your inventory. + +| What you run on Together AI | Where it lands on Runpod | +| --- | --- | +| A serverless chat model that Runpod hosts as a Public Endpoint | [Public Endpoint](/public-endpoints/reference), through its OpenAI-compatible route. No deployment. | +| Any other open chat or completion model | Serverless queue-based endpoint running the [vLLM worker](/serverless/vllm/overview), scaled to zero when idle. | +| A dedicated endpoint with `minReplicas` of 1 or more | The same vLLM endpoint with active workers set to the same number. | +| A fine-tuned model or LoRA adapter served on a dedicated endpoint | vLLM endpoint that loads the weights from Hugging Face or a network volume. | +| A model whose vLLM version is newer than the Hub worker's, or a custom server | Serverless [load balancing endpoint](/serverless/load-balancing/overview) running your own image. | +| A serverless image or video model that Runpod hosts | Public Endpoint, through the native `/run` and `/runsync` API. | +| Other image or video models | A ComfyUI worker from the [Hub](/hub/overview), or your own handler on a queue-based endpoint. | +| Speech-to-text (Whisper) | A Whisper worker from the Hub on a queue-based endpoint. | +| Text-to-speech | A Public Endpoint if one of the listed text-to-speech models fits, otherwise your own handler on a queue-based endpoint. | +| Embeddings or rerank on a dedicated endpoint | Load balancing endpoint running an embedding server, or a Hub embedding worker. | +| Batch API jobs | [Batch jobs](/serverless/batch-jobs) on your vLLM endpoint, or `/run` with webhooks. | +| A Sprocket worker in queue mode | Queue-based endpoint with a handler function. | +| A Sprocket worker in HTTP server mode | Load balancing endpoint. | +| A fine-tuning job | A GPU Pod, the [fine-tuning Pod](/fine-tune), or `@app.task` in the Runpod SDK (one Pod per call that runs to completion). | +| A Slurm GPU cluster | [Instant Cluster](/instant-clusters/slurm-clusters) with managed Slurm. | +| A Kubernetes GPU cluster used for distributed training | Instant Cluster with [PyTorch](/instant-clusters/pytorch) or Slurm. | +| A Kubernetes GPU cluster used for serving | Serverless endpoints, or an Instant Cluster with [Ray and vLLM](/instant-clusters/ray-vllm) for models that span nodes. | +| A code sandbox | A Pod. | + +### Model map for Public Endpoints + +These are the Together serverless model strings that have a Public Endpoint in the same family, based on the Runpod [model reference](/public-endpoints/reference) and model pages. Versions and parameters differ between the two platforms, so read the linked page before you switch. Every Together model not in this table lands on your own endpoint. + +| Together model string | Runpod Public Endpoint | How you call it | +| --- | --- | --- | +| `moonshotai/Kimi-K3` | [Moonshot Kimi](/public-endpoints/models/moonshot-kimi), `model` set to `kimi-k3` | OpenAI-compatible: `https://api.runpod.ai/v2/moonshot-kimi/openai/v1` | +| `moonshotai/Kimi-K3.1-Pro` | [Moonshot Kimi](/public-endpoints/models/moonshot-kimi), `model` set to `kimi-k3.1-pro` | Same base URL as above | +| `openai/gpt-oss-120b` | [GPT-OSS 120B](/public-endpoints/models/gpt-oss-120b). The model page sets `model` to `gpt-oss-120b`, and the [AI coding tools page](/public-endpoints/ai-coding-tools) uses `openai/gpt-oss-120b`, so call `/models` to confirm. | OpenAI-compatible: `https://api.runpod.ai/v2/gpt-oss-120b/openai/v1` | +| Qwen chat models (`Qwen/Qwen3...`) | [Qwen3 32B AWQ](/public-endpoints/models/qwen3-32b), a different Qwen3 model | OpenAI-compatible: `https://api.runpod.ai/v2/qwen3-32b-awq/openai/v1`, `model` set to `Qwen/Qwen3-32B-AWQ` | +| `ByteDance-Seed/Seedream-4.0` | [Seedream 4.0 T2I](/public-endpoints/models/seedream-4-t2i) and [Seedream 4.0 Edit](/public-endpoints/models/seedream-4-edit) | Native: `https://api.runpod.ai/v2/seedream-v4-t2i/runsync` | +| `Qwen/Qwen-Image` | [Qwen Image](/public-endpoints/models/qwen-image) | Native: `https://api.runpod.ai/v2/qwen-image-t2i/runsync` | +| `Wan-AI/Wan2.6-image` | [WAN 2.6 T2I](/public-endpoints/models/wan-2-6-t2i) | Native: `https://api.runpod.ai/v2/wan-2-6-t2i/runsync` | +| `google/flash-image-3.1` (Nano Banana 2) | [Nano Banana 2 Edit](/public-endpoints/models/nano-banana-2-edit), image editing | Native: `https://api.runpod.ai/v2/google-nano-banana-2-edit/runsync` | +| `google/gemini-3-pro-image` (Nano Banana Pro) | [Nano Banana Pro Edit](/public-endpoints/models/nano-banana-pro-edit), image editing | Native: `https://api.runpod.ai/v2/nano-banana-pro-edit/runsync` | +| FLUX models (`black-forest-labs/FLUX...`) | [Flux Dev](/public-endpoints/models/flux-dev), [Flux Schnell](/public-endpoints/models/flux-schnell), and [Flux Kontext Dev](/public-endpoints/models/flux-kontext-dev), which are different FLUX variants | Native: `https://api.runpod.ai/v2/black-forest-labs-flux-1-dev/runsync` | +| `ByteDance/Seedance-2.5` | [Seedance 2.5 T2V](/public-endpoints/models/seedance-2-5-t2v) and [Seedance 2.5 I2V](/public-endpoints/models/seedance-2-5-i2v) | Native. Copy the endpoint URL from the model page. | +| `vidu/vidu-q3` | [Vidu Q3 T2V](/public-endpoints/models/vidu-q3-t2v) and [Vidu Q3 I2V](/public-endpoints/models/vidu-q3-i2v) | Native: `https://api.runpod.ai/v2/vidu-q3-t2v/runsync` | +| Wan video models (`Wan-AI/wan2.7-...`) | WAN 2.1 to 2.6 video endpoints, which are earlier versions | Native. See the [model reference](/public-endpoints/reference). | + +Runpod's text catalog also includes [IBM Granite 4.0](/public-endpoints/models/granite-4), and its audio catalog has [Chatterbox Turbo](/public-endpoints/models/chatterbox-turbo) and [Minimax Speech 02 HD](/public-endpoints/models/minimax-speech) for text-to-speech. None of Together's serverless text-to-speech or speech-to-text models has a Public Endpoint with the same model. The catalog changes, so check the [Hub](https://console.runpod.io/hub?tabSelected=public_endpoints) for the current list. + +### GPU map + +Together's dedicated hardware is listed below against the Runpod GPU type ID you pass to `--gpu-id` and the Serverless pool that contains it, from the [GPU types reference](/references/gpu-types). + +| Together hardware | Where you see it | Runpod GPU type ID | Serverless pool | +| --- | --- | --- | --- | +| H100 80 GB | `1xnvidia-h100-80gb` (dedicated), `h100-80gb` (Jig), `H100_SXM` (clusters) | `NVIDIA H100 80GB HBM3` | `ADA_80_PRO` | +| H200 141 GB | `1xnvidia-h200-140gb` (dedicated), `H200_SXM` (clusters) | `NVIDIA H200` | `HOPPER_141` | +| B200 180 GB | `1xnvidia-b200-180gb` (dedicated), `b200-192gb` (Jig), `B200_SXM` (clusters) | `NVIDIA B200` | Use the GPU type ID | +| B300 | `1xnvidia-b300-280gb` (dedicated), `B300_SXM` (clusters) | `NVIDIA B300 SXM6 AC` | Use the GPU type ID | +| L40 | `L40_PCIE` (clusters) | `NVIDIA L40` | `ADA_48_PRO` | +| CPU only | `gpu_type = "none"` (Jig) | A CPU endpoint (`--compute-type CPU`) | Not applicable | + +You don't have to match Together's hardware. Together's dedicated hardware list starts at H100, and many models fit on smaller cards: the vLLM worker docs suggest 16 to 24 GB for models under 10B parameters and 32 to 48 GB for 10B to 30B. Choose the GPU that fits the model and context length you serve, using the [VRAM estimates](/serverless/vllm/configuration#gpu-selection). Stock varies by data center and changes through the day, so check what's available before you deploy and list more than one pool on production endpoints. + +```bash +runpodctl gpu list +``` + +Instant Clusters support B200, H200, H100, and A100 nodes. See [Instant Clusters](/instant-clusters#supported-hardware). + +## Step 4: Move your images and code + +If you only call Together's model APIs, you have no images to move. Skip to [Step 5](#step-5-move-your-data-and-models). + +### Sprocket workers (dedicated container inference) + +A Sprocket worker implements `setup()`, which runs once at startup, and `predict(args)`, which runs once per job and receives the job's `payload`. A Runpod queue-based worker does the same two things with module-level code and a handler function that receives the request's `input`. + +Here's the worker from Together's container quickstart. + +```python hello_world.py +import os + +import sprocket + + +class HelloWorld(sprocket.Sprocket): + def setup(self) -> None: + self.greeting = "Hello" + + def predict(self, args: dict) -> dict: + name = args.get("name", "world") + return {"message": f"{self.greeting}, {name}!"} + + +if __name__ == "__main__": + queue_name = os.environ.get("TOGETHER_DEPLOYMENT_NAME", "hello-world") + sprocket.run(HelloWorld(), queue_name) +``` + +The same worker on Runpod looks like this. + +```python handler.py +import runpod + +# Runs once when the worker starts. Load models here, not inside the handler. +GREETING = "Hello" + + +def handler(job): + args = job["input"] + name = args.get("name", "world") + return {"message": f"{GREETING}, {name}!"} + + +runpod.serverless.start({"handler": handler}) +``` + +Anything you loaded in `setup()` moves to module level so it loads once per worker. The return value becomes the job's `output`. If `predict()` raises an exception, Sprocket marks the job failed, and an exception in a Runpod handler does the same. Three Sprocket behaviors need explicit code on Runpod: + +- **Progress updates.** Replace `emit_info(...)` with `runpod.serverless.progress_update(job, ...)`. +- **Input files.** Sprocket downloads any HTTPS URL in the payload and hands your code a local path. A Runpod handler receives the URL as sent, so download it in the handler. +- **HTTP server mode.** A Sprocket worker that serves `predict_path` directly becomes a [load balancing endpoint](/serverless/load-balancing/build-a-worker): your own HTTP server with a `/ping` health check. + +### Container image + +Jig builds the image from `pyproject.toml` and pushes it to Together's registry. On Runpod you build and push it yourself, and the image must be `linux/amd64`. A minimal Dockerfile for the handler above: + +```dockerfile Dockerfile +FROM python:3.11-slim + +WORKDIR /app +RUN pip install --no-cache-dir runpod +COPY handler.py . + +CMD ["python", "-u", "handler.py"] +``` + +Build it for the right platform and push it to a registry Runpod can pull from. + +```bash +docker build --platform linux/amd64 -t YOUR_REGISTRY/hello-worker:v1 . +docker push YOUR_REGISTRY/hello-worker:v1 +``` + +For GPU workers, start from a CUDA base image that matches your framework, as you would for Jig. If the registry is private, add its credentials to Runpod and reference them from the template. See [Deploy workers from Docker Hub](/serverless/workers/deploy). Jig's `tg beta jig dockerfile` prints the Dockerfile Jig generates, which is a useful starting point for the system packages your worker needs. + +### Jig configuration + +Translate the `[tool.jig.deploy]` table using the [configuration field map](#configuration-field-map). Environment variables in `[tool.jig.deploy.environment_variables]` become template environment variables, Jig secrets become Runpod [secrets](/pods/templates/secrets), and Jig volume mounts become a network volume (mounted at `/runpod-volume` on Serverless) or files baked into the image. Remove any code that reads `TOGETHER_DEPLOYMENT_NAME` or other Together runtime variables. Inside a Runpod worker, `RUNPOD_ENDPOINT_ID` identifies the endpoint. + +### Prebuilt vLLM images + +If you deploy stock vLLM through Jig, you don't need to port the image. Use the vLLM worker from the Hub in [Step 6](#step-6-deploy-on-runpod), and move your `vllm serve` flags to environment variables: each flag becomes an upper snake case variable (`--max-model-len 8192` becomes `MAX_MODEL_LEN=8192`), and `VLLM_EXTRA_ARGS` passes anything else through. + +## Step 5: Move your data and models + +### Base models + +If you serve a stock open model, there's nothing to copy. The vLLM worker pulls the model from Hugging Face, and [cached models](/serverless/endpoints/model-caching) keep a copy on Runpod hosts so workers don't download it on every cold start. Find the Hugging Face repository that corresponds to your Together model string. Together's IDs follow the same `organization/model` convention but aren't always the Hugging Face repository name, and Together may serve a quantized build. For gated models, accept the license on Hugging Face and create an access token. + +### Fine-tunes + +Together lets you download the weights of a completed fine-tuning job. Choose the checkpoint type that matches how you'll serve it: `merged` gives you self-contained weights, and `adapter` gives you only the LoRA adapter. Full fine-tunes download as the full model. + +```bash +# Merged weights (the default for LoRA jobs) +tg fine-tuning download FT_ID --output-dir ./my-finetune + +# LoRA adapter only +tg fine-tuning download FT_ID --checkpoint-type adapter --output-dir ./my-adapter +``` + +The download is a `.tar.zst` archive. Decompress it to get a standard Hugging Face model directory. + +```bash +cd my-finetune +zstd -d model.tar.zst +tar -xvf model.tar +``` + +For a model you uploaded to Together's registry instead of training there, use `tg models download MODEL_ID ./local-dir`, or go back to the original source. + +Then put the weights where a Runpod worker can load them. You have two documented options. + + + + Create a private Hugging Face repository, then push the directory to it. This works with cached models, so cold starts don't re-download the weights. + + ```bash + hf auth login + hf repos create YOUR_HF_USER/my-finetune --private + hf upload YOUR_HF_USER/my-finetune ./my-finetune + ``` + + Create the repository first. `hf upload` creates a missing repository for you, and a repository created that way isn't private unless you ask for it. + + In Step 6, set `MODEL_NAME` to `YOUR_HF_USER/my-finetune` and add `HF_TOKEN` to the endpoint so workers can read the private repository. + + + Create a network volume in a data center that has the GPU you need and supports the [S3-compatible API](/storage/s3-api), then sync the directory to it. The bucket name is the volume ID. + + ```bash + runpodctl network-volume create --name "models" --size 100 --data-center-id "US-KS-2" + + aws s3 sync --region US-KS-2 \ + --endpoint-url https://s3api-us-ks-2.runpod.io/ \ + ./my-finetune \ + s3://NETWORK_VOLUME_ID/my-finetune + ``` + + The S3 API uses its own access key, which you create in the console. See [S3-compatible API](/storage/s3-api#setup-and-authentication). In Step 6, attach the volume to the endpoint and set `MODEL_NAME` to `/runpod-volume/my-finetune`. A network volume is pinned to one data center, so the endpoint's workers run only there. + + + + +A LoRA adapter served through the vLLM worker's `LORA_MODULES` setting is loaded from Hugging Face. If you can't publish the adapter there, serve the merged weights instead. + + +If you'd rather retrain than transfer, export your training data first. Together keeps uploaded files in its files API (`tg files retrieve-content FILE_ID --output ./data`), and fine-tuning expects JSONL or Parquet, which you can reuse as is. To train on Runpod, start a GPU Pod with a network volume, run your training script, and write the output to the volume so it outlives the Pod. + +```bash +runpodctl network-volume create --name ft-train --size 30 --data-center-id EU-RO-1 + +runpodctl pod create --name ft-train \ + --template-id runpod-torch-v280 --gpu-id "NVIDIA GeForce RTX 4090" \ + --data-center-ids EU-RO-1 \ + --network-volume-id NETWORK_VOLUME_ID --volume-mount-path /workspace \ + --ssh +``` + +The volume mounts at `/workspace` on the Pod and at `/runpod-volume` on Serverless workers, so an adapter written to `/workspace/outputs` during training is at `/runpod-volume/outputs` when you serve it. Delete the Pod when training finishes. For a guided setup with Axolotl, see [Fine-tune a model](/fine-tune). Hyperparameters don't carry over automatically: Together's job settings (epochs, learning rate, LoRA rank) are inputs to your own script now. + +### Batch files and cluster data + +Together keeps batch input, output, and error files for 7 days. Download any results you still need before you shut down (`tg batches download BATCH_ID --output ./out`). + +For GPU clusters, copy the contents of each shared volume to a Runpod network volume. From a node that mounts the shared volume (at `/data` on Slurm clusters), sync to the network volume through the S3-compatible API with the same `aws s3 sync` command shown above. Data on Together's local NVMe disks is ephemeral, so only the shared volume needs to move. + +## Step 6: Deploy on Runpod + +### Public Endpoints + +There's nothing to deploy. Confirm your account has credits, then go to [Step 7](#step-7-update-the-code-that-calls-together-ai). + +### A vLLM endpoint for an open model + +This is the landing spot for most Together serverless models and all dedicated endpoints. The example serves `Qwen/Qwen3-8B` with tool calling enabled. Replace the model, GPU pool, and vLLM settings with the ones for your model from [Configure vLLM to work with your model](/serverless/vllm/configuration). + + + + Look up the Hub listing ID for the vLLM worker, then create the endpoint from it. + + ```bash + runpodctl hub get runpod-workers/worker-vllm + + runpodctl serverless create \ + --hub-id HUB_ID \ + --name "qwen3-8b" \ + --gpu-id AMPERE_24 \ + --workers-min 0 \ + --workers-max 3 \ + --model-reference https://huggingface.co/Qwen/Qwen3-8B:main \ + --env MODEL_NAME=Qwen/Qwen3-8B \ + --env MAX_MODEL_LEN=8192 \ + --env ENABLE_AUTO_TOOL_CHOICE=true \ + --env TOOL_CALL_PARSER=hermes + ``` + + `--hub-id` pulls the worker image, container disk size, and default environment from the Hub release, and `--env` overrides individual values. `--gpu-id` accepts a pool ID or a GPU type ID; set it explicitly so the endpoint runs on the hardware you sized for. `--model-reference` attaches the Hugging Face model as a cached model: the `:main` suffix is the branch, tag, or revision, and the flag requires a GPU endpoint. The command prints the endpoint ID. For a gated or private model, add `--env HF_TOKEN=YOUR_HF_TOKEN`, or store the token as a [secret](/pods/templates/secrets). + + + Open the [vLLM worker](https://console.runpod.io/hub/runpod-workers/worker-vllm) in the Hub and click **Deploy**. Enter the Hugging Face model ID in the **Model** field, expand **Advanced** to set **Max Model Length** and other vLLM settings, click **Next**, then **Create Endpoint**. The full walkthrough is in the [vLLM quickstart](/serverless/vllm/get-started). + + + +Wait until a worker is ready before you send traffic. Creating an endpoint returns immediately, and the first worker still has to pull the image and load the model. + +```bash +curl -s "https://api.runpod.ai/v2/ENDPOINT_ID/health" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" +``` + +The response reports worker and job counts. Make the first request with the asynchronous `/run` route so a slow first load can't time out, then use the OpenAI-compatible route once a worker is warm. + +```bash +curl -s -X POST "https://api.runpod.ai/v2/ENDPOINT_ID/run" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"input": {"messages": [{"role": "user", "content": "Say hello in one word."}], "sampling_params": {"max_tokens": 10}}}' + +curl -s "https://api.runpod.ai/v2/ENDPOINT_ID/status/JOB_ID" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" +``` + +### Match a dedicated endpoint's scaling + +A Together deployment's replica bounds map onto worker counts. Set active workers to your old `minReplicas` and max workers to your old `maxReplicas`. + +Before, on Together: + +```bash +tg endpoints deploy Qwen/Qwen3.5-9B \ + --endpoint my-endpoint \ + --min-replicas 1 \ + --max-replicas 3 \ + --inactive-timeout 60 +``` + +After, on Runpod: + +```bash +runpodctl serverless update ENDPOINT_ID \ + --workers-min 1 \ + --workers-max 3 \ + --scaler-type REQUEST_COUNT \ + --scaler-value 4 +``` + +With `--workers-min 1`, one worker stays loaded and bills continuously, the same trade you made with `minReplicas`. Request count scaling adds workers based on queued plus in-progress requests divided by the scaler value, and it's the recommended scaler for LLM workloads. Runpod has no inactivity timeout because it isn't needed: with `--workers-min 0`, workers stop after the idle timeout and start again on the next request. To cut cold starts without paying for an always-on worker, keep the model cached, leave FlashBoot on, and raise the idle timeout (`--idle-timeout`, in seconds) so a worker stays warm between bursts. REST API v2 rejects an idle timeout on a queue-based endpoint that scales on request count, so pair a long idle timeout with queue delay scaling. + +The same change through REST API v2: + +```bash +curl -s -X PATCH "https://api.runpod.io/v2/serverless/ENDPOINT_ID" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"workers": {"min": 1, "max": 3}}' +``` + + +A deployment that needed several GPUs on Together (for example, a config with a GPU count above one) needs `--gpu-count` on Runpod. On multi-GPU workers, the vLLM worker sets tensor parallelism to the number of visible GPUs. + + +### Fine-tuned models and LoRA adapters + +For merged weights, deploy exactly as above with `MODEL_NAME` pointing at your Hugging Face repository (and `--model-reference` set to the same repository), or at the path on an attached network volume (`--network-volume-id NETWORK_VOLUME_ID` with `MODEL_NAME=/runpod-volume/my-finetune`). + +For a LoRA adapter, deploy the base model and attach the adapter with the vLLM worker's LoRA settings. + +```bash +runpodctl serverless create \ + --hub-id HUB_ID \ + --name "qwen3-8b-lora" \ + --gpu-id AMPERE_24 \ + --model-reference https://huggingface.co/Qwen/Qwen3-8B:main \ + --env MODEL_NAME=Qwen/Qwen3-8B \ + --env ENABLE_LORA=true \ + --env MAX_LORA_RANK=16 \ + --env 'LORA_MODULES=[{"name": "my-adapter", "path": "YOUR_HF_USER/my-adapter"}]' \ + --env HF_TOKEN=YOUR_HF_TOKEN +``` + +`MAX_LORA_RANK` must be at least the rank you trained with (the default is 16). After the endpoint is up, call `client.models.list()` to see the names it serves, and pass the adapter's name as `model` in requests. + +### Custom workers + +For a ported Sprocket worker, create a Serverless template from your image, then an endpoint from the template. + +```bash +runpodctl template create \ + --name "hello-worker" \ + --image "YOUR_REGISTRY/hello-worker:v1" \ + --serverless + +runpodctl serverless create \ + --template-id TEMPLATE_ID \ + --name "hello-worker" \ + --gpu-id "NVIDIA GeForce RTX 4090" \ + --workers-min 0 \ + --workers-max 3 +``` + +For a CPU-only worker (Jig's `gpu_type = "none"`), replace `--gpu-id` with `--compute-type CPU`. Environment variables for a template-based endpoint come from the template (`runpodctl template create --env`), not from `serverless create --env`. + +### Image, video, and audio workers + +For models that aren't Public Endpoints, deploy a prebuilt worker from the Hub. Search for one, check its release and GPU pool, and deploy it the same way as the vLLM worker. + +```bash +runpodctl hub search comfyui --type SERVERLESS +runpodctl hub search whisper --type SERVERLESS +runpodctl serverless create --hub-id HUB_ID --name "whisper" --workers-min 0 --workers-max 3 +``` + +Each Hub worker defines its own input schema, so read the listing before you write client code. See [Deploy ComfyUI on Serverless](/tutorials/serverless/comfyui) for an image generation walkthrough. + +### Instant Clusters + +Deploy a cluster from the [Instant Clusters page](https://console.runpod.io/cluster) in the console: choose the cluster type (Slurm, PyTorch, or Ray), GPU type, node count, region, and network volume. See [Slurm Clusters](/instant-clusters/slurm-clusters) for the Slurm flow. `sbatch` and `srun` scripts from a Together Slurm cluster carry over, with paths changed from `/data` to wherever you mount the network volume. Kubernetes manifests don't carry over: convert each job to a Slurm script or a container start command. Clusters are also available through REST API v2 (`POST /v2/clusters`) and Terraform (`runpod_cluster`). + +## Step 7: Update the code that calls Together AI + +### Chat completions through the OpenAI SDK + +If you already call Together through an OpenAI client, change the API key, the base URL, and the model name. + + + + Before: + + ```python + import os + + from openai import OpenAI + + client = OpenAI( + api_key=os.environ["TOGETHER_API_KEY"], + base_url="https://api.together.ai/v1", + ) + + response = client.chat.completions.create( + model="Qwen/Qwen3.5-9B", + messages=[{"role": "user", "content": "What are the top 3 things to do in New York?"}], + max_tokens=512, + ) + print(response.choices[0].message.content) + ``` + + After: + + ```python + import os + + from openai import OpenAI + + client = OpenAI( + api_key=os.environ["RUNPOD_API_KEY"], + base_url="https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1", + timeout=300, + ) + + response = client.chat.completions.create( + model="Qwen/Qwen3-8B", + messages=[{"role": "user", "content": "What are the top 3 things to do in New York?"}], + max_tokens=512, + ) + print(response.choices[0].message.content) + ``` + + + Before: + + ```typescript + import OpenAI from "openai"; + + const client = new OpenAI({ + apiKey: process.env.TOGETHER_API_KEY, + baseURL: "https://api.together.ai/v1", + }); + + const response = await client.chat.completions.create({ + model: "Qwen/Qwen3.5-9B", + messages: [{ role: "user", content: "What are the top 3 things to do in New York?" }], + max_tokens: 512, + }); + console.log(response.choices[0].message.content); + ``` + + After: + + ```typescript + import OpenAI from "openai"; + + const client = new OpenAI({ + apiKey: process.env.RUNPOD_API_KEY, + baseURL: "https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1", + timeout: 300_000, + }); + + const response = await client.chat.completions.create({ + model: "Qwen/Qwen3-8B", + messages: [{ role: "user", content: "What are the top 3 things to do in New York?" }], + max_tokens: 512, + }); + console.log(response.choices[0].message.content); + ``` + + + Before: + + ```bash + curl https://api.together.ai/v1/chat/completions \ + -H "Authorization: Bearer $TOGETHER_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "model": "Qwen/Qwen3.5-9B", + "messages": [{"role": "user", "content": "Hello!"}] + }' + ``` + + After: + + ```bash + curl https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1/chat/completions \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "model": "Qwen/Qwen3-8B", + "messages": [{"role": "user", "content": "Hello!"}], + "max_tokens": 512 + }' + ``` + + + +Set `max_tokens` on every request. The vLLM worker's documented default is 16 tokens, which truncates most answers. The longer client timeout covers a request that arrives while a worker is starting. To keep your existing model strings in client code, set `OPENAI_SERVED_MODEL_NAME_OVERRIDE` on the endpoint to the name your code already sends. + +For a model in the [Public Endpoints map](#model-map-for-public-endpoints), the change is the same, with the endpoint's slug in the base URL. + +```python +import os + +from openai import OpenAI + +client = OpenAI( + api_key=os.environ["RUNPOD_API_KEY"], + base_url="https://api.runpod.ai/v2/moonshot-kimi/openai/v1", +) + +response = client.chat.completions.create( + model="kimi-k3", + messages=[{"role": "user", "content": "What is Runpod?"}], + max_tokens=2048, +) +print(response.choices[0].message.content) +``` + +### The `together` SDKs + +Runpod doesn't document pointing the `together` or `together-ai` packages at a Runpod base URL, so replace them with the OpenAI SDK for chat and completions. The method names and response shapes are the same for these calls. + + + + Before: + + ```python + from together import Together + + client = Together() # reads TOGETHER_API_KEY + + stream = client.chat.completions.create( + model="Qwen/Qwen3.5-9B", + messages=[{"role": "user", "content": "Write a haiku about GPUs."}], + stream=True, + ) + for chunk in stream: + if chunk.choices and chunk.choices[0].delta.content: + print(chunk.choices[0].delta.content, end="", flush=True) + ``` + + After: + + ```python + import os + + from openai import OpenAI + + client = OpenAI( + api_key=os.environ["RUNPOD_API_KEY"], + base_url="https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1", + ) + + stream = client.chat.completions.create( + model="Qwen/Qwen3-8B", + messages=[{"role": "user", "content": "Write a haiku about GPUs."}], + max_tokens=200, + stream=True, + ) + for chunk in stream: + if chunk.choices and chunk.choices[0].delta.content: + print(chunk.choices[0].delta.content, end="", flush=True) + ``` + + + Before: + + ```typescript + import Together from "together-ai"; + + const together = new Together(); // reads TOGETHER_API_KEY + + const stream = await together.chat.completions.create({ + model: "Qwen/Qwen3.5-9B", + messages: [{ role: "user", content: "Write a haiku about GPUs." }], + stream: true, + }); + for await (const chunk of stream) { + process.stdout.write(chunk.choices[0]?.delta?.content ?? ""); + } + ``` + + After: + + ```typescript + import OpenAI from "openai"; + + const client = new OpenAI({ + apiKey: process.env.RUNPOD_API_KEY, + baseURL: "https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1", + }); + + const stream = await client.chat.completions.create({ + model: "Qwen/Qwen3-8B", + messages: [{ role: "user", content: "Write a haiku about GPUs." }], + max_tokens: 200, + stream: true, + }); + for await (const chunk of stream) { + process.stdout.write(chunk.choices[0]?.delta?.content ?? ""); + } + ``` + + + +Dedicated endpoint code changes the same way. Replace `Together(base_url="https://api-inference.together.ai")` and the endpoint string (`PROJECT_SLUG/ENDPOINT_NAME`) with the Runpod base URL and the served model name. Calls to `client.images`, `client.videos`, `client.audio`, `client.rerank`, `client.batches`, `client.files`, `client.fine_tuning`, and `client.endpoints` have no drop-in replacement and are covered below. + +If you use the Vercel AI SDK, replace the Together provider with [`@runpod/ai-sdk-provider`](/public-endpoints/ai-sdk), which works with Public Endpoints and with your own endpoint IDs. + +### Feature parity for the OpenAI-compatible surface + +| Feature | Together AI | Runpod vLLM endpoint | Runpod Public Endpoints | +| --- | --- | --- | --- | +| Chat completions | Supported | Supported (`/chat/completions`) | OpenAI-compatible route documented for Qwen3 32B AWQ, Moonshot Kimi, GPT-OSS 120B, and Cogito 671B. Granite 4.0 documents the native API only. | +| Text completions | Supported (legacy) | Supported (`/completions`) | Native `prompt` input on Qwen3 32B AWQ, GPT-OSS 120B, and Cogito 671B. For the OpenAI route, see the model page. | +| Model list | Supported | Supported (`/models`) | See the model page. | +| Streaming | Supported | Supported (`stream=True`; keep `RAW_OPENAI_OUTPUT=1`) | Supported on the OpenAI-compatible route. | +| Tool calling | Supported on models marked for function calling | Depends on the model. Set `ENABLE_AUTO_TOOL_CHOICE=true` and the matching `TOOL_CALL_PARSER`. | Documented for GPT-OSS 120B. For other models, check the model. | +| Structured outputs (`response_format`) | Supported on models marked for structured outputs (`json_schema`) | Not listed in the worker docs. vLLM guided decoding is configurable with `GUIDED_DECODING_BACKEND`. Test it with your model. | See the model page. | +| Vision (image inputs) | Supported on vision models | Depends on the model | Check the model. Kimi K2.6 lists multimodal input. | +| Reasoning output | `reasoning` or `reasoning_content`, depending on the model | Set `REASONING_PARSER` for the model. The field name depends on the vLLM version. | Kimi supports `reasoning_effort`. | +| Log probabilities | Supported, in Together's own response shape | `MAX_LOGPROBS` sets the limit (default 20). Test the response shape against your parser. | See the model page. | +| Embeddings | Dedicated endpoints only (`/v1/embeddings`) | Not among the documented routes | None listed | +| Rerank | Dedicated endpoints only (`/v1/rerank`) | Not among the documented routes | None listed | +| Batch | Together's batch API (JSONL file) | [Batch jobs](/serverless/batch-jobs) or `/run` with a webhook | Asynchronous `/run` | +| Images, speech, transcription through OpenAI SDK methods | Supported | Not applicable (text models) | Native `/run` and `/runsync` API, not the OpenAI image or audio routes | +| Responses API, Messages API | Supported | Not documented | Not documented | + +Tool calling and structured output requests use the same OpenAI request fields on both platforms (`tools`, `tool_choice`, `response_format`), so client code doesn't change once the endpoint is configured for the model. Together's `logprobs` shape differs from OpenAI's, so code that parses it needs a rewrite either way. + +### Images, video, and audio + +These calls move from Together's OpenAI-style media routes to Runpod's job API. The request body is wrapped in `input`, and the result is under `output`. + +Before, with the `together` SDK: + +```python +from together import Together + +client = Together() + +response = client.images.generate( + model="Qwen/Qwen-Image", + prompt="A serene mountain landscape at sunset", +) +print(response.data[0].url) +``` + +After, against a Public Endpoint: + +```python +import os + +import requests + +response = requests.post( + "https://api.runpod.ai/v2/qwen-image-t2i/runsync", + headers={ + "Authorization": f"Bearer {os.environ['RUNPOD_API_KEY']}", + "Content-Type": "application/json", + }, + json={"input": {"prompt": "A serene mountain landscape at sunset"}}, + timeout=300, +) +result = response.json() +print(result["output"]["result"]) +``` + +Each Public Endpoint defines its own input and output fields, so check the model page. Qwen Image returns the image URL in `output.result`, and most other image models return it in `output.image_url`. Input names such as `size`, `width`, `height`, or `aspect_ratio` also vary by model. Output URLs expire after 7 days, so download anything you need to keep. Together's video API already uses submit and poll (`client.videos.create`, then `client.videos.retrieve` until `completed`). On Runpod, submit with `/run`, poll `/status/JOB_ID` until `COMPLETED`, and read the URL from `output`. + +### Batch jobs + +Together's batch API takes a JSONL file where each line has a `custom_id` and a `body`. Runpod batch jobs take a JSON array where each element is a normal request with an `input` field, and they run on your endpoint's dedicated batch workers. + +Before: + +```bash +tg batches submit batch_input.jsonl --api chat.completions +tg batches get BATCH_ID +tg batches download BATCH_ID --output batch_output.jsonl +``` + +After: + +```bash +# Create the batch with its requests +curl -X POST https://api.runpod.ai/v2/ENDPOINT_ID/batch \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '[ + {"input": {"messages": [{"role": "user", "content": "Hello, world!"}], "sampling_params": {"max_tokens": 200}}}, + {"input": {"messages": [{"role": "user", "content": "Explain quantum computing."}], "sampling_params": {"max_tokens": 200}}} + ]' + +# Lock the batch so it starts running +curl -X POST https://api.runpod.ai/v2/ENDPOINT_ID/batch/BATCH_ID/finalize \ + -H "Authorization: Bearer $RUNPOD_API_KEY" + +# Check progress, then page through results +curl https://api.runpod.ai/v2/ENDPOINT_ID/batch/BATCH_ID \ + -H "Authorization: Bearer $RUNPOD_API_KEY" +curl "https://api.runpod.ai/v2/ENDPOINT_ID/batch/BATCH_ID/requests?offset=0&limit=50" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" +``` + +A batch doesn't run until you finalize it. The body of each request uses the vLLM worker's native input format (`messages` or `prompt`, with sampling parameters under `sampling_params`), not the OpenAI body. There's no `custom_id` field, so carry your own identifier inside `input` if your handler echoes it, or map results by the request IDs the API returns. Batch jobs are available on queue-based endpoints only. For smaller volumes, submit each request to `/run` with a `webhook` URL and let Runpod call you when each job finishes. + +### Container queue jobs + +Together's queue API and Runpod's job API line up closely. + +Before: + +```bash +curl -X POST "https://api.together.ai/v1/queue/submit" \ + -H "Authorization: Bearer $TOGETHER_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"model": "hello-world", "payload": {"name": "Together"}, "priority": 1}' + +curl "https://api.together.ai/v1/queue/status?model=hello-world&request_id=REQUEST_ID" \ + -H "Authorization: Bearer $TOGETHER_API_KEY" +``` + +After: + +```bash +curl -X POST "https://api.runpod.ai/v2/ENDPOINT_ID/run" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"input": {"name": "Runpod"}}' + +curl "https://api.runpod.ai/v2/ENDPOINT_ID/status/JOB_ID" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" +``` + +`payload` becomes `input`, `request_id` becomes `id`, and `outputs` becomes `output`. Runpod has no numeric job priority. The documented `lowPriority` policy flag only keeps a job from triggering worker scaling, so if you rely on Together's `priority` ordering, run a separate endpoint for high-priority traffic. Use `/runsync` to wait for the result in one call, and add a top-level `webhook` field to be notified instead of polling. From Python, `runpod.Endpoint("ENDPOINT_ID").run_sync({...})` wraps the same calls. + +### Control-plane automation + +Replace `tg` commands in scripts and CI with `runpodctl` or REST API v2, using the [CLI command map](#cli-command-map) and [API map](#api-map). In CI, set `RUNPOD_API_KEY` as a secret, the same way you set `TOGETHER_API_KEY`. + +## Step 8: Verify, cut over, and shut down + +### Verify + +Send real requests from outside Runpod, with the same client library and network path your application uses. + +1. Run your evaluation set against both platforms and compare outputs. Expect differences where the model version or quantization differs. +2. Test every feature from your inventory against the new endpoint: streaming, tool calls, `response_format`, image inputs, and long prompts near your context limit. +3. Measure latency for a warm worker and for a cold start. If the cold start is too slow for your use, set active workers or raise the idle timeout. +4. Run a load test at your peak concurrency and watch for `429` responses and queue delay. Raise max workers or `MAX_CONCURRENCY` if requests queue. +5. Check the job counts and worker state. + +```bash +curl -s "https://api.runpod.ai/v2/ENDPOINT_ID/health" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" +runpodctl serverless get ENDPOINT_ID --include-workers +``` + +### Cut over + +Run both platforms in parallel behind a configuration flag that selects the base URL, API key, and model name. Shift a small share of traffic to Runpod, compare error rates and latency, and increase the share in steps. Keep the Together path deployable until you've run at full traffic through at least one peak period. While both run, you pay for both. + +### Set a cost guard on Runpod + +Max workers is the ceiling on what an endpoint can spend per hour, so set it deliberately. Keep active workers at zero on anything that doesn't need to be warm, keep the execution timeout enabled so a stuck job can't run indefinitely, and delete test endpoints. Runpod also applies an account-wide [spend limit](/accounts-billing/billing#spending-limits) per hour, and scales down endpoints that receive no requests for several days. + +### Shut down Together AI + +Serverless models stop costing money when you stop calling them. Everything else needs an explicit action. Deleting is permanent, so confirm each resource against your inventory first. + +```bash +# Dedicated model inference: stop the deployment, then delete the endpoint and its deployments +tg endpoints update DEPLOYMENT_ID --min-replicas 0 --max-replicas 0 +tg endpoints rm ENDPOINT_ID --force + +# Dedicated container inference: run from the directory that holds the pyproject.toml +tg beta jig destroy + +# GPU clusters, then their shared volumes (volumes keep billing after the cluster is gone) +tg clusters delete CLUSTER_ID +tg clusters storage delete VOLUME_ID + +# Fine-tuning jobs still running +tg fine-tuning cancel FT_ID +``` + +Stop any remaining v1 dedicated endpoints through the v1 REST API. A stopped v1 endpoint can't be restarted. + +```bash +curl -s -X PATCH "https://api.together.ai/v1/endpoints/ENDPOINT_ID" \ + -H "Authorization: Bearer $TOGETHER_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"state": "STOPPED"}' +``` + +Then confirm that `tg endpoints ls`, `tg clusters list`, and `tg clusters storage list` show nothing running. Reserved clusters are charged for the full reservation, so deleting one early doesn't change what you've already paid. Finally, revoke the Together API keys your application no longer uses, in each project's API key settings, and remove `TOGETHER_API_KEY` from your secret stores. If you have auto-recharge on, turn it off. + +## Reference + +### CLI command map + +| Together CLI | Runpod | +| --- | --- | +| `tg login`, `tg whoami` | `runpodctl doctor`, or `rp login` for the Runpod SDK | +| `tg models public` | `runpodctl hub search`, and the [Public Endpoints reference](/public-endpoints/reference) | +| `tg models configs MODEL_ID` | `runpodctl gpu list` | +| `tg endpoints deploy MODEL --endpoint NAME` | `runpodctl serverless create --hub-id HUB_ID --name NAME` | +| `tg endpoints ls` | `runpodctl serverless list` | +| `tg endpoints get ID` | `runpodctl serverless get ENDPOINT_ID --include-workers` | +| `tg endpoints update DEPLOYMENT_ID --min-replicas N --max-replicas M` | `runpodctl serverless update ENDPOINT_ID --workers-min N --workers-max M` | +| `tg endpoints update DEPLOYMENT_ID --min-replicas 0 --max-replicas 0` (stop) | `runpodctl serverless update ENDPOINT_ID --workers-min 0` (scales to zero when idle) | +| `tg endpoints rm ID` | `runpodctl serverless delete ENDPOINT_ID` | +| `tg models upload`, `tg models remote-uploads create` | `hf upload`, or `aws s3 sync` to a network volume | +| `tg fine-tuning create` | `runpodctl pod create` and your training script | +| `tg fine-tuning download FT_ID` | No equivalent needed. Weights are already on your volume. | +| `tg batches submit FILE --api chat.completions` | `POST https://api.runpod.ai/v2/ENDPOINT_ID/batch`, then `/finalize` | +| `tg beta jig deploy` | `docker build`, `docker push`, `runpodctl template create --serverless`, `runpodctl serverless create --template-id` | +| `tg beta jig logs` | Endpoint logs in the console. See [Logs](/serverless/development/logs). | +| `tg beta jig secrets set` | [Secrets](/pods/templates/secrets) in the console | +| `tg beta jig volumes create` | `runpodctl network-volume create` | +| `tg clusters create` | Instant Clusters in the console, or `POST /v2/clusters` | +| `tg clusters storage create` | `runpodctl network-volume create` | +| `tg clusters list-regions` | `runpodctl datacenter list` | + +### API map + +| Together AI | Runpod | +| --- | --- | +| `POST https://api.together.ai/v1/chat/completions` | `POST https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1/chat/completions` | +| `POST /v1/completions` | `POST https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1/completions` | +| `GET /v1/models` | `GET https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1/models` | +| `POST /v1/images/generations`, `POST /v1/audio/speech` | `POST https://api.runpod.ai/v2/PUBLIC_ENDPOINT_SLUG/runsync` | +| Video create and retrieve | `POST https://api.runpod.ai/v2/PUBLIC_ENDPOINT_SLUG/run`, then `GET .../status/JOB_ID` | +| `POST /v1/queue/submit`, `GET /v1/queue/status` | `POST https://api.runpod.ai/v2/ENDPOINT_ID/run`, `GET .../status/JOB_ID` | +| Cancel a queued job | `POST https://api.runpod.ai/v2/ENDPOINT_ID/cancel/JOB_ID` | +| Clear a model's pending jobs | `POST https://api.runpod.ai/v2/ENDPOINT_ID/purge-queue` | +| Deployment health (`/v1/deployment-request/NAME/health`) | `GET https://api.runpod.ai/v2/ENDPOINT_ID/health` | +| Batch create, get, cancel | `POST /v2/ENDPOINT_ID/batch`, `GET /v2/ENDPOINT_ID/batch/BATCH_ID`, `POST /v2/ENDPOINT_ID/batch/BATCH_ID/cancel` on `api.runpod.ai` | +| Create, get, update, delete an endpoint or deployment | `POST`, `GET`, `PATCH`, `DELETE` on `https://api.runpod.io/v2/serverless` and `/v2/serverless/ENDPOINT_ID` | +| Endpoint analytics | `GET https://api.runpod.io/v2/serverless/ENDPOINT_ID/metrics/requests`, `/metrics/workers`, `/metrics/cold-starts` | +| Create, list, delete a GPU cluster | `POST`, `GET`, `DELETE` on `https://api.runpod.io/v2/clusters` | +| Create a shared volume | `POST https://api.runpod.io/v2/network-volumes` | +| List inference instance types | `GET https://api.runpod.io/v2/catalog/gpus` | +| `GET /v1/billing/usage` | `GET https://api.runpod.io/v2/billing` and `/v2/billing/serverless` | + +Note the two Runpod hosts: `api.runpod.ai` serves requests to endpoints, and `api.runpod.io` is the management API. + +### Configuration field map + +| Together AI | Runpod (`runpodctl` flag or REST API v2 field) | +| --- | --- | +| Model (`MODEL` argument, `ml_...`) | `MODEL_NAME` environment variable and `--model-reference` | +| Endpoint name (`--endpoint`) | `--name` (`name`). Requests use the endpoint ID, not the name. | +| Config or hardware (`--config cr_...`, `gpu_type`) | `--gpu-id` (`gpu.pools`) | +| GPU count in the config (`gpu_count`) | `--gpu-count` (`gpu.count`) | +| `--min-replicas` (`minReplicas`, `min_replicas`) | `--workers-min` (`workers.min`) | +| `--max-replicas` (`maxReplicas`, `max_replicas`) | `--workers-max` (`workers.max`) | +| `--scaling-metric`, `--scaling-target` | `--scale-by` and `--scale-threshold` on create, `--scaler-type` and `--scaler-value` on update (`scaling`) | +| `--scale-down-window` | `--idle-timeout` in seconds (`workers.idleTimeout`) | +| `--inactive-timeout` in minutes | Not needed. Use `--workers-min 0`. | +| `--max-concurrent-requests-per-replica` | `MAX_CONCURRENCY` environment variable on the vLLM worker | +| `--placement.regions` | `--data-center-ids` (`dataCenterIds`) | +| Jig `cmd` | The image's `CMD`, or `--docker-start-cmd` on the template | +| Jig `storage` | `--container-disk-in-gb` on the template | +| Jig `port` | Not used by queue-based endpoints. On load balancing endpoints, the `PORT` environment variable. | +| Jig `environment_variables` | `--env` on the template (`env`) | +| Jig `volume_mounts` | `--network-volume-id` (`mounts.network`) | +| Jig `cpu`, `memory` for CPU workers | `--compute-type CPU` with `--instance-id` | + +### Environment variable map + +| Together AI | Runpod | +| --- | --- | +| `TOGETHER_API_KEY` (client and CLI) | `RUNPOD_API_KEY` | +| `TOGETHER_PROJECT_ID` (client and CLI) | No equivalent. The API key determines the account. | +| `TOGETHER_DEPLOYMENT_NAME` (inside a Sprocket worker) | `RUNPOD_ENDPOINT_ID` (inside a worker) | +| `vllm serve` flags in a Jig `cmd` | Upper snake case environment variables on the vLLM worker, or `VLLM_EXTRA_ARGS` | +| Hugging Face token passed as a Jig secret | `HF_TOKEN` | + +### Status map + +| Together AI | Runpod | +| --- | --- | +| Deployment `PROVISIONING`, `SCALING` | Workers initializing. Check `/health`. | +| Deployment `READY` | At least one worker ready | +| Deployment `STOPPED` | Endpoint with no running workers. It starts one on the next request. | +| Deployment `DEGRADED` | Fewer workers than requested, often shown as throttled workers when a pool is out of stock | +| Queue job `pending` | Job `IN_QUEUE` | +| Queue job `running` | Job `IN_PROGRESS` | +| Queue job `done` | Job `COMPLETED` | +| Queue job `failed` | Job `FAILED` (or `TIMED_OUT` when it exceeds the execution timeout) | +| Queue job `canceled` | Job `CANCELLED` | +| Batch `VALIDATING` | Batch `DRAFT` (not started until you finalize it) | +| Batch `IN_PROGRESS` | Batch `FINALIZED` | +| Batch `COMPLETED`, `FAILED`, `CANCELLED` | Batch `COMPLETED`, `FAILED`, `CANCELLED` | +| Batch `EXPIRED` | No equivalent | + +See [Job states](/serverless/endpoints/job-states) for the full list on Runpod. + +## Migrate with a coding agent + +A coding agent can run the inventory, propose the mapping, and make the code changes. Install the Runpod skills and the MCP server first. + + + + The plugin installs the skills and connects the hosted MCP server. + + ```bash + claude plugin marketplace add runpod/runpod-plugins-official + claude plugin install runpod@runpod + ``` + + + Install the skills, then add the hosted MCP server. Replace `AGENT_ID` and `CLIENT_ID` with your agent's identifiers (for example, Codex or Cursor). + + ```bash + npx skills add runpod/runpod-plugins-official --skill '*' --agent AGENT_ID --global + npx @runpod/mcp-server@latest add --client CLIENT_ID + ``` + + + +The MCP server signs in with OAuth (**Sign in with Runpod**), so you don't need an API key to start. See [Agent skills](/get-started/agent-skills) and [MCP servers](/get-started/mcp-servers) for details. + +Then give the agent a prompt like this one, from the root of the repository that calls Together AI. + +```text +Migrate this project from Together AI to Runpod using the runpod-cloud-migrate skill. + +1. Inventory first, and change nothing yet. Search the codebase for the together and + together-ai SDKs, TOGETHER_API_KEY, api.together.ai, api-inference.together.ai, + tg CLI calls, Sprocket workers, and Jig pyproject.toml files. If the tg CLI is + logged in, also list dedicated endpoints, uploaded models, fine-tuning jobs, + batch jobs, GPU clusters, and cluster storage with read-only commands. For every + call site, record the model string, the API (chat, completions, images, video, + audio, rerank, batch), and the features used (streaming, tools, response_format, + vision, logprobs). +2. Show me a plan as a table: each workload, where it lands on Runpod (Public + Endpoint, vLLM Serverless endpoint, custom Serverless endpoint, Pod, or Instant + Cluster), the GPU you'd use and why it fits the model, the worker counts, and an + estimated cost at my current traffic. Call out anything with no direct equivalent + (embeddings, rerank, traffic splitting, Kubernetes clusters, code sandbox) and + ask me how to handle it. +3. Wait for my approval before you create anything billable on Runpod. +4. Migrate one workload first. Deploy it, wait until a worker is ready, and verify + it with a real request from outside Runpod. Compare the output to Together's for + the same prompt, then show me both. +5. Update the client code behind a configuration switch so I can route traffic to + either platform. Don't remove the Together path. +6. Never stop, delete, or modify any Together AI resource, and never revoke a + Together API key, without my explicit approval for that specific resource. +7. When you finish, list every Runpod resource you created, with its ID and what it + costs while idle. +``` + +## Troubleshooting + +**The first request returns a 524 or times out.** The request waited on a worker that was still starting and loading the model, and the synchronous route gave up. Send the first request with `/run` and poll `/status`, raise your client timeout, and for latency-sensitive traffic set active workers to 1 or more. Check worker state with the `/health` route. + +**Responses stop after a few words.** The vLLM worker's documented default for `max_tokens` is 16, so code that never set `max_tokens` gets truncated output. Set it on every request. + +**The API returns an invalid model error.** `model` must match the Hugging Face ID you deployed, or the value of `OPENAI_SERVED_MODEL_NAME_OVERRIDE`. Together model strings and endpoint strings (`PROJECT_SLUG/ENDPOINT_NAME`) don't resolve on Runpod. Call `client.models.list()` to see what the endpoint serves. + +**Requests return 401.** The client is still sending the Together key. Use your Runpod API key as the bearer token. If the key is restricted, confirm it has access to this endpoint. + +**Tool calls come back as plain text.** Tool calling isn't enabled for the model, or the parser doesn't match the model's format. Set `ENABLE_AUTO_TOOL_CHOICE=true` and the `TOOL_CALL_PARSER` for your model family from [Configure vLLM to work with your model](/serverless/vllm/configuration#model-specific-configurations). + +**Workers fail to start or run out of memory.** The model plus its KV cache doesn't fit on the GPU you chose. Together selected hardware for you, and on Runpod this is your decision. Lower `MAX_MODEL_LEN`, lower `GPU_MEMORY_UTILIZATION`, use a quantized repository, or move to a larger GPU pool. For models that take longer than the default 420 seconds to load, raise `RUNPOD_INIT_TIMEOUT`. + +**The endpoint stays at zero workers.** The GPU pool may be out of stock in the endpoint's data centers, or the endpoint's CUDA version constraint may exclude the machines that have stock. Add more pools, remove data center restrictions, and check the endpoint's CUDA settings. A cached model that isn't on any host yet also delays the first worker while it downloads, which shows up as a job sitting in the queue. + +**A gated model fails to download.** Add `HF_TOKEN` to the endpoint with a token that has accepted the model's license. Together handled model access for you. On Runpod the worker pulls from Hugging Face with your credentials. + +**Reasoning text is missing from streamed responses.** The field name differs between serving stacks: Together documents `reasoning` or `reasoning_content` depending on the model, and vLLM's field depends on its version. Read both fields in your client, and set `REASONING_PARSER` on the endpoint. + +**The bill is higher than expected at low traffic.** You're paying for worker seconds, including model load on each cold start and the idle window after each request. Check whether active workers is set above zero, how long the idle timeout is, and how often workers cold start. For a model that Runpod hosts, a Public Endpoint may fit low, bursty traffic better than your own endpoint. + +## Next steps + + + + Configure, deploy, and call OpenAI-compatible LLM endpoints. + + + Tune active workers, scaling, timeouts, and FlashBoot. + + + Browse the hosted models and their parameters. + + + Cut cold starts by caching Hugging Face models on Runpod hosts. + + diff --git a/migrate/vast-ai.mdx b/migrate/vast-ai.mdx new file mode 100644 index 000000000..07d8e3b83 --- /dev/null +++ b/migrate/vast-ai.mdx @@ -0,0 +1,1424 @@ +--- +title: "Migrate from Vast.ai to Runpod" +sidebarTitle: "Vast.ai" +description: "Move Vast.ai instances, templates, volumes, Serverless endpoints, and automation code to Runpod Pods, Serverless, and REST API v2." +--- + +This guide is for teams that run GPU workloads on [Vast.ai](https://vast.ai) and want to run them on Runpod. Vast [instances](https://docs.vast.ai/guides/instances/overview) (on-demand, interruptible, and reserved) map to Runpod [Pods](/pods/overview), [Vast Serverless](https://docs.vast.ai/guides/serverless/index) endpoints, workergroups, and PyWorkers map to Runpod [Serverless](/serverless/overview) endpoints, workers, and handlers, and Vast's machine-local [volumes](https://docs.vast.ai/guides/instances/storage/volumes) map to network volumes and global volumes. When you finish, your images, data, deploy scripts, and client code run against Runpod, and nothing is still billing on Vast. + + +You can hand most of this migration to a coding agent. See [Migrate with a coding agent](#migrate-with-a-coding-agent). + + +## How Vast.ai maps to Runpod + +The biggest structural change is how you get a GPU. On Vast you search a marketplace of offers from individual hosts and rent one specific machine. On Runpod you name a GPU type (and optionally a data center), and the scheduler places your container. + +| Vast.ai concept | Runpod equivalent | Notes | +| --- | --- | --- | +| Instance (Docker container on a rented offer) | [Pod](/pods/overview) | Both are containers with dedicated GPUs. A Pod is created from a GPU type ID, not an offer ID. | +| Offer search (`vastai search offers`) | GPU catalog: `runpodctl gpu list`, `runpodctl datacenter list`, `GET /v2/catalog/gpus` | The catalog reports stock per GPU type and data center. There's no per-machine reliability score, bandwidth price, or host choice. | +| On-demand instance | On-demand Pod | Resources are dedicated to your Pod and can't be displaced by other users. | +| Interruptible instance (`--bid_price`) | On-demand Pod | `runpodctl`, REST API v2, and the Terraform provider create on-demand Pods. See [What has no direct equivalent](#what-has-no-direct-equivalent). | +| Reserved instance (`vastai prepay instance`) | On-demand Pod | Runpod has no self-serve prepay discount. See [What has no direct equivalent](#what-has-no-direct-equivalent). | +| Verified machines, `datacenter=true`, Secure Cloud filter | Secure Cloud (default) and Community Cloud | Set with `--cloud-type` or REST `cloud`. Secure Cloud also accepts `--compliance` and `--country-code` filters. | +| Virtual machine instance (`vms_enabled=true`) | No equivalent | Pods are containers. See [What has no direct equivalent](#what-has-no-direct-equivalent). | +| Launch modes: entrypoint, SSH, Jupyter | One Pod type, with `--ssh` and `startJupyter` as options | Enabling SSH on a Pod doesn't replace your image's entrypoint. | +| `--onstart-cmd`, `--args`, `--entrypoint` | Start command: `--docker-args`, REST `cmd` and `entrypoint` | There's no separate on-start script. | +| `--env '-e KEY=value'` (Docker options) | `--env '{"KEY":"value"}'`, REST `env` | JSON object, not Docker flags. | +| `--env '-p 8000:8000'` | `--ports "8000/http,22/tcp"`, REST `ports` | Each port is HTTP (proxied) or TCP (public IP). | +| UDP port mapping (`-p 8082:8082/udp`) | No equivalent | Pods don't support UDP. | +| Template (`--template_hash`) | [Template](/pods/templates/overview) (`--template-id`) | Pod templates and Serverless templates are separate. | +| Container storage (`--disk`) | Container disk plus volume disk | Container disk is erased when a Pod stops. Volume disk at `/workspace` persists across stops. | +| Local volume (`vastai create volume`) | [Network volume](/storage/network-volumes) | A network volume is independent of any machine but pinned to one data center. | +| Object storage you manage (S3, R2, GCS) for cross-host data | Network volume, [global volume](/storage/globalvolume/overview), or keep your object storage | A global volume mounts in any data center. | +| Cloud Sync (`vastai cloud copy`) | [Cloud Sync](/pods/storage/cloud-sync), or the [S3-compatible API](/storage/s3-api) on network volumes | Provider lists differ. See [Step 5](#step-5-move-your-data-and-models). | +| `vastai copy`, `scp` | `runpodctl send` and `receive`, the [S3-compatible API](/storage/s3-api) on network volumes, `scp`, `rsync` | See [Step 5](#step-5-move-your-data-and-models). | +| Serverless endpoint plus workergroup | [Serverless endpoint](/serverless/overview) | One resource holds the image, GPU pools, and scaling settings. | +| Serverless worker running a PyWorker | Worker running a [handler](/serverless/workers/handler-functions), or your own HTTP server on a [load-balancing endpoint](/serverless/load-balancing/overview) | No benchmark step and no workload calculator. | +| Deployments (`vastai.Deployment`, `@app.remote`, beta) | Runpod SDK app (`runpod.App`, `@app.queue`, `@app.task`, `@app.api`) | Both define remote functions in one Python file. | +| OpenAI proxy (`https://openai.vast.ai/ENDPOINT_NAME`) | vLLM worker route (`https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1`) | Runpod's route needs the real model name in `model`. | +| Model Library and Template Library | [Hub](/hub/overview) and [Public Endpoints](/public-endpoints/overview) | Public Endpoints are hosted models you call without deploying anything. | +| Account environment variables (`vastai create env-var`) | [Secrets](/pods/templates/secrets), referenced as `{{ RUNPOD_SECRET_name }}` | Secrets are referenced per template or Pod, not injected into every container. | +| Docker login (`--login '-u USER -p TOKEN REGISTRY'`) | Registry credential (`runpodctl registry create`, `--registry-auth-id`) | Stored once, referenced by ID. | +| Pricing: per-host rates, per second, separate storage and bandwidth | Listed rates per GPU type, per second, no ingress or egress fees | See [Pod pricing](/pods/pricing) and [Serverless pricing](/serverless/pricing). | +| Prepaid credits, autobilling | Prepaid credits, auto-pay | See [Billing](/accounts-billing/billing). | +| Infrastructure as code: scripts over the `vastai` CLI or REST API | [Terraform provider](/integrations/terraform/overview) `runpod/runpod`, built on REST API v2 | You can keep scripting with `runpodctl` and REST instead. | +| `vastai` CLI | [`runpodctl`](/runpodctl/overview) | JSON output by default. | +| `vastai` Python package (`VastAI`, `Serverless`) | [Runpod SDK](/sdk/overview) (`pip install runpod`) | Apps, an endpoint client, the worker SDK, and an API wrapper. | +| REST API (`https://console.vast.ai/api/v0`) | [REST API v2](/api-reference-v2/overview) (`https://api.runpod.io/v2`) | Both use `Authorization: Bearer`. | +| API key (`VAST_API_KEY`, `~/.config/vastai/vast_api_key`) | API key (`RUNPOD_API_KEY`, `~/.runpod/config.toml`) | Both support scoped keys. | +| Teams and team roles | [Organizations](/organizations/orgs-overview), roles, and groups | | +| `vastai logs`, `vastai get endpt-logs` | `runpodctl pod logs`, `GET /v2/pods/POD_ID/logs`, [Serverless logs](/serverless/development/logs) | | +| Agent skill and plugins (`vast-ai/vast-cli`) | [Agent skills](/get-started/agent-skills) and the [MCP server](/get-started/mcp-servers) | | + +## What's different + +### What works differently + +**You pick a GPU type, not a machine.** A Vast create call takes an offer ID from a search. A Runpod create call takes a GPU type ID such as `NVIDIA GeForce RTX 4090` and places it. The create call doesn't search for capacity or fall back to another GPU, so automation that looped over offers becomes a loop over GPU types or data centers that you're willing to accept. Stock varies by data center and changes between a catalog read and a create. + +**Your image's entrypoint keeps running.** Vast's SSH and Jupyter launch modes replace the image entrypoint and run your `--onstart-cmd` instead. Runpod has no launch modes. SSH is an option on the Pod (`--ssh`, on by default in `runpodctl`), and your image's `ENTRYPOINT` and `CMD` run as built unless you override them. Move the contents of an on-start script into the Pod's start command or into the image. + +**Environment variables are visible in SSH sessions without extra steps.** On Vast you may have added `env >> /etc/environment` to on-start scripts so SSH and Jupyter sessions could see your variables. Drop that line. On Runpod you pass `env` as a JSON object, and the Pod's own variables use the `RUNPOD_` prefix. + +**Ports are typed.** Vast maps each `-p` port to a random external port on the host's shared IP over plain TCP. On Runpod, a port declared as `PORT/http` is served at `https://POD_ID-PORT.proxy.runpod.net` over HTTPS, and a port declared as `PORT/tcp` gets a public IP and a mapped external port. The proxy URL is public and unauthenticated, and it closes connections that run longer than 100 seconds, so add authentication in your app and use a TCP port for long-lived connections. Declare every port you need: the console accepts up to 10 HTTP ports per Pod. Requesting a port above 70000 for a matching internal and external port works the same way on both platforms. + +**Stopping a Pod clears its container disk.** A stopped Vast instance keeps everything on its disk. A stopped Pod keeps only the volume disk (mounted at `/workspace` by default) or an attached network volume. Anything written elsewhere in the container is gone after a stop, a restart caused by a config change, or a terminate. Write checkpoints and outputs under `/workspace`. + +**Storage isn't tied to a machine.** A Vast volume can only attach to instances on the physical machine where it was created. A Runpod network volume attaches to any Pod or Serverless worker in its data center, and a global volume attaches in any data center. You size a network volume up front and can grow it later. You can also change a Pod's container disk and volume disk size with `runpodctl pod update`, where Vast fixes disk size at creation. + +**Bandwidth isn't billed.** Vast hosts set per-GB upload and download prices. Runpod Pods bill compute and storage per second with no ingress or egress fees. Remove `inet_up_cost` and `inet_down_cost` terms from any cost model you carry over. + +**Status fields have different names and one false friend.** Vast scripts decide from `actual_status`. Runpod's REST API returns one `status` field, and `runpodctl` adds `runtimeStatus`. On Vast, `exited` means the container crashed. On Runpod, `EXITED` means the Pod is stopped and can be started again, and `ERROR` is the failure state. See the [status map](#status-map). + +**Serverless is one resource, and requests go through Runpod.** A Vast client asks the Serverless engine for a worker (`/route/`), then sends the request to that worker's address with signed `auth_data`. A Runpod client sends every request to a fixed endpoint URL with your API key, and Runpod queues or load-balances it. There's no workergroup: the endpoint itself holds the image, GPU pools, and scaling settings. + +**Serverless scaling uses different inputs.** Vast scales on measured load against benchmarked worker performance (`min_load`, `target_util`, `cold_mult`, `cold_workers`). Runpod scales on queue delay or request count, between a minimum and maximum worker count, with an idle timeout. Endpoints scale to zero by default. + +### What has no direct equivalent + +On the Runpod side: + +- **UDP ports.** Vast maps UDP ports (`-p 8082:8082/udp`, `VAST_UDP_PORT_X`). Pods don't support UDP. Move the protocol to TCP (for example WebRTC over TCP or TURN over TCP, or WireGuard replaced by a TCP-based tunnel), or keep the UDP-facing component elsewhere and connect it to the Pod over TCP. +- **Bidding for interruptible capacity.** Vast lets you set a bid and pauses the instance when you're outbid. Pod creation in `runpodctl` and REST API v2 has no bid or interruptible field. Run these jobs as on-demand Pods with checkpoints, as a Serverless queue endpoint that scales to zero between jobs, or as an `@app.task` function that provisions a Pod per call and terminates it when the function returns. +- **Self-serve reserved discounts.** `vastai prepay instance` has no counterpart. Runpod's savings plans are deprecated and can't be purchased. For committed capacity or custom pricing, [contact sales](https://www.runpod.io/contact). +- **Choosing a host.** Filters such as `reliability`, `inet_down`, `dlperf`, `static_ip`, and `machine_id` don't exist. You choose a GPU type, a cloud tier, a data center or country, a minimum CUDA version, and minimum vCPU and RAM per GPU. +- **Virtual machines and Docker inside the instance.** Vast VM instances support systemd and nested containers. Pods are containers and don't run Docker Compose. Package each service as its own image and run one Pod per service (Pods in the same account can reach each other over [global networking](/pods/networking)), or run the services as processes in one image. +- **ARM hosts.** Vast lists `cpu_arch=arm64` offers. Runpod images must be `linux/amd64`. +- **Copying data off a stopped instance.** `vastai copy` and Cloud Sync work on stopped Vast instances. On Runpod, put data you need without a running Pod on a network volume and read it through the S3-compatible API. +- **Scheduled Cloud Sync.** `vastai cloud copy --schedule` has no counterpart. Run `aws s3 sync` or `rclone` from cron inside the Pod, or from your own scheduler against the S3-compatible API. +- **Custom hostname (`-h`).** Pod create has no hostname field. Read `RUNPOD_POD_HOSTNAME` and `RUNPOD_POD_ID` instead. +- **Benchmark-driven Serverless scaling and mixed-hardware workergroups.** Runpod doesn't benchmark workers or weight requests by cost. Use GPU pools to allow several GPU types, and tune the scaler threshold. + +Runpod features that change how you'd build the same thing: + +- **Network-attached and cross-region volumes.** Vast volumes are machine-local, so shared data usually lives in external object storage. Network volumes and global volumes can replace that, or you can keep your object storage as is. +- **A URL derived from the Pod ID.** Vast's Cloudflare tunnel URLs are generated per instance by the Instance Portal. Runpod's proxy URL is built from the Pod ID and port, so you know it before the Pod starts. +- **Queued jobs with IDs.** Runpod queue endpoints accept a job, return an ID, and let you poll, stream, cancel, or receive a webhook. +- **Deploy when available.** REST `POST /v2/pods` accepts a `queue` object that holds a Secure Cloud GPU Pod request until capacity appears, depending on your plan. +- **Hosted models.** [Public Endpoints](/public-endpoints/overview) serve models with no deployment. +- **Multi-node clusters.** [Instant Clusters](/instant-clusters) provide multi-node Slurm, PyTorch, and Ray environments. +- **Terraform.** The `runpod/runpod` provider manages Pods, endpoints, templates, and volumes. + +## Requirements + +- A Vast.ai account with access to the resources you're migrating, and the `vastai` CLI authenticated (`vastai show user` succeeds). +- A [Runpod account](https://www.console.runpod.io) with credits. Runpod is prepaid, like Vast. +- Docker images built for `linux/amd64`, in a registry Runpod can pull from. +- An SSH key pair if you connect to Pods over SSH. +- `jq` and `curl` for the scripted steps. + +## Step 1: Take inventory of what you run on Vast.ai + +List every resource on the account, in JSON, so you have a record to work from. Run these with the `vastai` CLI. + +```bash inventory.sh +mkdir -p vast-inventory && cd vast-inventory + +vastai show instances --raw > instances.json # running and stopped instances +vastai show volumes --raw > volumes.json # local volumes and the instances using them +vastai show endpoints --raw > endpoints.json # Serverless endpoints +vastai show workergroups --raw > workergroups.json # Serverless workergroups +vastai show deployments --raw > deployments.json # Deployments (beta) +vastai show env-vars --raw > env-vars.json # account environment variables +vastai show connections --raw > connections.json # Cloud Sync connections +vastai show scheduled-jobs --raw > scheduled-jobs.json +vastai show ssh-keys --raw > ssh-keys.json +vastai show api-keys --raw > api-keys.json +vastai show invoices-v1 --raw > invoices.json +``` + +`env-vars.json` and `api-keys.json` can contain secrets. Keep the directory out of version control. + +For each instance, pull the fields you need to recreate it. The names below are the ones Vast returns from `show instances`. + +```bash +jq -r '.[] | [.id, .label, .actual_status, .gpu_name, .num_gpus, .image_uuid, .disk_space, .geolocation, .dph_total] | @tsv' instances.json +``` + +Then open one instance in full (`vastai show instance INSTANCE_ID --raw`) and record what the summary line doesn't show: + +- The image and tag, and whether it's private (you passed `--login`). +- The launch mode (`image_runtype`): entrypoint with arguments, or SSH or Jupyter with an on-start script (`onstart`). +- Environment variables and port mappings from the Docker options (`extra_env`), including any `/udp` ports. +- Disk size and what's stored on it, plus any attached volume and its mount path. +- Whether the instance is on-demand, interruptible (`is_bid` is true), or reserved (it has prepaid credit). +- The template it came from, if any. Templates you own are in the console under **Templates**, and `vastai search templates` finds them by name. + +For each Serverless endpoint, record the endpoint parameters (`max_workers`, `min_load`, `cold_workers`, `target_util`, `inactivity_timeout`) and, for each workergroup, the template or `launch_args`, the `search_params`, and `gpu_ram`. If the workergroup uses a custom PyWorker, note the `PYWORKER_REPO` value and read its `worker.py`. + +Finally, find the code that calls Vast. Run this from the root of each repository. + +```bash +grep -rnE "vastai|vast\.ai|VAST_API_KEY|CONTAINER_API_KEY|CONTAINER_ID|VAST_CONTAINERLABEL|VAST_(TCP|UDP)_PORT_|PUBLIC_IPADDR|PYWORKER_REPO|vast_api_key" . \ + --include='*.py' --include='*.ipynb' --include='*.sh' --include='*.js' --include='*.ts' --include='*.go' \ + --include='*.yml' --include='*.yaml' --include='*.toml' --include='*.json' --include='*.md' \ + --include='Dockerfile*' --include='requirements*.txt' --include='.env*' +``` + +Each match falls into one of three groups: control-plane code that rents and manages instances, client code that calls a Serverless endpoint, or code inside your container that reads Vast's runtime variables. Steps 4 and 7 cover each group. + +## Step 2: Set up Runpod + +1. Create an account in the [Runpod console](https://www.console.runpod.io) and add credits on the billing page. +2. Create an API key on the [Credentials page](/get-started/credentials). Your Vast key doesn't work on Runpod. +3. Install the tools and sign in. + +```bash +# CLI +curl -fsSL https://cli.runpod.net | sudo bash # or: brew install runpod/runpodctl/runpodctl +export RUNPOD_API_KEY=YOUR_API_KEY +runpodctl doctor + +# Python SDK 2.x (Python 3.10 to 3.14), which includes the rp CLI +pip install runpod +rp login + +# SSH key, so new Pods accept your key +runpodctl ssh add-key --key-file ~/.ssh/id_ed25519.pub +``` + +`rp login` stores the key in `~/.runpod/config.toml`. In CI, set `RUNPOD_API_KEY` as a secret instead. Register your SSH key before you create Pods: keys apply to Pods created afterward. + +If you manage infrastructure as code, the [Terraform provider](/integrations/terraform/overview) is built on REST API v2. To let an agent manage resources, connect the [MCP server](/get-started/mcp-servers). + +## Step 3: Choose where each workload lands + +Use the shape of the workload, not the Vast product it runs on today, to choose a target. + +| Vast.ai workload | Runpod target | Why | +| --- | --- | --- | +| On-demand instance you SSH into or use with Jupyter | Pod | Same interactive model, with a volume disk or network volume for persistence. | +| Entrypoint-mode instance serving an API all day | Pod, or a Serverless load-balancing endpoint if traffic varies | A load-balancing endpoint runs the same HTTP server and adds autoscaling. | +| Instance your script rents, runs one job on, and destroys | `@app.task` function, or a Pod created and deleted through REST API v2 | `@app.task` provisions a Pod per call and terminates it when the function returns. | +| Interruptible instances working through a batch | Serverless queue endpoint | The queue holds jobs, and workers scale to zero when it's empty. | +| Reserved instance running for months | On-demand Pod | Talk to sales about committed capacity. | +| Serverless endpoint with a vLLM template | Serverless endpoint running the [vLLM worker](/serverless/vllm/get-started) | Includes an OpenAI-compatible route. | +| Serverless endpoint with a custom PyWorker in front of an HTTP model server | Serverless load-balancing endpoint | Your server receives requests directly on its own paths. | +| Serverless endpoint doing request and response jobs | Serverless queue endpoint with a handler | Async jobs, status polling, streaming, and webhooks. | +| Deployment using `@app.remote` | Runpod SDK app with `@app.queue` | Same decorator-on-a-function model. | +| Calls to a common open model through your own endpoint | [Public Endpoint](/public-endpoints/overview) | Nothing to deploy or scale. | +| Several instances wired together for distributed training | [Instant Cluster](/instant-clusters) | Multi-node networking with Slurm, PyTorch, or Ray. | +| VM instance running Docker Compose | One Pod per service | Pods don't run nested containers. | + +### Map your GPUs + +Vast search queries use `gpu_name` with underscores. Runpod uses GPU type IDs for Pods and GPU pool IDs for Serverless endpoints. The full list is in the [GPU types reference](/references/gpu-types). + +| Vast.ai `gpu_name` | Runpod GPU type ID (Pods) | Serverless pool | +| --- | --- | --- | +| `RTX_3090` | `NVIDIA GeForce RTX 3090` | `AMPERE_24` | +| `RTX_4090` | `NVIDIA GeForce RTX 4090` | `ADA_24` | +| `RTX_5090` | `NVIDIA GeForce RTX 5090` | Check `pool` in the catalog. | +| `RTX_A6000` | `NVIDIA RTX A6000` | `AMPERE_48` | +| `RTX_6000Ada` | `NVIDIA RTX 6000 Ada Generation` | `ADA_48_PRO` | +| `RTX_PRO_6000_S` | `NVIDIA RTX PRO 6000 Blackwell Server Edition` | Check `pool` in the catalog. | +| `RTX_PRO_6000_WS` | `NVIDIA RTX PRO 6000 Blackwell Workstation Edition` | Check `pool` in the catalog. | +| `L4` | `NVIDIA L4` | `AMPERE_24` | +| `L40`, `L40S` | `NVIDIA L40`, `NVIDIA L40S` | `ADA_48_PRO` | +| `A40` | `NVIDIA A40` | `AMPERE_48` | +| `A100_PCIE` with `gpu_ram>=79` | `NVIDIA A100 80GB PCIe` | `AMPERE_80` | +| `A100_SXM4` with `gpu_ram>=79` | `NVIDIA A100-SXM4-80GB` | `AMPERE_80` | +| `H100_SXM` | `NVIDIA H100 80GB HBM3` | `ADA_80_PRO` | +| `H100_PCIE`, `H100_NVL` | `NVIDIA H100 PCIe`, `NVIDIA H100 NVL` | `ADA_80_PRO` for H100 PCIe. Check `pool` for NVL. | +| `H200`, `H200_NVL` | `NVIDIA H200`, `NVIDIA H200 NVL` | `HOPPER_141` for H200. Check `pool` for NVL. | +| `B200` | `NVIDIA B200` | Check `pool` in the catalog. | +| `B300` | `NVIDIA B300 SXM6 AC` | Check `pool` in the catalog. | + +Choose the GPU by what the workload needs (VRAM, GPU count, CUDA version), then check live stock before you commit to a data center. + +```bash +# What's in stock now, across clouds +runpodctl gpu list + +# Stock per data center +runpodctl datacenter list + +# Pod stock, per data center, with CUDA versions, for the default Secure Cloud +curl -s "https://api.runpod.io/v2/catalog/gpus?include=AVAILABILITY&product=POD" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + | jq '.gpus[] | {id, memory, pool, availability, dataCenters: [.dataCenters[]? | {id, availability}]}' +``` + +Each catalog entry includes `id` (for Pods), `pool` (for Serverless), `memory` in GB, `maxCount` per Pod, and, with `include=AVAILABILITY`, an `availability` level of `NONE`, `LOW`, `MEDIUM`, or `HIGH`. Stock changes continuously, so treat the result as a way to order your candidates. + +These Vast search filters translate to Pod create options. The rest have no counterpart. + +| Vast.ai search filter | Runpod option | +| --- | --- | +| `gpu_name=...` | `--gpu-id`, REST `gpu.id` | +| `num_gpus=N` | `--gpu-count`, REST `gpu.count` | +| `cuda_vers>=12.4` | `--min-cuda-version 12.4`, REST `gpu.minCudaVersion` | +| `cpu_cores_effective>=N` | REST `gpu.minVcpuCountPerGpu` (per GPU, so divide by the GPU count) | +| `cpu_ram>=N` | REST `gpu.minRamPerGpu` (GB per GPU) | +| `geolocation in [US,CA]` | `--country-code` (Secure Cloud GPU Pods) or `--data-center-ids` | +| `datacenter=true` | `--cloud-type SECURE` (the default) | +| `verified=false`, non-datacenter hosts | `--cloud-type COMMUNITY` | +| `direct_port_count>=1` | Use Secure Cloud. Community Cloud Pods aren't guaranteed a public IP. | + +## Step 4: Move your images and code + +### Check the image + +Images that run on Vast in entrypoint mode usually run on Runpod unchanged. Check four things. + +- **Architecture.** Runpod pulls `linux/amd64` images. If you build on Apple silicon, build with `docker buildx build --platform linux/amd64`. +- **Registry access.** Replace `--login '-u USER -p TOKEN REGISTRY'` with a stored credential, then pass its ID when you create a Pod or template. +- **CUDA.** Vast's `@vastai-automatic-tag` picks an image build to match each host. Runpod doesn't rewrite tags, so pin a real tag and set a CUDA floor on the Pod (`--min-cuda-version`) that matches the image. +- **Vast base image features.** Images built on `vastai/base-image` include the Instance Portal, Cloudflare tunnels, and Vast's Jupyter certificate setup. Those services expect Vast's environment. Use your framework's upstream image or a Runpod image such as `runpod/pytorch` as the base, and reach HTTP services through the Pod proxy. + +Store a private registry credential like this. + +```bash +printenv REGISTRY_TOKEN | runpodctl registry create --name "my-registry" --username "USER" --password-stdin +runpodctl registry list # copy the ID for --registry-auth-id +``` + +### Translate the launch configuration + +The before and after below launch the same vLLM server: one image, one port, one environment variable, and arguments passed to the image's entrypoint. + + + + ```bash + vastai create instance OFFER_ID \ + --image vllm/vllm-openai:latest \ + --disk 40 \ + --env "-p 8000:8000 -e HF_TOKEN=hf_xxxxx" \ + --args --model Qwen/Qwen2.5-7B-Instruct --max-model-len 4096 + ``` + + + ```bash + runpodctl pod create \ + --name vllm-server \ + --image vllm/vllm-openai:latest \ + --gpu-id "NVIDIA GeForce RTX 4090" \ + --min-cuda-version 12.8 \ + --container-disk-in-gb 40 \ + --ports "8000/http" \ + --env '{"HF_TOKEN":"{{ RUNPOD_SECRET_hf_token }}"}' \ + --docker-args "--model Qwen/Qwen2.5-7B-Instruct --max-model-len 4096" + ``` + + + ```bash + curl -s -X POST "https://api.runpod.io/v2/pods" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "vllm-server", + "image": "vllm/vllm-openai:latest", + "gpu": {"id": "NVIDIA GeForce RTX 4090", "count": 1, "minCudaVersion": "12.8"}, + "disk": 40, + "ports": ["8000/http"], + "env": {"HF_TOKEN": "{{ RUNPOD_SECRET_hf_token }}"}, + "cmd": ["--model", "Qwen/Qwen2.5-7B-Instruct", "--max-model-len", "4096"] + }' + ``` + + + +The server is then reachable at `https://POD_ID-8000.proxy.runpod.net/v1/models`. Three things changed: there's no offer ID, the port carries a type, and the Hugging Face token comes from a [secret](/pods/templates/secrets) instead of sitting in the command. Set the CUDA floor to the version your image was built for. + +Each launch mode translates as follows. + +| Vast.ai launch | Runpod equivalent | +| --- | --- | +| Entrypoint mode with `--args A B C` | REST `cmd: ["A", "B", "C"]`, or `--docker-args "A B C"`. The image's `ENTRYPOINT` receives them. | +| `--entrypoint CMD` | REST `entrypoint: ["CMD"]`, or `--docker-entrypoint` on a template. | +| `--ssh` or `--jupyter` with `--onstart-cmd 'COMMANDS'` | REST `cmd: ["bash", "-c", "COMMANDS"]` with `startSsh: true`. Your image must start `sshd` when it sees the `PUBLIC_KEY` variable, which Runpod's official images do. If your command replaces the image's own start script, start `sshd` yourself or keep SSH to images you control. | +| `--onstart FILE` | Bake the script into the image, or fetch and run it from the start command. | +| `--jupyter` (port 8080) | REST `startJupyter: true` with `8888/http` in `ports`, on an image that includes JupyterLab, such as Runpod's PyTorch images. | +| `--direct` | Add `22/tcp` to `ports` for direct SSH with `scp`, `rsync`, and port forwarding. Without it you get proxied SSH, which carries an interactive shell only. | +| `--label NAME` | `--name NAME` | +| `--disk GB` | `--container-disk-in-gb GB`, plus `--volume-in-gb GB` for the data you need to keep across stops. | +| `--cancel-unavail` | Not needed. A create that can't be placed returns an error and creates nothing. | + +### Convert templates + +A Vast template stores the image, Docker options, launch mode, on-start script, disk size, and search filters. A Runpod template stores the image, start command, environment, ports, and disk sizes. GPU choice stays on the Pod or endpoint. + +```bash +runpodctl template create \ + --name "vllm-qwen" \ + --image "vllm/vllm-openai:latest" \ + --container-disk-in-gb 40 \ + --ports "8000/http,22/tcp" \ + --env '{"HF_TOKEN":"{{ RUNPOD_SECRET_hf_token }}"}' \ + --docker-start-cmd "--model,Qwen/Qwen2.5-7B-Instruct,--max-model-len,4096" +``` + +`--docker-start-cmd` takes a comma-separated list. Add `--serverless` to create a template for a Serverless endpoint: Pod templates and Serverless templates aren't interchangeable, and a Serverless template binds to one endpoint at a time. For Vast's recommended templates (PyTorch, ComfyUI, vLLM), look for a maintained counterpart with `runpodctl template search NAME --type official` before you convert your own. + +### Replace Vast runtime variables in your code + +Code inside the container that reads Vast's predefined variables needs new names. The full table is in the [environment variable map](#environment-variable-map). The common cases are `CONTAINER_ID` to `RUNPOD_POD_ID`, `PUBLIC_IPADDR` to `RUNPOD_PUBLIC_IP`, `GPU_COUNT` to `RUNPOD_GPU_COUNT`, and `VAST_TCP_PORT_22` to `RUNPOD_TCP_PORT_22`. + +Code that builds its own public URL from `PUBLIC_IPADDR` and `VAST_TCP_PORT_8000` changes shape for HTTP ports, because the proxy URL needs only the Pod ID. + + + + ```python + import os + + public_url = f"http://{os.environ['PUBLIC_IPADDR']}:{os.environ['VAST_TCP_PORT_8000']}" + ``` + + + ```python + import os + + public_url = f"https://{os.environ['RUNPOD_POD_ID']}-8000.proxy.runpod.net" + ``` + + + +Scripts that shut the instance down from inside (`vastai stop instance $CONTAINER_ID`) can call REST API v2 instead. Runpod sets a Pod-scoped key in `RUNPOD_API_KEY`. Test that it's allowed to stop its own Pod in your setup, and pass your own key as a secret if it isn't. + +```bash +curl -s -X POST "https://api.runpod.io/v2/pods/$RUNPOD_POD_ID/action" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"action": "stop"}' +``` + +### Port a PyWorker to a Runpod worker + +A PyWorker is an HTTP proxy that sits in front of your model server, reports load, and runs a benchmark before the worker is marked ready. Runpod workers have no proxy layer, so `worker.py` goes away. What replaces it depends on the endpoint type. + +Here's a typical PyWorker for a model server with one `/generate` route. + +```python worker.py +from vastai import Worker, WorkerConfig, HandlerConfig, BenchmarkConfig, LogActionConfig + +worker_config = WorkerConfig( + model_server_url="http://127.0.0.1", + model_server_port=18000, + model_log_file="/var/log/model/server.log", + handlers=[ + HandlerConfig( + route="/generate", + allow_parallel_requests=False, + max_queue_time=60.0, + workload_calculator=lambda payload: 100.0, + benchmark_config=BenchmarkConfig( + dataset=[{"prompt": "hello world", "max_new_tokens": 64}], + runs=8, + concurrency=1, + ), + ), + ], + log_action_config=LogActionConfig( + on_load=["Application startup complete."], + on_error=["Traceback (most recent call last):"], + ), +) + +Worker(worker_config).run() +``` + + + + A queue endpoint calls a handler function once per job. Load the model at import time so it stays in memory between jobs, and return the result. The file below is complete: replace the pipeline with your own model code. + + ```python handler.py + import runpod + from transformers import pipeline + + # Loaded once per worker, reused for every job + generator = pipeline("text-generation", model="Qwen/Qwen2.5-0.5B-Instruct", device=0) + + + def handler(job): + job_input = job["input"] + result = generator( + job_input["prompt"], + max_new_tokens=job_input.get("max_new_tokens", 64), + ) + return {"text": result[0]["generated_text"]} + + + runpod.serverless.start({"handler": handler}) + ``` + + The image's start command runs `python -u handler.py`. `allow_parallel_requests=False` matches the default of one job per worker at a time. To process several jobs at once on a worker, see [concurrent handlers](/serverless/workers/concurrent-handler). + + + A load-balancing endpoint sends HTTP requests straight to your server's own paths. Keep the model server, bind it to the port in `PORT`, and add a `/ping` route that returns `200` when the model is ready. + + ```python server.py + import os + + import uvicorn + from fastapi import FastAPI + from pydantic import BaseModel + from transformers import pipeline + + app = FastAPI() + generator = pipeline("text-generation", model="Qwen/Qwen2.5-0.5B-Instruct", device=0) + + + class GenerateRequest(BaseModel): + prompt: str + max_new_tokens: int = 64 + + + @app.get("/ping") + def ping(): + return {"status": "healthy"} + + + @app.post("/generate") + def generate(request: GenerateRequest): + result = generator(request.prompt, max_new_tokens=request.max_new_tokens) + return {"text": result[0]["generated_text"]} + + + if __name__ == "__main__": + uvicorn.run(app, host="0.0.0.0", port=int(os.getenv("PORT", "80"))) + ``` + + Clients call `https://ENDPOINT_ID.api.runpod.ai/generate`. If you change `PORT`, set `PORT_HEALTH` to the same value. The `/ping` route does the job of the PyWorker's `on_load` log match. + + + +The PyWorker pieces map as follows. + +| PyWorker piece | Runpod equivalent | +| --- | --- | +| `HandlerConfig(route=...)` | The handler function (queue), or a route on your server (load balancing). | +| `LogActionConfig.on_load` | Model loaded before `runpod.serverless.start` (queue), or `/ping` returning `200` (load balancing). | +| `LogActionConfig.on_error` | Raise an exception in the handler, or fail the `/ping` check. | +| `workload_calculator`, `BenchmarkConfig` | Not needed. Scaling uses queue delay or request count. | +| `allow_parallel_requests=True` | A concurrent handler (queue), or your server's own concurrency (load balancing). | +| `max_queue_time` | The endpoint's execution timeout and job TTL settings. See [endpoint settings](/serverless/endpoints/endpoint-configurations). | +| `PYWORKER_REPO` cloned at start | Bake the code into the image, or deploy from GitHub. See [deploy workers](/serverless/workers/deploy). | + +### Port a Deployment to a Runpod SDK app + +Vast Deployments (beta) and Runpod SDK apps both turn a decorated Python function into a remote endpoint. + + + + ```python deploy.py + from vastai import Deployment + from vastai.data.query import gpu_name, RTX_4090, RTX_5090 + + app = Deployment(name="square") + + + @app.remote(benchmark_dataset=[{"x": 2}]) + async def square(x): + return x * x + + + image = app.image("vastai/base-image:@vastai-automatic-tag", 16) + image.require(gpu_name.in_([RTX_4090, RTX_5090])) + app.configure_autoscaling(min_load=1000) + app.ensure_ready() + ``` + + + ```python main.py + import runpod + + app = runpod.App("square") + + + @app.queue(gpu="4090", workers=(0, 3)) + def square(x: int): + return x * x + + + @runpod.local_entrypoint + def main(): + print(square.remote(5)) + ``` + + + +Run the Runpod version with `rp flash dev main.py` while you iterate and `rp flash deploy` to create the production endpoint. There's no benchmark dataset, the `gpu` argument replaces `image.require` (pass a list to allow several GPUs), and `workers=(0, 3)` replaces the load-based autoscaling settings. An `@app.context` class that loads a model once per worker maps most closely to an `@app.api` class with a `@runpod.init` method, which runs once when each worker starts. See [Build an app](/sdk/apps/build-app). + +## Step 5: Move your data and models + +### Choose the storage + +| What you have on Vast.ai | Where it goes on Runpod | +| --- | --- | +| Scratch data on the instance disk | Container disk. It's erased when the Pod stops. | +| Data on the instance disk that you keep across stops | Volume disk at `/workspace`. It's deleted when you delete the Pod. | +| A local volume reused across instances on one machine | Network volume, which outlives Pods and attaches to any Pod or worker in its data center. | +| Datasets and weights in S3, R2, or GCS, pulled at boot | Keep them there, or copy them once to a network volume or global volume to skip the download at start. | +| Weights pulled from Hugging Face at boot on Serverless | [Cached models](/serverless/endpoints/model-caching) (`--model-reference`), so workers start with the weights already on the host. | + +A network volume lives in one data center, and every Pod or worker that mounts it must run there. Pick the data center by GPU stock first (`runpodctl datacenter list`), then create the volume. If you want to use the S3-compatible API, choose a data center [where it's available](/storage/s3-api#datacenter-availability). + +```bash +runpodctl network-volume create --name "models" --size 200 --data-center-id "US-KS-2" +``` + +A network volume mounts at `/workspace` on Pods and at `/runpod-volume` on Serverless workers. If your Vast volume was mounted at `/data` or `/mnt`, either update the paths in your code or mount the Runpod volume at the old path with `--volume-mount-path` (volume disk) or REST `mounts.network[].path` (network volume). A [global volume](/storage/globalvolume/overview) isn't pinned to a data center and suits read-heavy data such as model weights. It's object-backed, so check its limitations before you use it for write-heavy work. + +### Transfer the data + +Pick the route that matches where the data is now. + + + + If your Vast instances pulled from S3, R2, or GCS, the fastest path is to pull from the same place inside a Pod with the volume attached. No data passes through your machine. + + ```bash + # On a Pod with the network volume mounted at /workspace + aws s3 sync s3://my-bucket/datasets /workspace/datasets + rclone copy r2:my-bucket/models /workspace/models --transfers 16 + hf download Qwen/Qwen2.5-7B-Instruct --local-dir /workspace/models/qwen2.5-7b + ``` + + Runpod's [Cloud Sync](/pods/storage/cloud-sync) does the same from the console for Amazon S3, Google Cloud Storage, Azure Blob Storage, Backblaze B2, and Dropbox. Vast's Cloud Sync also connects Google Drive and Hugging Face buckets, which Runpod's doesn't: use `rclone` or `hf download` in the Pod for those. + + + To copy straight from a running Vast instance to a running Pod, use `runpodctl send` and `receive`. It needs no API key and no open port on either side. + + ```bash + # On the Vast instance (install runpodctl there first) + curl -fsSL https://cli.runpod.net | bash + cd /workspace && runpodctl send ./checkpoints + # prints a one-time code + + # On the Pod + cd /workspace && runpodctl receive CODE + ``` + + For large or resumable transfers, use `rsync` over the Pod's direct SSH port. The Pod needs `22/tcp` in its ports. Read the host and port from `runpodctl ssh info POD_ID`. + + ```bash + # On the Vast instance + rsync -avzP --inplace -e "ssh -p POD_SSH_PORT" /workspace/ root@POD_IP:/workspace/ + ``` + + Vast bills bandwidth per GB at the host's rate, so check `inet_up_cost` on the instance before you push a large dataset out of it. + + + The [S3-compatible API](/storage/s3-api) writes to a network volume without a Pod. It needs an S3 API key from the [Credentials page](/get-started/credentials) (separate from your Runpod API key) and a volume in a data center with the S3-compatible API. Configure the AWS CLI with that key as described in [Setup and authentication](/storage/s3-api#setup-and-authentication). + + The bucket name is the volume ID, and the region is the data center ID. + + ```bash + # First pull the data down from Vast + vastai copy C.INSTANCE_ID:/workspace/datasets local:./datasets + + # Then push it to the volume + aws s3 sync ./datasets s3://VOLUME_ID/datasets \ + --region US-KS-2 \ + --endpoint-url https://s3api-us-ks-2.runpod.io/ + + # List what landed + aws s3 ls s3://VOLUME_ID/datasets/ \ + --region US-KS-2 \ + --endpoint-url https://s3api-us-ks-2.runpod.io/ + ``` + + With `aws s3 sync`, very large file counts (10,000 or more) can run into issues, so archive small files into a tar first and copy that with `aws s3 cp`. + + + Vast volumes can't be copied to a local path. Copy the volume to a running instance or to a cloud connection first, then use one of the other routes. + + ```bash + vastai copy V.VOLUME_ID:/data/ C.INSTANCE_ID:/workspace/data/ + ``` + + + +After a transfer, compare file counts and sizes on both sides (`du -sh` and `find . -type f | wc -l`) before you delete anything on Vast. + +## Step 6: Deploy on Runpod + +### Pods + +Create the Pod with the image, ports, and storage you settled on. This example attaches the network volume from Step 5 and opens one HTTP port and direct SSH. + +```bash +runpodctl pod create \ + --name "trainer" \ + --image "runpod/pytorch:1.0.3-cu1281-torch291-ubuntu2404" \ + --gpu-id "NVIDIA GeForce RTX 4090" \ + --min-cuda-version 12.8 \ + --container-disk-in-gb 30 \ + --network-volume-id VOLUME_ID \ + --data-center-ids "US-KS-2" \ + --ports "8888/http,22/tcp" \ + --env '{"MODE":"fast"}' +``` + +The command prints the Pod as JSON, including its `id`. A network volume takes the place of the volume disk, so don't pass `--volume-in-gb` with it, and set `--data-center-ids` to the volume's data center. Wait for the container before you connect. + +```bash +until [ "$(runpodctl pod get POD_ID --output json | jq -r '.runtimeStatus')" = "running" ]; do + sleep 5 +done +runpodctl ssh info POD_ID # prints the ssh command +runpodctl pod logs POD_ID # container and system logs +``` + +`runtimeStatus: running` means the container is up, not that your service is listening. Probe the port before you send traffic. In the console, the same flow is **Pods**, then **Deploy**. For every flag, see the [`runpodctl pod` reference](/runpodctl/reference/runpodctl-pod). + +### Serverless queue endpoints + +A Vast endpoint and its workergroup collapse into one Runpod endpoint. Build and push the worker image from Step 4, create a Serverless template for it, then create the endpoint. + + + + ```bash + vastai create endpoint \ + --endpoint_name "my-endpoint" \ + --max_workers 5 \ + --cold_workers 1 \ + --target_util 0.9 + + vastai create workergroup \ + --endpoint_name "my-endpoint" \ + --search_params "gpu_ram>=24 num_gpus=1 rentable=true verified=true" \ + --launch_args "--image my-org/my-worker:1.0 --disk 64 --env '-e PYWORKER_REPO=https://github.com/my-org/my-pyworker -e MODEL_NAME=my-model'" \ + --gpu_ram 24 + ``` + + + ```bash + runpodctl template create \ + --name "my-worker" \ + --image "my-org/my-worker:1.0" \ + --container-disk-in-gb 64 \ + --env '{"MODEL_NAME":"my-model"}' \ + --serverless + # copy the template ID from the output + + runpodctl serverless create \ + --name "my-endpoint" \ + --template-id TEMPLATE_ID \ + --gpu-id "ADA_24" \ + --workers-min 0 \ + --workers-max 5 \ + --idle-timeout 60 \ + --scale-by delay \ + --scale-threshold 4 + ``` + + + ```bash + curl -s -X POST "https://api.runpod.io/v2/serverless" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "my-endpoint", + "type": "QUEUE", + "image": "my-org/my-worker:1.0", + "disk": 64, + "env": {"MODEL_NAME": "my-model"}, + "gpu": {"pools": ["ADA_24"], "count": 1}, + "workers": {"min": 0, "max": 5, "idleTimeout": 60}, + "scaling": {"type": "QUEUE_DELAY", "queueDelay": 4} + }' + ``` + + + +`gpu_ram>=24` becomes a GPU pool. `max_workers` becomes the maximum worker count. Vast's `cold_workers` (stopped workers kept ready) has no direct counterpart: Runpod workers scale to zero and start on demand, and a minimum worker count above zero keeps workers running and billing. With `--template-id`, environment variables come from the template, so set them there. + +For a vLLM endpoint, skip the custom image and deploy the vLLM worker from the Hub with a cached model. + +```bash +runpodctl hub search vllm # find the Hub ID +runpodctl serverless create --hub-id HUB_ID --name "my-vllm" \ + --gpu-id "ADA_24" \ + --env MODEL_NAME=Qwen/Qwen2.5-7B-Instruct \ + --model-reference https://huggingface.co/Qwen/Qwen2.5-7B-Instruct:main +``` + +### Serverless load-balancing endpoints + +For the HTTP server from Step 4, create the endpoint with `type` set to `LOAD_BALANCER`. Load-balancing endpoints scale on request count. + +```bash +curl -s -X POST "https://api.runpod.io/v2/serverless" \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{ + "name": "my-api", + "type": "LOAD_BALANCER", + "image": "my-org/my-server:1.0", + "disk": 30, + "env": {"PORT": "8000", "PORT_HEALTH": "8000"}, + "gpu": {"pools": ["ADA_24"], "count": 1}, + "workers": {"min": 0, "max": 3}, + "scaling": {"type": "REQUEST_COUNT", "requestCount": 4} + }' +``` + +The response includes the endpoint `id`. Requests go to `https://ENDPOINT_ID.api.runpod.ai/PATH`. Load-balancing endpoints have request and processing time limits, listed in the [load-balancing overview](/serverless/load-balancing/overview). + +### Run-to-completion jobs + +A script that rented an instance, ran one job, and destroyed it can become a function. `@app.task` provisions a Pod for each call and terminates it when the function returns. + +```python main.py +import runpod + +app = runpod.App("report-job") + + +@app.task(gpu="4090", env={"MODE": "fast"}, container_disk_gb=30) +def report(): + import os + + import torch + + return {"gpu": torch.cuda.get_device_name(0), "mode": os.environ["MODE"]} + + +@runpod.local_entrypoint +def main(): + print(report.remote()) +``` + +Run it with `rp flash dev main.py`. `torch` is already installed on the SDK's built-in GPU images. If you need to keep full control of the Pod lifecycle, use the REST version in Step 7. + +## Step 7: Update the code that calls Vast.ai + +### Serverless clients + +A Vast client resolves an endpoint by name and sends a request to a route on a worker. A Runpod queue client posts a job to a fixed URL that includes the endpoint ID. + + + + ```python + import asyncio + + from vastai import Serverless + + + async def main(): + client = Serverless() # reads VAST_API_KEY + endpoint = await client.get_endpoint(name="my-endpoint") + result = await endpoint.request( + "/generate", + {"prompt": "Explain quantum computing", "max_new_tokens": 100}, + cost=100, + ) + print(result["response"]) + await client.close() + + + asyncio.run(main()) + ``` + + + ```python + import os + + import requests + + response = requests.post( + "https://api.runpod.ai/v2/ENDPOINT_ID/runsync", + headers={"Authorization": f"Bearer {os.environ['RUNPOD_API_KEY']}"}, + json={"input": {"prompt": "Explain quantum computing", "max_new_tokens": 100}}, + timeout=120, + ) + response.raise_for_status() + job = response.json() + print(job["status"], job.get("output")) + ``` + + + ```python + import runpod + + endpoint = runpod.Endpoint("ENDPOINT_ID") # reads RUNPOD_API_KEY or the rp login key + result = endpoint.run_sync({"prompt": "Explain quantum computing", "max_new_tokens": 100}) + print(result) + ``` + + + +What changes in the client: + +- **Addressing.** Endpoints are addressed by ID, not name. Store the ID in configuration. +- **Body.** The HTTP body wraps your payload in `input`. There's no `cost` argument. +- **Response.** A queue endpoint returns a job object with `id`, `status`, and `output`. Vast's wrapper with `response` goes away. +- **Long jobs.** Use `/run` to get a job ID, then poll `/status/JOB_ID`, read `/stream/JOB_ID`, or pass a `webhook` URL in the request. `/runsync` holds the connection open for short jobs. +- **Retries.** The Vast SDK retries when a worker returns `429`. With a queue endpoint, jobs wait in the queue for a worker instead. Check the job `status` (`IN_QUEUE`, `IN_PROGRESS`, `COMPLETED`, `FAILED`, `CANCELLED`, `TIMED_OUT`). +- **Sessions.** Vast worker sessions pin requests to one worker. On a load-balancing endpoint, use [worker affinity](/serverless/load-balancing/worker-affinity). + +For OpenAI-compatible clients, swap the base URL and key, and send the real model name. + + + + ```python + from openai import OpenAI + + client = OpenAI( + api_key="YOUR_VAST_API_KEY", + base_url="https://openai.vast.ai/ENDPOINT_NAME", + ) + response = client.chat.completions.create( + model="", + messages=[{"role": "user", "content": "Hello"}], + ) + ``` + + + ```python + import os + + from openai import OpenAI + + client = OpenAI( + api_key=os.environ["RUNPOD_API_KEY"], + base_url="https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1", + ) + response = client.chat.completions.create( + model="Qwen/Qwen2.5-7B-Instruct", + messages=[{"role": "user", "content": "Hello"}], + ) + ``` + + + +Vast's proxy ignores `model`. Runpod's vLLM worker requires the deployed model name, or the name you set with `OPENAI_SERVED_MODEL_NAME_OVERRIDE`. See [OpenAI compatibility](/serverless/vllm/openai-compatibility). + +### Control-plane code: a complete Python program + +This program rents a GPU with a given image, environment variable, start command, and HTTP port, waits for the job's output, saves it, and deletes the GPU. The first version uses the `vastai` Python SDK. The second does the same job against Runpod REST API v2. + +```python before_vast.py +import os +import time + +import requests +from vastai import VastAI + +vast = VastAI(api_key=os.environ["VAST_API_KEY"]) +IMAGE = "pytorch/pytorch:2.4.0-cuda12.4-cudnn9-runtime" +START = ( + "mkdir -p /workspace/out && cd /workspace/out && " + 'python -c "import torch; print(torch.cuda.get_device_name(0))" > report.txt && ' + 'echo "mode=$MODE" >> report.txt && python -m http.server 8000' +) + +offers = vast.search_offers( + query="gpu_name=RTX_4090 num_gpus=1 reliability>0.98 direct_port_count>=1 dph_total<=0.60", + order="dph_total", + limit=5, + storage=30, +) +instance_id = None +for offer in offers: + created = vast.create_instance( + id=offer["id"], + image=IMAGE, + disk=30, + label="report-job", + env="-e MODE=fast -p 8000:8000", + args=["bash", "-c", START], + cancel_unavail=True, + ) + if created.get("new_contract"): + instance_id = created["new_contract"] + break +if instance_id is None: + raise SystemExit("No offer could be rented") + +try: + deadline = time.time() + 20 * 60 + saved = False + while not saved and time.time() < deadline: + instance = vast.show_instance(id=instance_id) or {} + status = instance.get("actual_status") + if status in ("exited", "unknown", "offline"): + raise RuntimeError(f"Instance {instance_id} failed: {status}") + port = (instance.get("ports") or {}).get("8000/tcp") + if status == "running" and port: + url = f"http://{instance['public_ipaddr'].strip()}:{port[0]['HostPort']}/report.txt" + try: + response = requests.get(url, timeout=10) + if response.ok: + with open("report.txt", "w") as f: + f.write(response.text) + saved = True + except requests.RequestException: + pass + if not saved: + time.sleep(10) + if not saved: + raise RuntimeError("Timed out waiting for report.txt") +finally: + vast.destroy_instance(id=instance_id) + +print(open("report.txt").read()) +``` + +```python after_runpod.py +import os +import time + +import requests + +API = "https://api.runpod.io/v2" +HEADERS = {"Authorization": f"Bearer {os.environ['RUNPOD_API_KEY']}"} +IMAGE = "pytorch/pytorch:2.4.0-cuda12.4-cudnn9-runtime" +START = ( + "mkdir -p /workspace/out && cd /workspace/out && " + 'python -c "import torch; print(torch.cuda.get_device_name(0))" > report.txt && ' + 'echo "mode=$MODE" >> report.txt && python -m http.server 8000' +) +# GPU types you accept, in order of preference. This replaces the offer search. +GPU_CANDIDATES = ["NVIDIA GeForce RTX 4090", "NVIDIA RTX A5000", "NVIDIA GeForce RTX 3090"] + + +def create_pod(): + for gpu_id in GPU_CANDIDATES: + response = requests.post( + f"{API}/pods", + headers=HEADERS, + json={ + "name": "report-job", + "image": IMAGE, + "gpu": {"id": gpu_id, "count": 1, "minCudaVersion": "12.4"}, + "disk": 30, + "env": {"MODE": "fast"}, + "ports": ["8000/http"], + "cmd": ["bash", "-c", START], + }, + timeout=60, + ) + if response.status_code == 201: + return response.json() + if response.status_code in (400, 403): + continue # couldn't be placed, or no access to this GPU type: try the next one + response.raise_for_status() # 401, 402, 422, 429: fix the problem, don't loop + raise SystemExit("None of the candidate GPU types could be placed") + + +def delete_pod(pod_id): + requests.delete(f"{API}/pods/{pod_id}", headers=HEADERS, timeout=60).raise_for_status() + for _ in range(12): # confirm it's gone + if requests.get(f"{API}/pods/{pod_id}", headers=HEADERS, timeout=30).status_code == 404: + return + time.sleep(5) + raise RuntimeError(f"Pod {pod_id} still exists. Check the Runpod console.") + + +pod = create_pod() +pod_id = pod["id"] +print(f"Created Pod {pod_id} on {pod['gpu']['id']} at ${pod['cost']}/hour") + +try: + url = f"https://{pod_id}-8000.proxy.runpod.net/report.txt" + deadline = time.time() + 20 * 60 + saved = False + while not saved and time.time() < deadline: + status = requests.get(f"{API}/pods/{pod_id}", headers=HEADERS, timeout=30).json()["status"] + if status in ("ERROR", "EXITED", "TERMINATED"): + raise RuntimeError(f"Pod {pod_id} failed: {status}") + if status == "RUNNING": + try: + response = requests.get(url, timeout=10) + if response.ok: + with open("report.txt", "w") as f: + f.write(response.text) + saved = True + except requests.RequestException: + pass + if not saved: + time.sleep(10) + if not saved: + raise RuntimeError("Timed out waiting for report.txt") +finally: + delete_pod(pod_id) + +print(open("report.txt").read()) +``` + +What changed: + +- The offer search and price cap are gone. You list the GPU types you accept, and a `400` on create means that type couldn't be placed (and a `403` that your account can't use it), so you try the next. A failed create leaves nothing behind. The Pod's hourly rate comes back in `cost`. To cap spend, read `price` from `GET /v2/catalog/gpus` and filter your candidates before you create. +- `env` is a JSON object, and the port moves out of the Docker options into `ports` with a type. +- `args` becomes `cmd`. +- The port lookup (`public_ipaddr` plus `ports["8000/tcp"]`) becomes a URL built from the Pod ID, over HTTPS. +- `actual_status` becomes `status`, with different values. +- `/workspace` here is on the container disk, which is fine for a job that's deleted at the end. To keep outputs after the Pod is deleted, attach a network volume. + +### Control-plane code: a Bash batch job + +This script uploads inputs, runs a GPU job over SSH, downloads the results, and deletes the Pod. If the job or the download fails, it keeps the Pod so you can recover the results. It needs `runpodctl`, `curl`, `jq`, `ssh`, and `scp`, and an SSH key registered with `runpodctl ssh add-key`. + +```bash batch_job.sh +#!/usr/bin/env bash +set -euo pipefail + +API="https://api.runpod.io/v2" +IMAGE="runpod/pytorch:1.0.3-cu1281-torch291-ubuntu2404" +GPU_ID="NVIDIA GeForce RTX 4090" +JOB_SECONDS=3600 # the job is stopped after this long +NAME="my-job-$(date +%s)-$RANDOM" # unique per run, so you can always find what you created +OPTS=(-o StrictHostKeyChecking=accept-new -o ConnectTimeout=10 -o ServerAliveInterval=30 -o ServerAliveCountMax=4) +mkdir -p inputs && echo "hello" > inputs/example.txt # sample input: replace with your files + +pod_json() { curl -fsS "$API/pods/$POD_ID" -H "Authorization: Bearer $RUNPOD_API_KEY"; } + +# 1. Create the Pod with a volume disk at /workspace and direct SSH on 22/tcp +POD_ID=$(runpodctl pod create \ + --name "$NAME" \ + --image "$IMAGE" \ + --gpu-id "$GPU_ID" \ + --min-cuda-version 12.8 \ + --container-disk-in-gb 20 \ + --volume-in-gb 30 \ + --ports "22/tcp" | jq -r '.id') +[ -n "$POD_ID" ] && [ "$POD_ID" != "null" ] || { echo "Pod create failed" >&2; exit 1; } +echo "Created Pod $POD_ID ($NAME)" + +# 2. On exit, delete the Pod unless its results still need recovery +KEEP=0 +cleanup() { + if [ "$KEEP" = 1 ]; then + echo "Kept Pod $POD_ID because its results weren't downloaded. It bills until you delete it." >&2 + echo "Copy /workspace/outputs off it, then run: runpodctl pod delete $POD_ID" >&2 + return + fi + runpodctl pod delete "$POD_ID" >/dev/null || true + for _ in $(seq 12); do + CODE=$(curl -s -o /dev/null -w '%{http_code}' "$API/pods/$POD_ID" -H "Authorization: Bearer $RUNPOD_API_KEY") + [ "$CODE" = "404" ] && return 0 + sleep 5 + done + echo "Deletion not confirmed for Pod $POD_ID. Check the Runpod console." >&2 + exit 1 +} +trap cleanup EXIT + +# 3. Wait for RUNNING and a direct SSH mapping (up to 15 minutes) +HOST="" PORT="" +for _ in $(seq 90); do + POD=$(pod_json) + STATUS=$(jq -r '.status' <<<"$POD") + case "$STATUS" in ERROR|EXITED|TERMINATED) echo "Pod failed: $STATUS" >&2; exit 1;; esac + HOST=$(jq -r '.ssh.direct.host // empty' <<<"$POD") + PORT=$(jq -r '.ssh.direct.port // empty' <<<"$POD") + [ "$STATUS" = "RUNNING" ] && [ -n "$HOST" ] && [ -n "$PORT" ] && break + sleep 10 +done +[ -n "$HOST" ] && [ -n "$PORT" ] || { echo "Timed out waiting for the Pod" >&2; exit 1; } + +# 4. Wait for sshd to answer +for _ in $(seq 18); do ssh "${OPTS[@]}" -p "$PORT" "root@$HOST" true 2>/dev/null && break; sleep 10; done +ssh "${OPTS[@]}" -p "$PORT" "root@$HOST" true || { echo "SSH never came up on Pod $POD_ID" >&2; exit 1; } + +# 5. Upload, run the job with a time limit, download the results +scp "${OPTS[@]}" -P "$PORT" -r ./inputs "root@$HOST:/workspace/" +KEEP=1 +ssh "${OPTS[@]}" -p "$PORT" "root@$HOST" "timeout $JOB_SECONDS bash -c \ + 'cd /workspace && mkdir -p outputs && nvidia-smi > outputs/gpu.txt && cp inputs/* outputs/'" +scp "${OPTS[@]}" -P "$PORT" -r "root@$HOST:/workspace/outputs" ./ +KEEP=0 +echo "Results are in ./outputs" +``` + +Compared with a Vast version of this job, there's no offer loop and no `--cancel-unavail`. `/workspace` exists because the Pod has a volume disk. The SSH host and port come from the Pod's `ssh.direct` block, which is populated once `22/tcp` is mapped. Direct SSH needs a public IP, which Secure Cloud (the default) provides. If you only need to run a short command and read its output, `POST /v2/pods/POD_ID/exec` runs an argv array without SSH and waits up to 25 seconds. + +For other calls, use the [CLI command map](#cli-command-map) and the [API map](#api-map). + +## Step 8: Verify, cut over, and shut down + +**Verify from outside.** Send a real request from a machine that isn't on Runpod, using the production payload. + +```bash +# Pod HTTP service +curl -s https://POD_ID-8000.proxy.runpod.net/v1/models + +# Serverless queue endpoint +curl -s -X POST https://api.runpod.ai/v2/ENDPOINT_ID/runsync \ + -H "Authorization: Bearer $RUNPOD_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"input": {"prompt": "Hello"}}' + +# Endpoint health and worker counts +curl -s https://api.runpod.ai/v2/ENDPOINT_ID/health -H "Authorization: Bearer $RUNPOD_API_KEY" +``` + +Compare outputs against Vast for the same inputs and seeds. Measure latency for both a warm request and a cold start, because Vast's cold workers and Runpod's scale-to-zero workers behave differently on the first request after idle. + +**Run both in parallel.** Keep the Vast deployment serving while you shift a share of traffic to Runpod at your load balancer or in client configuration. Watch error rates, queue delay, and job `status` values, then move the rest. + +**Set a cost guard on Runpod.** Set `workers.max` and the idle timeout on each endpoint, enable low balance notifications and auto-pay on the [billing page](/accounts-billing/billing), and review `runpodctl pod list --all` and `runpodctl billing pods` after the first day. A stopped Pod still bills for its volume disk, and a network volume bills until you delete it. + +**Shut down Vast.** Stopped Vast instances and volumes keep billing for storage, so destroy them once the data is verified on Runpod. Delete endpoints and workergroups first, then destroy any worker instances they leave behind. + +```bash +# Serverless +vastai show endpoints --raw | jq -r '.[].id' +vastai delete workergroup WORKERGROUP_ID +vastai delete endpoint ENDPOINT_ID + +# Instances (this deletes their disks) +vastai show instances --raw | jq -r '.[] | "\(.id)\t\(.label)\t\(.actual_status)"' +vastai destroy instance INSTANCE_ID -y + +# Volumes (only after every instance using them is destroyed) +vastai show volumes --raw +vastai delete volume VOLUME_ID + +# Confirm nothing is left, then check the balance +vastai show instances --raw | jq 'length' +vastai show user +``` + +A reserved instance has prepaid credit locked to it. Check Vast's [reserved instances](https://docs.vast.ai/guides/instances/choosing/reserved-instances) page for the refund terms before you destroy one. Finally, turn off autobilling in the Vast console, delete the API keys your automation used, and remove `VAST_API_KEY` from CI secrets. + +## Reference + +### CLI command map + +| Task | Vast.ai CLI | Runpod | +| --- | --- | --- | +| Authenticate | `vastai set api-key KEY` | `export RUNPOD_API_KEY=KEY`, or `rp login` | +| Check the account | `vastai show user` | `runpodctl doctor`, `runpodctl user` | +| Find GPUs | `vastai search offers 'QUERY'` | `runpodctl gpu list`, `runpodctl datacenter list` | +| Create | `vastai create instance OFFER_ID --image IMG --disk 30` | `runpodctl pod create --image IMG --gpu-id "GPU_ID" --container-disk-in-gb 30` | +| Create from a template | `--template_hash HASH` | `--template-id TEMPLATE_ID` | +| List | `vastai show instances` | `runpodctl pod list --all` | +| Inspect | `vastai show instance ID` | `runpodctl pod get POD_ID` | +| Stop, start | `vastai stop instance ID`, `vastai start instance ID` | `runpodctl pod stop POD_ID`, `runpodctl pod start POD_ID` | +| Reboot | `vastai reboot instance ID` | `runpodctl pod restart POD_ID` | +| Rename | `vastai label instance ID NAME` | `runpodctl pod update POD_ID --name NAME` | +| Change the image | `vastai update instance ID --image IMG` | `runpodctl pod update POD_ID --image IMG` | +| Delete | `vastai destroy instance ID -y` | `runpodctl pod delete POD_ID` | +| Logs | `vastai logs ID --tail 100` | `runpodctl pod logs POD_ID` | +| SSH details | `vastai ssh-url ID` | `runpodctl ssh info POD_ID` | +| Register an SSH key | `vastai create ssh-key "KEY"` | `runpodctl ssh add-key --key-file PATH` | +| Run a command | `vastai execute ID 'ls -l'` (limited to `ls`, `rm`, `du`), or SSH | SSH, or `POST /v2/pods/POD_ID/exec` | +| Copy files | `vastai copy SRC DST`, `scp` | `runpodctl send` and `receive`, `scp`, `rsync` | +| Sync with cloud storage | `vastai cloud copy` | Cloud Sync in the console, or `aws s3` and `rclone` in the Pod | +| Create a volume | `vastai create volume OFFER_ID -s GB -n NAME` | `runpodctl network-volume create --name NAME --size GB --data-center-id DC` | +| List, delete volumes | `vastai show volumes`, `vastai delete volume ID` | `runpodctl network-volume list`, `runpodctl network-volume delete VOLUME_ID` | +| Create a template | `vastai create template --name N --image IMG` | `runpodctl template create --name N --image IMG` | +| Find templates | `vastai search templates 'QUERY'` | `runpodctl template search QUERY` | +| Registry login | `--login '-u USER -p TOKEN REGISTRY'` | `runpodctl registry create --name N --username USER --password-stdin` | +| Create a Serverless endpoint | `vastai create endpoint` plus `vastai create workergroup` | `runpodctl serverless create` | +| List endpoints | `vastai show endpoints` | `runpodctl serverless list` | +| Update scaling | `vastai update endpoint ID ...` | `runpodctl serverless update ENDPOINT_ID --workers-max N` | +| Delete an endpoint | `vastai delete endpoint ID` | `runpodctl serverless delete ENDPOINT_ID` | +| Billing history | `vastai show invoices-v1` | `runpodctl billing pods`, `runpodctl billing serverless` | +| JSON output | `--raw` | JSON by default | + +### API map + +Vast's base URL is `https://console.vast.ai/api/v0`. Runpod's is `https://api.runpod.io/v2`. Both take `Authorization: Bearer KEY`. + +| Task | Vast.ai REST and SDK | Runpod REST API v2 | +| --- | --- | --- | +| Search for GPUs | `POST /bundles/`, `search_offers()` | `GET /v2/catalog/gpus?include=AVAILABILITY&product=POD` | +| List data centers | `geolocation` in offers | `GET /v2/catalog/datacenters` | +| Create | `PUT /asks/OFFER_ID/`, `create_instance()` | `POST /v2/pods` | +| List | `GET /api/v1/instances/`, `show_instances()` | `GET /v2/pods` (follow `pagination.nextCursor`) | +| Get one | `GET /instances/ID/`, `show_instance()` | `GET /v2/pods/POD_ID` | +| Stop, start | `PUT /instances/ID/` with `{"state": "stopped"}` or `{"state": "running"}` | `POST /v2/pods/POD_ID/action` with `{"action": "stop"}` or `{"action": "start"}` | +| Restart | `reboot_instance()` | `POST /v2/pods/POD_ID/action` with `{"action": "restart"}` | +| Rename, update | `PUT /instances/ID/` with `{"label": "NAME"}`, `update_instance()` | `PATCH /v2/pods/POD_ID` | +| Delete | `DELETE /instances/ID/`, `destroy_instance()` | `DELETE /v2/pods/POD_ID` (returns `204`) | +| Logs | `logs()` | `GET /v2/pods/POD_ID/logs` (Server-Sent Events) | +| Run a command | `execute()` | `POST /v2/pods/POD_ID/exec` | +| SSH keys | `create_ssh_key()` | `PUT /v2/account/ssh-keys` (replaces the full set) | +| Templates | `create_template()` | `POST /v2/templates` | +| Volumes | `create_volume()` | `POST /v2/network-volumes`, `POST /v2/global-volumes` | +| Registry credentials | `login` on create | `POST /v2/registries` | +| Serverless endpoints | `create_endpoint()`, `create_workergroup()` | `POST /v2/serverless` | +| Endpoint workers | Endpoint workers API | `GET /v2/serverless/ENDPOINT_ID/workers` | +| Account secrets | `create_env_var()` | `POST /v2/account/secrets` | +| Billing | `show_invoices_v1()` | `GET /v2/billing/pods`, `GET /v2/billing/serverless` | +| Change a bid | `PUT /instances/bid_price/ID/` | No equivalent. | + +REST API v2 validates request bodies strictly. A `422` lists each violation in `errors`, and a `400` on Pod create means the body was valid but couldn't be placed. + +### Configuration field map + +| Vast.ai create field | `runpodctl pod create` | REST `POST /v2/pods` | +| --- | --- | --- | +| Offer `id` | `--gpu-id`, `--gpu-count` | `gpu.id`, `gpu.count` | +| `--image` | `--image` | `image` | +| `--label` | `--name` | `name` (required) | +| `--disk` | `--container-disk-in-gb`, `--volume-in-gb` | `disk`, `mounts.persistent.size` and `.path` | +| `--env '-e K=V'` | `--env '{"K":"V"}'` | `env` | +| `--env '-p 8000:8000'` | `--ports "8000/http"` or `"8000/tcp"` | `ports` | +| `--args` | `--docker-args` | `cmd` | +| `--entrypoint` | Set on a template with `--docker-entrypoint` | `entrypoint` | +| `--onstart-cmd` | `--docker-args` | `cmd: ["bash", "-c", "..."]` | +| `--ssh`, `--direct` | `--ssh` (default on), plus `22/tcp` in `--ports` | `startSsh: true`, plus `22/tcp` in `ports` | +| `--jupyter` | Use a template with JupyterLab | `startJupyter: true`, plus `8888/http` in `ports` | +| `--login` | `--registry-auth-id` | `registry` | +| `--template_hash` | `--template-id` | `templateId` | +| `--create-volume`, `--link-volume`, `--mount-path` | `--network-volume-id` | `mounts.network[].volumeId` and `.path` | +| `--bid_price` | No equivalent. | No equivalent. | +| `--cancel-unavail` | Not needed. | Not needed. Optional `queue` to wait for capacity. | +| Search `cuda_vers>=` | `--min-cuda-version` | `gpu.minCudaVersion` | +| Search `geolocation` | `--country-code`, `--data-center-ids` | `dataCenterIds` | +| Search `datacenter=true` | `--cloud-type SECURE` | `cloud: "SECURE"` | + +Serverless fields map like this. + +| Vast.ai Serverless field | `runpodctl serverless create` | REST `POST /v2/serverless` | +| --- | --- | --- | +| `--endpoint_name` | `--name` | `name` | +| `--max_workers` | `--workers-max` | `workers.max` | +| `--min_load` above zero (keeps workers active) | `--workers-min` | `workers.min` | +| `--cold_workers`, `--cold_mult`, `--min_cold_load` | No equivalent. | No equivalent. | +| `--target_util` | `--scale-by`, `--scale-threshold` | `scaling.type` with `queueDelay` or `requestCount` | +| `--inactivity_timeout` | `--idle-timeout` | `workers.idleTimeout` | +| Workergroup `--template_hash` | `--template-id` or `--hub-id` | `templateId` | +| Workergroup `--launch_args "--image ... --disk ..."` | Template `--image`, `--container-disk-in-gb` | `image`, `disk`, `env` | +| Workergroup `--search_params`, `--gpu_ram` | `--gpu-id` (a pool or type ID), `--gpu-count` | `gpu.pools`, `gpu.excludedTypes`, `gpu.count` | +| `search_params` with `geolocation` | `--data-center-ids` | `dataCenterIds` | + +### Environment variable map + +These are the variables each platform sets inside the container. + +| Vast.ai variable | Runpod Pod variable | Notes | +| --- | --- | --- | +| `CONTAINER_ID` | `RUNPOD_POD_ID` | A Vast ID is an integer. A Pod ID is a string. | +| `VAST_CONTAINERLABEL` | `RUNPOD_POD_HOSTNAME` | Both identify the container. The values have different formats. | +| `CONTAINER_API_KEY` | `RUNPOD_API_KEY` | Both are scoped to the instance or Pod. | +| `PUBLIC_IPADDR` | `RUNPOD_PUBLIC_IP` | Set only when the Pod has a public IP. | +| `VAST_TCP_PORT_22` | `RUNPOD_TCP_PORT_22` | Public port mapped to SSH. | +| `VAST_TCP_PORT_X` | `RUNPOD_TCP_PORT_X` | Documented for TCP ports requested above 70000. For HTTP ports, build the proxy URL from `RUNPOD_POD_ID`. | +| `VAST_UDP_PORT_X` | No equivalent. | Pods don't support UDP. | +| `GPU_COUNT` | `RUNPOD_GPU_COUNT` | | +| `SSH_PUBLIC_KEY` | `PUBLIC_KEY` | Authorized SSH public keys. | +| `PYTORCH_VERSION` | `PYTORCH_VERSION` | Set by the image on both. | +| `DATA_DIRECTORY` | No equivalent. | The default persistent path is `/workspace`. | +| `JUPYTER_TOKEN`, `JUPYTER_SERVER_URL`, `JUPYTER_SERVER_ROOT` | `JUPYTER_PASSWORD` | Generated when `startJupyter` is set. | +| `OPEN_BUTTON_PORT`, `JUPYTER_PORT` | No equivalent. | Port labels on a template name the ports shown in the console. | +| No equivalent. | `RUNPOD_DC_ID` | Data center ID. | +| No equivalent. | `RUNPOD_CPU_COUNT` | | +| No equivalent. | `RUNPOD_VOLUME_ID` | Attached network volume ID. | +| No equivalent. | `CUDA_VERSION` | | + +Serverless workers set their own variables. See [Serverless environment variables](/serverless/development/environment-variables). + +### Status map + +Vast reports `actual_status` (what the machine reports) and `intended_status` (what you asked for). Runpod's REST API reports one `status`, and `runpodctl` output adds `runtimeStatus` next to `desiredStatus`. + +| Vast.ai `actual_status` | Runpod REST `status` | `runpodctl` `runtimeStatus` | What to do | +| --- | --- | --- | --- | +| `null`, `created` | `PROVISIONING` | `initializing` | Keep polling, with a time limit. | +| `loading` | `STARTING` | `initializing` | Keep polling. | +| `running` | `RUNNING` | `running` | Connect. Probe your service's port before sending traffic. | +| `stopped` | `EXITED` | `stopped` | Start it or delete it. The volume disk still bills. | +| `exited` (container crashed) | `ERROR` | `error` | Read the logs, then delete or recreate. | +| `unknown`, `offline` (host unreachable) | No direct equivalent. | `unknown` means the runtime lookup failed, not that the host is down. | Keep polling on `unknown`. Treat `ERROR` as terminal. | +| `rebooting` | `STARTING` | `initializing` | Keep polling. | +| `frozen` | No equivalent. | No equivalent. | | +| Destroyed (no longer listed) | `TERMINATED`, then `404` | `terminated`, then not found | Treat `404` as the confirmation that a Pod is gone. | + +Each Pod response also includes `actions`, the state transitions that are valid right now (`start`, `stop`, `restart`, `terminate`). + +Vast Serverless worker states map loosely to Runpod's, because Runpod has no stopped-but-ready worker. + +| Vast.ai worker state | Runpod equivalent | +| --- | --- | +| Ready (`idle`) | A running worker, idle or processing a job. | +| Loading, Model Loading | A worker that's initializing (cold start). | +| Inactive (`stopped`), Unavailable | No equivalent. Idle flex workers shut down after the idle timeout. | +| Error | An unhealthy worker. Check the endpoint's worker list and logs. | + +## Migrate with a coding agent + +A coding agent can run the inventory, propose the mapping, and make the changes while you approve each step that costs money. + +First, install the Runpod skills and connect the MCP server. MCP sign-in uses OAuth ("Sign in with Runpod"), so you don't need an API key to start. For details, see [agent skills](/get-started/agent-skills) and [MCP servers](/get-started/mcp-servers). + + + + ```bash + # Claude Code (the plugin also connects the hosted MCP server) + claude plugin marketplace add runpod/runpod-plugins-official + claude plugin install runpod@runpod + ``` + + + ```bash + # Any other agent (Codex, Cursor, and others): skills, then the hosted MCP server + npx skills add runpod/runpod-plugins-official --skill '*' --agent AGENT_ID --global + npx @runpod/mcp-server@latest add --client CLIENT_ID + ``` + + + +Then open your project in the agent and paste this prompt. + +```text +Migrate this project from Vast.ai to Runpod using the runpod-cloud-migrate skill +and its Vast.ai reference. + +1. Inventory first. Run `vastai show instances --raw`, `vastai show volumes --raw`, + `vastai show endpoints --raw`, and `vastai show workergroups --raw`, and grep the + repo for vastai, VAST_API_KEY, CONTAINER_ID, VAST_TCP_PORT_, VAST_UDP_PORT_, + PUBLIC_IPADDR, and PYWORKER_REPO. Don't change anything yet. +2. Show me a plan: for each instance, endpoint, volume, and call site, the Runpod + target (Pod, Serverless queue endpoint, load-balancing endpoint, @app.task, + network volume, or global volume), the GPU type or pool, and the data center. + Flag anything that uses UDP ports, interruptible bids, reserved instances, VM + instances, or host-level search filters, and tell me the options. +3. Give me a cost estimate from the Runpod GPU catalog for each resource, and wait + for my approval before you create anything billable. +4. Migrate one workload end to end. Replace search-then-create with a create on a + GPU type, move Docker options to env and ports, and replace Vast runtime + variables with the RUNPOD_ ones. +5. Verify with a real request from outside Runpod and show me the output next to + the Vast output. +6. Never stop, destroy, or delete anything on Vast.ai without asking me first. + When I approve, destroy instances and volumes and confirm nothing is left. +7. Finish with a report: files changed, resources created with IDs and hourly + rates, gaps that remain, and what is still running on each platform. +``` + +## Troubleshooting + +**The Pod create call returns `400` for a GPU that the catalog showed as available.** Stock changed between the read and the create, or the GPU isn't available in the data center you pinned. Try the next GPU type or data center in your list, or drop `dataCenterIds` if you don't have a network volume. Don't retry a `422`: it means the body is wrong. + +**The container exits right away or ignores your startup commands.** On Vast, SSH and Jupyter modes replaced the entrypoint and ran your on-start script. On Runpod the image's own `ENTRYPOINT` runs, and `cmd` is passed to it as arguments. If the image has an entrypoint, either pass arguments it understands or override it with `entrypoint`. Check `runpodctl pod logs POD_ID` to see what ran. + +**SSH is refused, or `scp` fails while an interactive shell works.** Proxied SSH carries a shell only. Add `22/tcp` to the Pod's ports for direct SSH, register your key before you create the Pod, and use an image that starts `sshd` from the `PUBLIC_KEY` variable. `runpodctl ssh info POD_ID` reports which condition isn't met. Note that `runpodctl pod update --ports` replaces the whole port list, so include your existing ports. + +**Files are missing after a stop and start.** Only `/workspace` (the volume disk or a network volume) survives a stop. Vast kept the whole instance disk. Move outputs, checkpoints, and caches such as `HF_HOME` under `/workspace`. + +**A service works on the TCP port but returns `524` through the proxy URL.** The HTTP proxy closes connections after 100 seconds. Return a job ID and poll, stream the response, or expose the service on a TCP port instead. + +**A service that listens on UDP is unreachable.** Pods don't route UDP. Switch the service to TCP, or run the UDP-facing part outside Runpod. + +**`nvidia-smi` works but your framework reports a CUDA error.** The host's CUDA version is older than the image expects. Vast's `cuda_vers` filter or `@vastai-automatic-tag` handled this for you. Set `--min-cuda-version` (REST `gpu.minCudaVersion`) to the version the image was built for. + +**A network volume won't attach, or the Pod can't be placed with it.** The Pod must run in the volume's data center, and that data center needs stock of your GPU type. Check `runpodctl datacenter list` before you create the volume, or use a global volume for data that must follow the workload across data centers. + +**Serverless requests sit in `IN_QUEUE`.** No worker has started. Check that the image is `linux/amd64` and pullable, that the GPU pool has stock (add a second pool), and that `workers.max` is above zero. A handler that never calls `runpod.serverless.start` also leaves jobs queued. + +**Code still reads `CONTAINER_ID` or `VAST_TCP_PORT_8000` and gets nothing.** Those variables don't exist on Runpod. Rerun the grep from Step 1 inside the image's source and replace them with the names in the [environment variable map](#environment-variable-map). + +## Next steps + + + + Pick a GPU, cloud tier, and storage for your workload. + + + Learn how endpoints, workers, and handlers fit together. + + + Share persistent data across Pods and Serverless workers. + + + Manage Pods, endpoints, templates, and volumes programmatically. + + From 46e3978bf8fd3296efda2b61ad0b283ea91280a5 Mon Sep 17 00:00:00 2001 From: Luke Piette Date: Sat, 14 Nov 2026 14:44:00 -0500 Subject: [PATCH 2/3] docs(migrate): apply second accuracy review to the migration guides A mechanical pass validated every runpodctl invocation, REST v2 body, SDK symbol and Terraform block against the command surface, the OpenAPI spec, the runpod package source and the provider schemas. Four adversarial passes re-checked the other platforms' facts against their live docs and walked each migration end to end. Data-loss and billing fixes: - CoreWeave: shutdown no longer deletes the PVC with the Deployment - Lambda: copy root-volume data before terminating instances - Google Cloud: warn that terraform import can force Pod replacement - Vast.ai: shutdown now deletes Deployments; rsync step needs a key the Pod accepts - Baseten: orchestrator polls a long job instead of resubmitting it Runpod-side corrections: - vLLM worker settings the current worker ignores (structured output, speculative decoding), and routes it does serve (Responses, Messages) - Remove Public Endpoints that are not in the live catalog - Custom handlers point at the cached-model path so weights are not downloaded on each cold start - Load-balancing endpoints: set ports, and a timeout within the request limit - serverless update scaling flags, zero-balance behavior, global networking scope Co-Authored-By: Claude Fable 5.2 --- migrate/aws.mdx | 25 +++++++++++++------------ migrate/azure.mdx | 15 ++++++++++----- migrate/baseten.mdx | 39 ++++++++++++++++++++++++++------------- migrate/coreweave.mdx | 23 ++++++++++++----------- migrate/fal.mdx | 18 +++++++++++------- migrate/fireworks-ai.mdx | 29 ++++++++++++++--------------- migrate/google-cloud.mdx | 16 ++++++++++------ migrate/hugging-face.mdx | 2 +- migrate/lambda.mdx | 10 ++++++---- migrate/modal.mdx | 8 ++++---- migrate/paperspace.mdx | 24 ++++++++++++------------ migrate/replicate.mdx | 26 ++++++++++++-------------- migrate/together-ai.mdx | 32 +++++++++++++++++--------------- migrate/vast-ai.mdx | 16 +++++++++++----- 14 files changed, 159 insertions(+), 124 deletions(-) diff --git a/migrate/aws.mdx b/migrate/aws.mdx index 1d55bd4dc..9cedfe5e2 100644 --- a/migrate/aws.mdx +++ b/migrate/aws.mdx @@ -23,7 +23,7 @@ You can hand this migration to a coding agent. See [Migrate with a coding agent] | SageMaker asynchronous inference, batch transform | Serverless queue endpoint (`/run` with a webhook) | Jobs queue, and workers scale to zero when idle. | | SageMaker Serverless Inference | Serverless endpoint | SageMaker Serverless Inference doesn't support GPUs. Runpod Serverless runs on GPUs or CPUs. | | SageMaker training and processing jobs | Pod, or `@app.task` in the [Runpod SDK](/sdk/overview) | `@app.task` creates one Pod per call that runs to completion and terminates. | -| SageMaker Studio, notebook instances | Pod with JupyterLab | Runpod official images start JupyterLab on port 8888. | +| SageMaker Studio, notebook instances | Pod with JupyterLab | Runpod official PyTorch images start JupyterLab on port 8888 when `JUPYTER_PASSWORD` is set. The console sets it for you. | | SageMaker JumpStart, Amazon Bedrock | [Public Endpoints](/public-endpoints/overview) or a Serverless [vLLM endpoint](/serverless/vllm/overview) | Public Endpoints cover only the models in the [model reference](/public-endpoints/reference). For any other open-weight model, deploy vLLM. | | AWS Batch, ECS on GPU, EKS GPU node groups | Serverless queue endpoint, Pods, or [SkyPilot](/integrations/skypilot) and [dstack](/integrations/dstack) | Run each container as an endpoint or a Pod. SkyPilot and dstack schedule jobs onto Runpod from their own control planes. | | AWS ParallelCluster, SageMaker HyperPod | [Instant Clusters](/instant-clusters) with [Slurm](/instant-clusters/slurm-clusters) | Multi-node clusters with Slurm preinstalled. Capacity depends on current stock. HyperPod's EKS orchestration has no managed counterpart. | @@ -33,7 +33,7 @@ You can hand this migration to a coding agent. See [Migrate with a coding agent] | Amazon EFS, FSx for Lustre | Network volume, [high-performance storage](/storage/high-performance-storage) | Pinned to one data center. | | Amazon S3 | Keep S3, or a network volume with the [S3-compatible API](/storage/s3-api) | The Runpod S3-compatible API is a file interface to a network volume. It isn't a general object store. | | Security groups, load balancers | [Exposed ports](/pods/configuration/expose-ports), Serverless endpoint URLs | HTTP ports get an HTTPS proxy URL. TCP ports get a public IP and mapped port. | -| VPC private networking | [Global networking](/pods/networking) between Pods | A private network across your own Pods. It doesn't peer with a VPC. | +| VPC private networking | [Global networking](/pods/networking) between Pods | A private network across your own Pods. You enable it when you deploy, on NVIDIA GPU Pods in supported data centers. It doesn't peer with a VPC. | | AWS Secrets Manager, SSM Parameter Store | [Secrets](/pods/templates/secrets) | Referenced from environment variables as `{{ RUNPOD_SECRET_name }}`. | | Regions and Availability Zones | Data centers | List them with `runpodctl datacenter list`. | | On-Demand, per-second billing | On-demand Pods and Serverless, per-second billing | No data ingress or egress fees on Runpod. | @@ -42,8 +42,8 @@ You can hand this migration to a coding agent. See [Migrate with a coding agent] | Terraform AWS provider, CloudFormation | [Runpod Terraform provider](/integrations/terraform/overview) 2.x | Built on REST API v2. | | AWS CLI, boto3, AWS APIs | [runpodctl](/runpodctl/overview), the [Runpod SDK](/sdk/overview), [REST API v2](/api-reference-v2/overview) | One API at `https://api.runpod.io/v2`. | | IAM users, roles, SigV4 | [API keys](/get-started/credentials) sent as a Bearer token | Keys can be scoped as All, Restricted, or Read Only. | -| AWS Organizations, IAM Identity Center | Teams and [organizations](/organizations/orgs-overview) | Organizations have built-in roles (admin, billing, developer, contributor) and custom roles. | -| CloudWatch Logs and metrics | `runpodctl pod logs`, `runpodctl serverless logs`, endpoint metrics | Logs are read live from Pods and workers. To keep history, ship logs from your container to CloudWatch or another log service. | +| AWS Organizations, IAM Identity Center | Teams and [organizations](/organizations/orgs-overview) | Organizations are in beta and provisioned by Runpod. They have built-in roles (admin, billing, developer, contributor) and custom roles. | +| CloudWatch Logs and metrics | `runpodctl pod logs`, `runpodctl serverless logs`, endpoint metrics | Serverless endpoint logs are kept for 90 days. Worker logs are removed when the worker terminates. To keep longer history, ship logs from your container to CloudWatch or another log service. | ## What's different @@ -65,15 +65,15 @@ You can hand this migration to a coding agent. See [Migrate with a coding agent] **Storage is tied to a data center.** A network volume lives in one data center, and anything that mounts it runs there. Choose the data center by GPU stock first, then create the volume. -**Billing is prepaid by default.** Runpod accounts hold a credit balance. When the balance reaches \$0, running Pods are stopped, and Pods without a network volume are terminated. Organizations, the enterprise account model, are invoiced post-paid instead. See [Pod pricing](/pods/pricing). +**Billing is prepaid by default.** Runpod accounts hold a credit balance. When the balance reaches \$0, Pods with a network volume are stopped and Pods without one are terminated. Organizations, the enterprise account model, are invoiced post-paid instead. See [Pod pricing](/pods/pricing). ### What has no direct equivalent On the Runpod side: - **VPC, security groups, PrivateLink.** Runpod has no VPC peering or private link into AWS. Exposed Pod ports are reachable from the internet, and the HTTP proxy URL is unauthenticated, so put your own auth in front of anything you expose. Workaround: call AWS over its public endpoints with scoped credentials, or call your own API gateway. -- **IAM roles and instance profiles.** A Pod or worker has no AWS identity. Workaround: create a narrowly scoped IAM user or role, store its keys as Runpod secrets, and inject them as `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. For single objects, pass presigned S3 URLs in the request instead of credentials. -- **KMS and customer-managed keys.** The Runpod S3-compatible API doesn't support bucket policies or ACLs, so there's nowhere to attach a KMS key policy. Workaround: keep regulated data in S3 under your KMS keys and read it at run time, or encrypt before writing to a volume. +- **IAM roles and instance profiles.** A Pod or worker has no AWS identity. Workaround: create a narrowly scoped IAM user or role, store its keys as Runpod secrets, and inject them as `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`. These are long-lived credentials that work from anywhere, so limit the policy to the buckets and actions the workload needs and rotate the keys. For single objects, pass presigned S3 URLs in the request instead of credentials. +- **KMS and customer-managed keys.** The Runpod S3-compatible API doesn't support bucket policies or ACLs, so there's nowhere to attach a KMS key policy. Runpod can encrypt a Pod's volume disk with a key it manages, but bring-your-own-key isn't supported, and container disks and network volumes can't be encrypted this way. Workaround: keep regulated data in S3 under your KMS keys and read it at run time, or encrypt before writing to a volume. - **Multi-model endpoints, production variants, shadow variants, Model Monitor.** One Runpod endpoint runs one image. Workaround: one endpoint per model, a router in your own code inside a load balancing worker, and [rolling releases](/serverless/endpoints/rolling-releases) for updates. - **Bedrock features beyond inference** (Guardrails, Knowledge Bases, Agents) and models that are only offered through Bedrock. A vLLM endpoint serves open-weight models, and Public Endpoints serve the listed models. Keep the other calls on Bedrock. - **Capacity Blocks, Capacity Reservations, and Savings Plans.** Runpod has no self-serve future-dated reservation, and its savings plans can no longer be purchased. A running on-demand Pod isn't displaced by other users, but a stopped Pod releases its GPU, so starting it again depends on stock at that moment. For committed capacity, talk to Runpod sales. @@ -102,7 +102,7 @@ List running GPU instances with their type, AMI, and Availability Zone: ```bash aws ec2 describe-instances \ - --filters "Name=instance-type,Values=g4dn.*,g5.*,g5g.*,g6.*,g6e.*,g7.*,g7e.*,p3dn.*,p4d.*,p4de.*,p5.*,p5e.*,p5en.*,p6*" \ + --filters "Name=instance-type,Values=g3.*,g3s.*,g4ad.*,g4dn.*,g5.*,g5g.*,g6.*,g6e.*,g6f.*,gr6.*,gr6f.*,g7.*,g7e.*,p2.*,p3.*,p3dn.*,p4d.*,p4de.*,p5.*,p5e.*,p5en.*,p6*" \ "Name=instance-state-name,Values=pending,running,stopping,stopped" \ --query "Reservations[].Instances[].[InstanceId,InstanceType,ImageId,State.Name,Placement.AvailabilityZone,Tags[?Key=='Name']|[0].Value]" \ --output table @@ -657,7 +657,7 @@ curl -s -X POST https://api.runpod.io/v2/serverless \ "gpu": {"pools": ["AMPERE_24"], "count": 1, "minCudaVersion": "12.8"}, "workers": {"min": 0, "max": 3, "idleTimeout": 30}, "scaling": {"type": "REQUEST_COUNT", "requestCount": 4}, - "timeout": 600000 + "timeout": 300000 }' ``` @@ -776,7 +776,7 @@ SageMaker asynchronous inference reads the request from S3, writes the result to -`/run` accepts payloads up to 10 MB, so keep large inputs in S3 and pass a presigned URL, which your handler downloads. Have the handler upload large outputs to S3 with a presigned PUT URL or scoped keys, and return the object location. The webhook receives the job result when it finishes and should return `200`. Without a webhook, poll `GET /v2/ENDPOINT_ID/status/JOB_ID`. Results are kept for 30 minutes after completion. For batch transform, submit one `/run` job per record or per shard and let the endpoint scale out to its max workers. +`/run` accepts payloads up to 10 MB, so keep large inputs in S3 and pass a presigned URL, which your handler downloads. Anyone who holds a presigned URL can use it until it expires, so set an expiry that covers the time the job can wait in the queue and no longer. Have the handler upload large outputs to S3 with a presigned PUT URL or scoped keys, and return the object location. The webhook receives the job result when it finishes and should return `200`. Without a webhook, poll `GET /v2/ENDPOINT_ID/status/JOB_ID`. Results are kept for 30 minutes after completion. For batch transform, submit one `/run` job per record or per shard and let the endpoint scale out to its max workers. ### Replace Bedrock calls @@ -876,6 +876,7 @@ aws sagemaker delete-endpoint --endpoint-name my-endpoint aws sagemaker delete-endpoint-config --endpoint-config-name my-endpoint-config aws sagemaker delete-model --model-name my-model aws sagemaker stop-notebook-instance --notebook-instance-name my-notebook +aws sagemaker wait notebook-instance-stopped --notebook-instance-name my-notebook aws sagemaker delete-notebook-instance --notebook-instance-name my-notebook aws bedrock delete-provisioned-model-throughput --provisioned-model-id PROVISIONED_MODEL_ARN ``` @@ -1066,7 +1067,7 @@ The `depends_on` matters because Runpod validates the repository policy when the -The subnet, security group, and instance profile have no Runpod resource. Ports replace the security group, and secret-backed environment variables replace the instance profile. The key pair becomes account-level SSH keys (`runpod_account_ssh_keys`), which a Pod receives when you set `start_ssh = true`. Changing `gpu_type_id`, `gpu_count`, `data_center_ids`, or `min_cuda_version` replaces the Pod, so keep state you care about on the network volume. To stop a Pod without removing it from the configuration, set `desired_state = "EXITED"`. +The subnet, security group, and instance profile have no Runpod resource. Ports replace the security group, and secret-backed environment variables replace the instance profile. The key pair becomes account-level SSH keys (`runpod_account_ssh_keys`), which a Pod receives when you set `start_ssh = true`. Changing `gpu_type_id`, `gpu_count`, `cloud`, `data_center_ids`, or `min_cuda_version` replaces the Pod, so keep state you care about on the network volume. To stop a Pod without removing it from the configuration, set `desired_state = "EXITED"`. ### SageMaker endpoint with autoscaling to a Serverless endpoint @@ -1168,7 +1169,7 @@ The subnet, security group, and instance profile have no Runpod resource. Ports workers_min = 1 workers_max = 4 idle_timeout = 30 - timeout = 600000 + timeout = 300000 } output "classifier_url" { diff --git a/migrate/azure.mdx b/migrate/azure.mdx index 2d179bfb1..8880c4a44 100644 --- a/migrate/azure.mdx +++ b/migrate/azure.mdx @@ -22,7 +22,7 @@ You can hand most of this migration to a coding agent. See [Migrate with a codin | Azure Machine Learning managed online endpoint (endpoint plus deployments) | [Serverless queue-based endpoint](/serverless/overview) or [load balancing endpoint](/serverless/load-balancing/overview) | A scoring script becomes a handler function. A custom container with its own routes becomes a load balancing endpoint. | | Azure Machine Learning batch endpoint | Queue-based endpoint with `/run` and webhooks | Jobs queue and run asynchronously. You pass data references in the request. | | Azure Machine Learning jobs, compute clusters, pipelines | Pods, or the Runpod SDK `@app.task` | `@app.task` runs one ephemeral Pod per call. Bring your own orchestrator for multi-step pipelines. | -| Azure Machine Learning compute instance | Pod with JupyterLab | Official Runpod images start JupyterLab on port 8888. | +| Azure Machine Learning compute instance | Pod with JupyterLab | Official Runpod PyTorch images start JupyterLab on port 8888 when `JUPYTER_PASSWORD` is set. The console sets it for you. | | Azure OpenAI in Microsoft Foundry Models, Foundry serverless API deployments | [Public Endpoints](/public-endpoints/overview), or a [Serverless vLLM endpoint](/serverless/vllm/get-started) | Applies to open-weight models. See [What has no direct equivalent](#what-has-no-direct-equivalent). | | Azure Container Apps with serverless GPUs | [Load balancing endpoint](/serverless/load-balancing/overview) | Both run your HTTP server and scale to zero. The health check and port contract differ. | | Azure Container Instances GPU | Pods or Serverless | Microsoft's documentation lists GPU container groups as retired as of July 14, 2025. | @@ -74,7 +74,7 @@ You can hand most of this migration to a coding agent. See [Migrate with a codin **On Runpod:** - **OpenAI's proprietary GPT chat models.** Runpod doesn't host the GPT deployments you create in Azure OpenAI. The migration applies when you move to an open-weight model, either one listed in the [Public Endpoint model reference](/public-endpoints/reference) or one you serve yourself with vLLM. If you need to keep a proprietary model, keep that call on Azure. -- **VNet, NSGs, and Private Link.** Runpod has no customer-defined private network or private connectivity to Azure. Pods in your account can reach each other over [global networking](/pods/networking) at `POD_ID.runpod.internal`. Traffic between Runpod and Azure goes over the public internet with TLS, so Azure resources behind private endpoints need a public endpoint with firewall rules, or a proxy you run. +- **VNet, NSGs, and Private Link.** Runpod has no customer-defined private network or private connectivity to Azure. Pods in your account can reach each other over [global networking](/pods/networking) at `POD_ID.runpod.internal`, if you enable it when you deploy them. It's available on NVIDIA GPU Pods in supported data centers. Traffic between Runpod and Azure goes over the public internet with TLS, so Azure resources behind private endpoints need a public endpoint with firewall rules, or a proxy you run. - **Managed identities and Entra ID RBAC for workloads.** A Runpod worker can't assume an Azure identity. Store a least-privilege service principal secret or a SAS token as a Runpod secret and pass it as an environment variable. - **Key Vault.** Runpod secrets store values and inject them at boot. They don't provide key operations, certificates, or versioning. Keep Key Vault for those and copy only the values workloads need. - **Windows and VDI.** NV and NG family workloads, Windows GPU VMs, and Data Science VM Windows images stay on Azure. @@ -428,13 +428,13 @@ Run it with `rp flash dev train.py`. The Pod is created for the call and termina ### Keep data in Blob Storage -In a hybrid setup, the simplest option is to leave data in Blob Storage and read it over HTTPS from Runpod. Create a SAS token scoped to the container and the permissions the workload needs, or a service principal with a narrow role assignment, and store it as a Runpod secret: +In a hybrid setup, the simplest option is to leave data in Blob Storage and read it over HTTPS from Runpod. Create a SAS token scoped to the container and the permissions the workload needs, with an expiry date, or a service principal with a narrow role assignment. Store the container SAS URL (`https://ACCOUNT.blob.core.windows.net/CONTAINER?SAS_TOKEN`) as a Runpod secret, reading it from an environment variable so it stays out of your shell history: ```bash curl -s -X POST https://api.runpod.io/v2/account/secrets \ -H "Authorization: Bearer $RUNPOD_API_KEY" \ -H "Content-Type: application/json" \ - -d '{"name": "azure_container_sas_url", "value": "https://ACCOUNT.blob.core.windows.net/CONTAINER?SAS_TOKEN"}' + -d "{\"name\": \"azure_container_sas_url\", \"value\": \"$AZURE_CONTAINER_SAS_URL\"}" ``` Reference the secret from an environment variable on the Pod, template, or endpoint by setting the value to `{{ RUNPOD_SECRET_azure_container_sas_url }}`. Inside the workload, the Azure SDK reads it like any other URL: @@ -960,6 +960,10 @@ provider "azurerm" { # Reads the key from the RUNPOD_API_KEY environment variable. provider "runpod" {} + +variable "subscription_id" { + type = string +} ``` Keep whatever `azurerm` version constraint you already use. Add the Runpod resources first and apply, verify the workload, move traffic, and only then remove the Azure resource blocks in a separate change, so `terraform destroy` behavior for the Azure side is reviewed on its own. Resources from the two providers can reference each other, for example an Azure storage account name in a Runpod endpoint's `env`. @@ -1064,7 +1068,7 @@ output "pod_id" { } ``` -`start_ssh` injects the SSH keys registered on your Runpod account, which replaces `admin_ssh_key`, and needs an image that honors the convention (the official Runpod images do). `disk` is the ephemeral container disk, so anything that must survive goes on the network volume. Changing `gpu_type_id`, `gpu_count`, `cloud`, or `data_center_ids` replaces the Pod. To stop the Pod without removing it from the configuration, set `desired_state = "EXITED"`. +`start_ssh` injects the SSH keys registered on your Runpod account, which replaces `admin_ssh_key`, and needs an image that honors the convention (the official Runpod images do). `disk` is the ephemeral container disk, so anything that must survive goes on the network volume. Changing `gpu_type_id`, `gpu_count`, `cloud`, `min_cuda_version`, or `data_center_ids` replaces the Pod. To stop the Pod without removing it from the configuration, set `desired_state = "EXITED"`. ### Example 2: Container Apps GPU app to a Serverless endpoint @@ -1159,6 +1163,7 @@ resource "runpod_serverless_endpoint" "api" { name = "gpu-api" template_id = runpod_template.api.id endpoint_type = "LOAD_BALANCER" + ports = ["8000/http"] gpu_pools = ["AMPERE_16"] gpu_count = 1 min_cuda_version = "12.8" diff --git a/migrate/baseten.mdx b/migrate/baseten.mdx index 0ba5d3896..59cef484a 100644 --- a/migrate/baseten.mdx +++ b/migrate/baseten.mdx @@ -81,7 +81,7 @@ You can hand this migration to a coding agent. See [Migrate with a coding agent] | Single-tenant, self-hosted, and hybrid hosting | Runpod runs in Runpod data centers | For compliance details see [Security and compliance](/references/security-and-compliance). | | Prometheus metrics export and OpenTelemetry tracing | Console metrics and REST v2 metrics routes | Export from your own code if you need traces. | -Going the other way, these Runpod features don't have a counterpart in the Baseten docs this guide was checked against: Pods you can SSH into and keep running as long-lived GPU machines, writable shared [network volumes](/storage/network-volumes) (Baseten's BDN volumes are immutable and mounted read-only), and job results you can fetch from `/status` after the fact. +Going the other way, these Runpod features don't have a counterpart in the Baseten docs this guide was checked against: Pods that keep running as long-lived, general-purpose GPU machines outside of a deployment or training job (Baseten's docs describe SSH into running deployments and training jobs), writable shared [network volumes](/storage/network-volumes) (Baseten's BDN volumes are immutable and mounted read-only), and job results you can fetch from `/status` after the fact. ## Requirements @@ -434,7 +434,7 @@ runpod.serverless.start({ ``` -Clients read the chunks from `/stream/JOB_ID`. With `return_aggregate_stream` set, the full output is also available from `/run` and `/runsync`. See [Streaming handlers](/serverless/workers/handler-functions#streaming-handlers). +Clients read the chunks from `/stream/JOB_ID`. With `return_aggregate_stream` set, the full output is also available from `/run` (through `/status`) and `/runsync`, as a list of the yielded chunks instead of one string. See [Streaming handlers](/serverless/workers/handler-functions#streaming-handlers). ### Concurrency @@ -617,11 +617,14 @@ runpod.serverless.start({"handler": handler}) ```python hello_world/handler.py (Runpod) import os +import time import requests import runpod -RAND_INT_URL = f"https://api.runpod.ai/v2/{os.environ['RAND_INT_ENDPOINT_ID']}/runsync" +RAND_INT_BASE = f"https://api.runpod.ai/v2/{os.environ['RAND_INT_ENDPOINT_ID']}" +RAND_INT_URL = f"{RAND_INT_BASE}/runsync" +STATUS_URL = f"{RAND_INT_BASE}/status" HEADERS = {"Authorization": f"Bearer {os.environ['ORCHESTRATOR_API_KEY']}"} @@ -633,8 +636,18 @@ def call_rand_int(max_value, retries=3): json={"input": {"max_value": max_value}}, timeout=120, ) - if response.ok and response.json().get("status") == "COMPLETED": - return response.json()["output"] + response.raise_for_status() + job = response.json() + # /runsync returns before the job finishes if it runs long. The job is + # still running, so poll it instead of submitting a second one. + while job["status"] in ("IN_QUEUE", "IN_PROGRESS"): + time.sleep(1) + status = requests.get(f"{STATUS_URL}/{job['id']}", headers=HEADERS, timeout=30) + status.raise_for_status() + job = status.json() + if job["status"] == "COMPLETED": + return job["output"] + # FAILED, CANCELLED or TIMED_OUT: submit a new job on the next attempt. raise RuntimeError("rand_int did not complete") @@ -653,7 +666,7 @@ What you give up compared with Chains: typed stubs between steps, a single `trus ## Step 5: Move your data and models -**Hugging Face weights.** If your Truss uses `weights` with an `hf://` source, or `model_cache` with a Hugging Face `repo_id`, use a [cached model](/serverless/endpoints/model-caching). Runpod schedules workers on hosts that already hold the model and doesn't bill worker time for the download. Cached models are mounted at `/runpod-volume/huggingface-cache/hub/` in the Hugging Face cache layout, so the path changes from your Truss `mount_location` or `/app/model_cache/VOLUME_FOLDER`. [Use Hugging Face models](/serverless/development/huggingface-models#use-cached-models) has a helper that resolves the snapshot path. Each endpoint takes one cached model. +**Hugging Face weights.** If your Truss uses `weights` with an `hf://` source, or `model_cache` with a Hugging Face `repo_id`, use a [cached model](/serverless/endpoints/model-caching). Runpod tries to start workers on hosts that already hold the model and doesn't bill worker time for the download. Cached models are mounted at `/runpod-volume/huggingface-cache/hub/` in the Hugging Face cache layout, so the path changes from your Truss `mount_location` or `/app/model_cache/VOLUME_FOLDER`. [Use Hugging Face models](/serverless/development/huggingface-models#use-cached-models) has a helper that resolves the snapshot path. Each endpoint takes one cached model. **Weights in S3, GCS, R2, or Azure, and BDN volumes.** Put them on a [network volume](/storage/network-volumes), which mounts at `/runpod-volume` on Serverless workers. A network volume lives in one data center, and an endpoint that attaches it runs only there, so choose a data center that stocks the GPU pool you need. @@ -707,7 +720,7 @@ On Runpod, create a Serverless template for the image, then an endpoint that use --image docker.io/DOCKER_USERNAME/text-generator:v1.0.0 \ --serverless \ --container-disk-in-gb 20 \ - --env '{"MODEL_ID":"HuggingFaceTB/SmolLM2-1.7B-Instruct","HF_TOKEN":"{{ RUNPOD_SECRET_hf_access_token }}"}' + --env '{"MODEL_ID":"HuggingFaceTB/SmolLM2-1.7B-Instruct","HF_TOKEN":"{{ RUNPOD_SECRET_hf_access_token }}","HF_HUB_CACHE":"/runpod-volume/huggingface-cache/hub"}' runpodctl serverless create \ --name text-generator \ @@ -721,7 +734,7 @@ On Runpod, create a Serverless template for the image, then an endpoint that use --model-reference https://huggingface.co/HuggingFaceTB/SmolLM2-1.7B-Instruct:main ``` - `TEMPLATE_ID` comes from the output of the first command. Each Serverless template binds to one endpoint, so create one template per endpoint. For a private image, run `runpodctl registry create` and add `--registry-auth-id` to the template. + `TEMPLATE_ID` comes from the output of the first command. `HF_HUB_CACHE` points `transformers` at the cached model that `--model-reference` attaches. Without it, the handler downloads the weights again on every cold start. Each Serverless template binds to one endpoint, so create one template per endpoint. For a private image, run `runpodctl registry create` and add `--registry-auth-id` to the template. ```bash @@ -759,7 +772,7 @@ The autoscaling flags map as follows: | `--min-replica` | `--workers-min` (active workers) | Active workers are billed continuously. | | `--max-replica` | `--workers-max` | Baseten defaults to 1, Runpod to 3. | | `--scale-down-delay` (default 900 s) | `--idle-timeout` | A worker is billed while it waits. The console default is 5 seconds. | -| `--concurrency-target` | `--scale-by requests --scale-threshold N` | Or keep queue delay scaling and set per-worker concurrency in the handler. | +| `--concurrency-target` | `--scale-by requests --scale-threshold N` | On a queue endpoint, request count scaling can't be combined with `--idle-timeout`. Or keep queue delay scaling and set per-worker concurrency in the handler. | | `--target-utilization-percentage` | No setting | Lower the request count threshold to leave headroom. | | `--autoscaling-window`, `--max-scale-down-rate` | No setting | | | `--instance-type` | `--gpu-id`, `--gpu-count`, or `--compute-type CPU --instance-id` | | @@ -999,11 +1012,11 @@ import os from openai import OpenAI client = OpenAI( - base_url="https://api.runpod.ai/v2/gpt-oss-120b/openai/v1", + base_url="https://api.runpod.ai/v2/qwen3-32b-awq/openai/v1", api_key=os.environ["RUNPOD_API_KEY"], ) response = client.chat.completions.create( - model="openai/gpt-oss-120b", + model="Qwen/Qwen3-32B-AWQ", messages=[{"role": "user", "content": "Reply with exactly: Paris."}], ) print(response.choices[0].message.content) @@ -1026,7 +1039,7 @@ print(response.choices[0].message.content) ``` -Each Public Endpoint has its own base URL that contains the model's endpoint ID, unlike Baseten's single base URL with a model slug per request. The OpenAI-compatible text models are [GPT-OSS 120B](/public-endpoints/models/gpt-oss-120b) (`gpt-oss-120b`, model `openai/gpt-oss-120b`; the model page's own examples use `gpt-oss-120b`, so try that name if a request is rejected) and [Qwen3 32B AWQ](/public-endpoints/models/qwen3-32b) (`qwen3-32b-awq`, model `Qwen/Qwen3-32B-AWQ`). For any other Model APIs slug, deploy that model on a vLLM endpoint. The vLLM worker serves `/chat/completions`, `/completions`, and `/models`. If you use Baseten's Responses or Anthropic Messages protocols, move those calls to chat completions or serve the protocol from your own server on a load balancing endpoint. For a load balancing endpoint that runs an OpenAI-compatible server, the base URL is `https://ENDPOINT_ID.api.runpod.ai/v1`. +Each Public Endpoint has its own base URL that contains the model's endpoint ID, unlike Baseten's single base URL with a model slug per request. The example uses [Qwen3 32B AWQ](/public-endpoints/models/qwen3-32b) (`qwen3-32b-awq`, model `Qwen/Qwen3-32B-AWQ`), which documents an OpenAI-compatible route. The [model reference](/public-endpoints/reference) lists the text models Runpod hosts, and the catalog is smaller than Baseten's Model APIs, so check it for the model you call. For any other Model APIs slug, deploy that model on a vLLM endpoint. The vLLM worker serves `/chat/completions`, `/completions`, and `/models`. If you use Baseten's Responses or Anthropic Messages protocols, move those calls to chat completions or serve the protocol from your own server on a load balancing endpoint. For a load balancing endpoint that runs an OpenAI-compatible server, the base URL is `https://ENDPOINT_ID.api.runpod.ai/v1`. ### Control-plane automation @@ -1278,7 +1291,7 @@ Both providers read the API key from the environment (`BASETEN_API_KEY` and `RUN | `IN_PROGRESS` | `IN_PROGRESS` | | `SUCCEEDED` | `COMPLETED` | | `FAILED` | `FAILED` | -| `EXPIRED` | `TIMED_OUT` | +| `EXPIRED` | `TIMED_OUT`. Once a job's `ttl` passes, its data is deleted and `/status` stops returning it. | | `CANCELED` | `CANCELLED` | | `WEBHOOK_FAILED` | `COMPLETED`. The result is still available from `/status`. | diff --git a/migrate/coreweave.mdx b/migrate/coreweave.mdx index ac32382da..c086cc862 100644 --- a/migrate/coreweave.mdx +++ b/migrate/coreweave.mdx @@ -14,7 +14,7 @@ You can hand this migration to a coding agent. See [Migrate with a coding agent] | CoreWeave concept | Runpod equivalent | Notes | | --- | --- | --- | -| CKS cluster (managed control plane in its own VPC) | No cluster object | You create Pods, Serverless endpoints, and Instant Clusters directly with the API, CLI, or Terraform. There's no control plane to provision or upgrade. | +| CKS cluster (managed control plane, attached to a VPC) | No cluster object | You create Pods, Serverless endpoints, and Instant Clusters directly with the API, CLI, or Terraform. There's no control plane to provision or upgrade. | | Node Pool (`NodePool`, whole bare-metal nodes by `instanceType`) | GPU type and GPU count on each Pod or worker | CoreWeave GPU instances are whole nodes, typically with 8 GPUs. On Runpod you request GPUs per container. | | Node Pool autoscaling (Cluster Autoscaler) | Serverless worker scaling | Serverless scales workers between `workers.min` and `workers.max`, including to zero. Pods don't autoscale. | | Deployment + Service + HPA, KEDA, or Knative | [Serverless endpoint](/serverless/overview), queue-based or load-balancing | One endpoint replaces the Deployment, the Service, and the autoscaler. | @@ -33,7 +33,7 @@ You can hand this migration to a coding agent. See [Migrate with a coding agent] | Kubernetes Secret | [Runpod secret](/pods/templates/secrets) | Referenced from an environment variable as `{{ RUNPOD_SECRET_name }}`. | | ConfigMap | Environment variables, or files on a volume | Set `env` on the Pod, endpoint, or template. | | Region and Availability Zone (`US-EAST-04A`) | Data center ID (`US-KS-2`) | List data centers with `runpodctl datacenter list`. | -| Reserved, On-Demand, and Spot capacity plans | On-demand, per-second billing | See [Pod pricing](/pods/pricing) and [Serverless pricing](/serverless/pricing). For committed or reserved capacity, [contact sales](https://ecykq.share.hsforms.com/2MZdZATC3Rb62Dgci7knjbA). | +| Flex Reservations, Reserved Instances, Spot Instances, and On-Demand capacity plans | On-demand, per-second billing | See [Pod pricing](/pods/pricing) and [Serverless pricing](/serverless/pricing). For committed or reserved capacity, [contact sales](https://ecykq.share.hsforms.com/2MZdZATC3Rb62Dgci7knjbA). | | CoreWeave Terraform provider (`coreweave/coreweave`) | [Runpod Terraform provider](/integrations/terraform/overview) (`runpod/runpod`) | See [Migrate your Terraform](#migrate-your-terraform). | | `kubectl`, `cwic`, CoreWeave REST APIs | [`runpodctl`](/runpodctl/overview), [REST API v2](/api-reference-v2/overview), the Runpod SDK, the MCP server | The Terraform provider and the SDK are built on REST API v2. | | API access token (`CW-SECRET-...`), kubeconfig, Object Storage access key | One Runpod API key (`RUNPOD_API_KEY`), plus an S3 API key for the S3-compatible API | See [Credentials](/get-started/credentials). | @@ -46,7 +46,7 @@ You can hand this migration to a coding agent. See [Migrate with a coding agent] **You request GPUs, not nodes.** A CoreWeave Node Pool gives you whole bare-metal nodes, and Kubernetes packs your containers onto them. On Runpod, each Pod or Serverless worker asks for a GPU type and a GPU count, and Runpod places it. There's no bin-packing, node affinity, taints, or tolerations to carry over. Replace `nodeSelector` on `node.coreweave.cloud/type` with a GPU type ID or pool. -**Autoscaling is a property of the endpoint.** On CKS you combine an HPA or KEDA with Node Pool autoscaling, and CoreWeave documents node scale-up as taking 5 to 15 minutes. A Serverless endpoint has one scaler: workers between a minimum and maximum, driven by queue delay or request count, with scale to zero as the default. +**Autoscaling is a property of the endpoint.** On CKS you combine an HPA or KEDA with Node Pool autoscaling, and CoreWeave documents node scale-up as taking 5 to 15 minutes, depending on the instance type. A Serverless endpoint has one scaler: workers between a minimum and maximum, driven by queue delay or request count, with scale to zero as the default. **Requests reach your container through Runpod's URLs.** A load-balancing endpoint serves your HTTP server at `https://ENDPOINT_ID.api.runpod.ai/PATH` and requires your Runpod API key as a bearer token. A queue-based endpoint wraps your code in a handler behind `/run` and `/runsync`. A Pod's HTTP port is served at `https://POD_ID-PORT.proxy.runpod.net`, which is public, unauthenticated, and has a 100-second timeout. @@ -63,8 +63,8 @@ You can hand this migration to a coding agent. See [Migrate with a coding agent] On the Runpod side: - **The Kubernetes API.** Operators, CRDs, DaemonSets, admission webhooks, Helm charts, and anything that calls the Kubernetes API from inside a container don't carry over. Rebuild the behavior with the Runpod API, or keep that component on a Kubernetes cluster elsewhere. -- **Multi-container Pods and sidecars.** A Runpod Pod or worker runs one container image, and Pods don't support Docker Compose. Combine the processes into one image with a process supervisor, or run the sidecar as its own Pod and connect over global networking. -- **VPC-level networking.** CoreWeave gives each cluster a VPC with CIDRs you choose, and offers Direct Connect. Runpod's global networking gives Pods private DNS names inside your account at 100 Mbps, on NVIDIA GPU Pods in supported data centers. Pods don't support UDP. +- **Multi-container Pods and sidecars.** A Runpod Pod or worker runs one container image, and Pods don't support Docker Compose. Combine the processes into one image with a process supervisor, or run the sidecar as its own Pod and connect over global networking, which requires both to be NVIDIA GPU Pods. +- **VPC-level networking.** Every CKS cluster attaches to a VPC with CIDRs you can choose, and CoreWeave offers Direct Connect. Runpod's global networking gives Pods private DNS names inside your account at 100 Mbps, on NVIDIA GPU Pods in supported data centers. Pods don't support UDP. - **Object storage features.** AI Object Storage has bucket policies, versioning, lifecycle rules, and LOTA caching. A network volume is a file system with a fixed size and an S3-compatible API on top. If you depend on bucket features, keep an object store and mount or sync from it. - **Whole-node control.** Kernel settings, GPU driver versions per Node Pool, kubelet overrides, and node shell access have no counterpart. Set a CUDA floor with `minCudaVersion` instead of pinning a driver. - **CronJob schedules.** Keep the schedule in a scheduler you already run (CI, Airflow, cron) and have it submit a job to the endpoint. @@ -72,7 +72,7 @@ On the Runpod side: On the CoreWeave side: -- **Scale to zero for inference.** CoreWeave documents that Dedicated Inference doesn't support scale to zero, and on CKS, GPU capacity scales at node granularity through the Cluster Autoscaler. A Serverless endpoint with `workers.min` set to `0` runs no workers while idle. +- **Scale to zero per endpoint.** CoreWeave documents that Dedicated Inference doesn't support scale to zero. On CKS, a Node Pool can scale to zero nodes through the Cluster Autoscaler, but GPU capacity scales at whole-node granularity. A Serverless endpoint with `workers.min` set to `0` runs no workers while idle. - **Per-container GPU requests without a cluster.** There's no cluster, VPC, or Node Pool to create before the first container starts. - **A managed request queue.** Queue-based endpoints give you async submission, status polling, streaming, retries, and webhooks without deploying a queue. @@ -266,7 +266,7 @@ Reference it from any Pod, endpoint, or template by setting an environment varia ### Multi-container Pods -If a Kubernetes Pod has more than one container, decide per container. Init containers that download models become a one-time copy onto a network volume (Step 5). Log and metrics sidecars usually go away, because Runpod collects container logs. A sidecar your app talks to over `localhost` either moves into the same image under a process supervisor, or becomes its own Pod that you reach at `POD_ID.runpod.internal` with [global networking](/pods/networking). +If a Kubernetes Pod has more than one container, decide per container. Init containers that download models become a one-time copy onto a network volume (Step 5). Log and metrics sidecars usually go away, because Runpod collects container logs. A sidecar your app talks to over `localhost` either moves into the same image under a process supervisor, or becomes its own Pod that you reach at `POD_ID.runpod.internal` with [global networking](/pods/networking). Global networking works between NVIDIA GPU Pods only, so a CPU-only sidecar such as a cache belongs in the same image. ## Step 5: Move your data and models @@ -396,7 +396,7 @@ Create the `runpod-s3` Secret from your Runpod S3 API key before you apply the J ### Global volumes -A global volume isn't tied to a data center and is designed for data that many Pods or workers read, such as model weights. The S3-compatible API doesn't work with global volumes, so load one by attaching it to a Pod and copying onto the mount with the Pod method above. See [Global volumes](/storage/globalvolume/overview) for mount paths, request costs, and when to prefer a network volume. +A global volume isn't tied to a data center and is designed for data that many Pods or workers read, such as model weights. The S3-compatible API doesn't work with global volumes, so load one by attaching it to a Pod and running the same sync commands inside it. Attach it to a GPU Pod in the console, or set `mounts.global` when you create the Pod with the REST API. See [Global volumes](/storage/globalvolume/overview) for mount paths, request costs, and when to prefer a network volume. ## Step 6: Deploy on Runpod @@ -952,8 +952,9 @@ curl -s "https://api.runpod.io/v2/serverless/ENDPOINT_ID/metrics/requests?interv **Shut down CoreWeave.** Do this only after you've confirmed the data copy and kept the CoreWeave side idle for a period you're comfortable with. Delete in this order: ```bash -# 1. Workloads, Services, and autoscalers -kubectl delete -f llm.yaml +# 1. Workloads, Services, and autoscalers. Don't run `kubectl delete -f llm.yaml` here: +# that file also declares the PVC, and this would delete it with the workloads. +kubectl delete deployment,service,hpa llama-3-1-8b # 2. Volumes (after verifying the copy on Runpod) kubectl delete pvc model-cache @@ -970,7 +971,7 @@ aws s3 rm s3://BUCKET_NAME --recursive --profile cw aws s3api delete-bucket --bucket BUCKET_NAME --profile cw ``` -Then delete each CKS cluster from the **Clusters** page of the CoreWeave Cloud Console (the cluster must be in the Running, Failed, or Preempted state), and delete API access tokens and Object Storage access keys you no longer need. If you manage CoreWeave with Terraform, run `terraform destroy` on that configuration instead. +If you run SUNK, delete the SUNK cluster as well, because a `SunkCluster` manages its own Node Pools. Then delete each CKS cluster from the **Clusters** page of the CoreWeave Cloud Console (the cluster must be in the Running, Failed, or Preempted state), and delete API access tokens and Object Storage access keys you no longer need. If you manage CoreWeave with Terraform, run `terraform destroy` on that configuration instead. Deleting resources doesn't end a commitment. CoreWeave bills Reserved Instances at a flat rate for the term whether or not instances are running, so talk to your CoreWeave account team about reserved capacity before you plan the cutover date. Versioned buckets also need their object versions and delete markers removed before the bucket can be deleted. diff --git a/migrate/fal.mdx b/migrate/fal.mdx index fd3642353..1048e6cd2 100644 --- a/migrate/fal.mdx +++ b/migrate/fal.mdx @@ -79,7 +79,7 @@ This guide is for teams that run production workloads on [fal](https://fal.ai/do | `shared` auth (callers pay with their own key) | Each account calls its own endpoints | Publish the worker to the [Hub](/hub/overview) so others deploy it in their account. | | `concurrency_buffer`, `concurrency_buffer_perc` | Autoscaling by queue delay or request count | Keep active workers for headroom and tune the scaler threshold. | | 90-day request history and per-request payload APIs | Results expire in minutes | Persist inputs and outputs from your webhook receiver. | -| Automatic retries of failed runner attempts (up to 10) | `/retry/JOB_ID` requeues a failed or timed-out job | Call `/retry` from your client or webhook receiver. | +| Automatic retries of failed runner attempts (up to 10) | Queue-based endpoints retry on failure, but the docs don't state a count. `/retry/JOB_ID` requeues a failed or timed-out job | Call `/retry` from your client or webhook receiver for jobs that still end as `FAILED` or `TIMED_OUT`. | Going the other way, some things exist on Runpod that you may not have used on fal: text models with an OpenAI-compatible route on Public Endpoints, `@app.task` for run-to-completion jobs on a dedicated Pod, per-second Pod billing, and Community Cloud or Secure Cloud placement for Pods. @@ -215,7 +215,7 @@ These fal model IDs have a counterpart in the Runpod [model reference](/public-e A matching name doesn't mean matching output. The two platforms can run different versions, tiers, default steps, and guidance values of the same model family, and fal lists both pro and standard tiers for some Kling endpoints. Run your own prompts through both before you switch traffic, and compare parameters on each Runpod model page. -Runpod also hosts text models (for example `qwen3-32b-awq` and `gpt-oss-120b`) with an OpenAI-compatible route at `https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1`. For fal models that aren't in the table, such as models from the FLUX Pro, Veo, Ideogram, and ElevenLabs families, or other Kling and Sora variants, check the model reference for a current listing, then plan to keep the call on a provider that hosts the model or self-host an open alternative on Serverless. +Runpod also hosts text models (for example `qwen3-32b-awq`) with an OpenAI-compatible route at `https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1`. For fal models that aren't in the table, such as models from the FLUX Pro, Veo, Ideogram, and ElevenLabs families, or other Kling and Sora variants, check the model reference for a current listing, then plan to keep the call on a provider that hosts the model or self-host an open alternative on Serverless. ### GPU map @@ -430,9 +430,13 @@ import torch from diffusers import StableDiffusionXLPipeline from runpod.serverless.utils import rp_upload +# Where a cached model attached to the endpoint appears (see Step 6) +HF_CACHE = "/runpod-volume/huggingface-cache/hub" + # Module level: runs once when the worker starts, like fal's setup() pipe = StableDiffusionXLPipeline.from_pretrained( "stabilityai/stable-diffusion-xl-base-1.0", + cache_dir=HF_CACHE, torch_dtype=torch.float16, variant="fp16", use_safetensors=True, @@ -465,7 +469,7 @@ def handler(job): runpod.serverless.start({"handler": handler}) ``` -The return value keeps fal's `image.url` shape, so callers change `result["image"]["url"]` to `result["output"]["image"]["url"]`. Input validation that pydantic did on fal is now your code. Returning a dictionary with an `error` key marks the job as failed. +The return value keeps fal's `image.url` shape, so callers change `result["image"]["url"]` to `result["output"]["image"]["url"]`. Input validation that pydantic did on fal is now your code. Returning a dictionary with an `error` key marks the job as failed. `rp_upload.upload_image` returns a presigned URL that's valid for 7 days, so generate your own URL or copy the file if callers need a longer-lived link. The Dockerfile installs the handler's dependencies on a CUDA base image and starts the handler. @@ -491,7 +495,7 @@ docker push DOCKER_USERNAME/my-demo-app:v1 For a private registry, save credentials with `runpodctl registry create --name "docker-hub" --username "DOCKER_USERNAME"` and pass the returned ID as `--registry-auth-id` when you create the template. This replaces the `registries` parameter on fal's `ContainerImage`. - This handler downloads the SDXL weights to the container disk on every cold start. Add a cached model in [Step 6](#step-6-deploy-on-runpod), bake the weights into the image, or use a network volume. See [Step 5](#step-5-move-your-data-and-models). + Without a cached model, this handler downloads the SDXL weights on every cold start. Add a cached model in [Step 6](#step-6-deploy-on-runpod), which the handler reads through `cache_dir`, bake the weights into the image, or use a network volume. See [Step 5](#step-5-move-your-data-and-models). ### Existing HTTP servers @@ -654,7 +658,7 @@ runpodctl serverless create --name "my-demo-app" \ --model-reference https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0:main ``` -`--gpu-id` takes a pool ID or a GPU type ID. `--model-reference` attaches a cached model so workers don't download the weights. To attach the network volume from Step 5, add `--network-volume-id NETWORK_VOLUME_ID`. Each Serverless template binds to one endpoint, so create a template per endpoint. Check `runpodctl serverless create --help` for the full flag list. +`--gpu-id` takes a pool ID or a GPU type ID. `--model-reference` attaches a cached model at `/runpod-volume/huggingface-cache/hub/`. A custom handler has to load from that path, which is what `cache_dir` does in the Option B handler. To attach the network volume from Step 5, add `--network-volume-id NETWORK_VOLUME_ID`. Each Serverless template binds to one endpoint, so create a template per endpoint. Check `runpodctl serverless create --help` for the full flag list. The same endpoint through REST API v2, without a separate template: @@ -706,7 +710,7 @@ With `workers.min` at 1, one worker bills continuously. Wait for `GET https://EN ### Pods and Instant Clusters (fal Compute) -A fal Compute instance is an Ubuntu machine you SSH into. A Pod is a container with SSH, so install your stack in the image or on a volume rather than with a VM-level setup, and note that Pods don't run Docker Compose. +A fal Compute instance is a dedicated machine you SSH into. A Pod is a container with SSH, so install your stack in the image or on a volume rather than with a VM-level setup, and note that Pods don't run Docker Compose. ```bash runpodctl pod create --name "train-h100" \ @@ -1058,7 +1062,7 @@ If you call fal through `@ai-sdk/fal`, switch to [`@runpod/ai-sdk-provider`](/pu prompt: "A serene mountain landscape at sunset", aspectRatio: "16:9", providerOptions: { - fal: { guidance_scale: 3.5, num_inference_steps: 28 }, + fal: { guidanceScale: 3.5, numInferenceSteps: 28 }, }, }); ``` diff --git a/migrate/fireworks-ai.mdx b/migrate/fireworks-ai.mdx index c8f5b2bb6..1c97a7100 100644 --- a/migrate/fireworks-ai.mdx +++ b/migrate/fireworks-ai.mdx @@ -73,8 +73,8 @@ Fireworks is a model platform: you pick a model ID and Fireworks runs the infere | --- | --- | --- | | Large shared serverless catalog, including Priority, Fast, and Spot variants | Public Endpoints host a smaller catalog. Service tiers aren't part of it. | Deploy the model on your own vLLM endpoint. Use active workers for consistent latency. | | Embeddings (`/v1/embeddings`) and reranking (`/v1/rerank`) | The vLLM worker docs list `/chat/completions`, `/completions`, and `/models`. | Run an embedding server on a [load balancing endpoint](/serverless/load-balancing/overview) or a custom [queue worker](/serverless/workers/handler-functions). Budget time for this. | -| Responses API (`/v1/responses`) with stored state and server-run MCP tools | Not documented for the vLLM worker or Public Endpoints. | Use chat completions and keep conversation state in your application. | -| Anthropic-compatible Messages API | Not documented on Runpod. | Use the OpenAI-compatible route, or put a translation gateway in front. | +| Responses API (`/v1/responses`) with stored state and server-run MCP tools | Not covered in the Runpod docs. The [worker README](https://github.com/runpod-workers/worker-vllm#usage-openai-compatibility) lists a `/openai/v1/responses` route for the request shape. Stored responses and server-run MCP tools aren't documented. | Test the route with your requests, or use chat completions, and keep conversation state in your application. | +| Anthropic-compatible Messages API | Not covered in the Runpod docs. The worker README lists a `/openai/v1/messages` route. | Test that route with your client, or use the OpenAI-compatible route. | | Deployment shapes (validated hardware and precision per model) | You size the GPU yourself. | Use the [VRAM estimates](/serverless/vllm/configuration#gpu-selection), then test. | | Managed Training API | Runpod doesn't run training jobs for you. | Train on a Pod with your own code. See [Step 5](#step-5-move-your-data-and-models). | | Serverless LoRA inference billed per token | LoRA adapters run on your own endpoint. | Serve adapters with `ENABLE_LORA`. | @@ -168,19 +168,17 @@ Use this table to place each item from your inventory. ### Models that Runpod hosts -Build your model map from the [Public Endpoints model reference](/public-endpoints/reference), which lists image, video, audio (text-to-speech), and text models with their prices. These are the text models with pages in the Runpod docs today. +Build your model map from the [Public Endpoints model reference](/public-endpoints/reference), which lists image, video, audio (text-to-speech), and text models with their prices. These are the text models in the model reference today. Confirm that a model is live in the [Hub](https://console.runpod.io/hub?tabSelected=public_endpoints) before you map a call site to it. | Runpod model | Endpoint slug | `model` value on the OpenAI route | Notes | | --- | --- | --- | --- | | [Qwen3 32B AWQ](/public-endpoints/models/qwen3-32b) | `qwen3-32b-awq` | `Qwen/Qwen3-32B-AWQ` | Documented as OpenAI-compatible. | | [Moonshot Kimi](/public-endpoints/models/moonshot-kimi) | `moonshot-kimi` | `kimi-k2.6` (default), `kimi-k2.7-code`, `kimi-k3`, `kimi-k3.1-pro`, `kimi-k3.1-pro-highspeed` | Documented as OpenAI-compatible. One endpoint serves all five variants. | -| [GPT-OSS 120B](/public-endpoints/models/gpt-oss-120b) | `gpt-oss-120b` | `gpt-oss-120b` on the model page, `openai/gpt-oss-120b` on the [AI coding tools page](/public-endpoints/ai-coding-tools) | Documented as OpenAI-compatible, with tool calling in the [AI SDK guide](/public-endpoints/ai-sdk). Call `/models` to confirm the ID. Not in the model reference table. | -| [Cogito 671B v2.1](/public-endpoints/models/cogito-671b) | `cogito-671b-v2-1-fp8-dynamic` | `cogito-671b-v2-1-fp8-dynamic` | Documented as OpenAI-compatible. Not in the model reference table, so confirm it in the playground. | | [IBM Granite 4.0](/public-endpoints/models/granite-4) | `granite-4-0-h-small` | Check the model | The model page documents the native `/runsync` route with `input.messages`. | The OpenAI-compatible base URL for a Public Endpoint is `https://api.runpod.ai/v2/SLUG/openai/v1`. Every Public Endpoint also accepts the native routes: `https://api.runpod.ai/v2/SLUG/runsync` and `/run`. -For Fireworks model IDs that match these, such as `accounts/fireworks/models/gpt-oss-120b` or `accounts/fireworks/models/kimi-k3`, compare the model card on each side before you assume they're the same checkpoint and precision. For everything else, the general answer is your own vLLM endpoint: Fireworks' serverless catalog is larger than Runpod's Public Endpoints catalog. +For Fireworks model IDs that match these, such as `accounts/fireworks/models/kimi-k3`, compare the model card on each side before you assume they're the same checkpoint and precision. For everything else, the general answer is your own vLLM endpoint: Fireworks' serverless catalog is larger than Runpod's Public Endpoints catalog. ### GPU mapping @@ -216,7 +214,7 @@ The vLLM worker turns any `vllm serve` flag into an environment variable in uppe | Reasoning | `REASONING_PARSER` (for example `deepseek_r1`, `qwen3`) | | Chat template | The model's own template, or `CUSTOM_CHAT_TEMPLATE` | | GPUs per replica (`--accelerator-count`) | GPUs per worker. The worker sets `TENSOR_PARALLEL_SIZE` to the number of visible GPUs. | -| Speculative decoding (`--draft-model`) | `SPECULATIVE_MODEL`, `NUM_SPECULATIVE_TOKENS` | +| Speculative decoding (`--draft-model`) | `SPECULATIVE_CONFIG`, a JSON object such as `{"model": "ORG/DRAFT", "num_speculative_tokens": 3}`. The current worker ignores the older `SPECULATIVE_MODEL` and `NUM_SPECULATIVE_TOKENS` variables. | | LoRA addons (`--enable-addons`) | `ENABLE_LORA=true`, `LORA_MODULES`, `MAX_LORAS`, `MAX_LORA_RANK` | | Gated or private weights | `HF_TOKEN`, from a Runpod secret | | Served model name | `OPENAI_SERVED_MODEL_NAME_OVERRIDE` | @@ -411,7 +409,7 @@ curl -X POST "https://api.runpod.io/v2/serverless" \ "workers": { "min": 0, "max": 3, "idleTimeout": 600 }, "scaling": { "type": "REQUEST_COUNT", "requestCount": 4 }, "flashboot": "FLASHBOOT", - "timeout": 600000 + "timeout": 300000 }' ``` @@ -662,17 +660,17 @@ Each cell reflects what the platform's docs state. "Check the model" means the d | `/chat/completions` | Yes | Yes | Yes, for the text models documented as OpenAI-compatible | | `/completions` | Yes | Yes | Check the model | | `/models` | Check the Fireworks API reference | Yes | Check the model | -| Streaming (`stream: true`) | Yes | Yes | Yes (documented for Qwen3, Kimi, GPT-OSS, Cogito) | +| Streaming (`stream: true`) | Yes | Yes | Yes (documented for Qwen3 and Kimi) | | `usage` in the final stream chunk | Yes, by default | Check the model | Check the model | -| Tool calling | Yes, on supported models | Yes, with `ENABLE_AUTO_TOOL_CHOICE` and `TOOL_CALL_PARSER`. Depends on the model. | Documented for GPT-OSS 120B. Check other models. | -| JSON mode and JSON schema (`response_format`) | Yes (`json_object`, `json_schema`) | Not covered in the Runpod docs. The worker exposes `GUIDED_DECODING_BACKEND`. Check the model. | Check the model | +| Tool calling | Yes, on supported models | Yes, with `ENABLE_AUTO_TOOL_CHOICE` and `TOOL_CALL_PARSER`. Depends on the model. | Not covered on the model pages. Check the model. | +| JSON mode and JSON schema (`response_format`) | Yes (`json_object`, `json_schema`) | Not covered in the Runpod docs. The request goes to vLLM, which supports structured outputs, and `STRUCTURED_OUTPUTS_CONFIG` sets the backend. Check the model. | Check the model | | Grammar mode (BNF) | Yes | Check the model | Check the model | | Reasoning | Yes (`reasoning_effort`, output in `reasoning_content`) | Yes, with `REASONING_PARSER`. Output in `reasoning` on vLLM v0.32.0. | Kimi documents `reasoning_effort`. Check other models. | | Vision input (`image_url`) | Yes, on vision models | Depends on the model you deploy | Kimi K2.6 is described as multimodal. Check the model. | | Embeddings | Yes (`/v1/embeddings`) | Not listed for the vLLM worker | Not listed | | Reranking | Yes (`/v1/rerank`) | Not listed | Not listed | -| Responses API | Yes (`/v1/responses`) | Not listed | Not listed | -| Anthropic Messages API | Yes | Not listed | Not listed | +| Responses API | Yes (`/v1/responses`) | Not in the Runpod docs. The [worker README](https://github.com/runpod-workers/worker-vllm#usage-openai-compatibility) lists `/openai/v1/responses`. Test it. | Not listed | +| Anthropic Messages API | Yes | Not in the Runpod docs. The worker README lists `/openai/v1/messages`. Test it. | Not listed | | LoRA adapters | Yes (deployments, and serverless for select bases) | Yes (`ENABLE_LORA`, `LORA_MODULES`) | Not applicable for text models | | Async jobs | The Responses API can run jobs in the background. Chat completions are synchronous. | Yes (`/run`, `/status`, webhooks) | Yes (`/run`, `/status`) | @@ -808,7 +806,8 @@ On the Runpod side, guard cost with three settings: max workers caps concurrency | --- | --- | | `POST https://api.fireworks.ai/inference/v1/chat/completions` | `POST https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1/chat/completions` | | `POST https://api.fireworks.ai/inference/v1/completions` | `POST https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1/completions` | -| `POST https://api.fireworks.ai/inference/v1/embeddings`, `/rerank`, `/responses` | No prebuilt route. See [What has no direct equivalent](#what-has-no-direct-equivalent). | +| `POST https://api.fireworks.ai/inference/v1/embeddings`, `/rerank` | No prebuilt route. See [What has no direct equivalent](#what-has-no-direct-equivalent). | +| `POST https://api.fireworks.ai/inference/v1/responses` | `POST https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1/responses`, listed in the worker README. Test it. | | `POST https://api.fireworks.ai/v1/accounts/ACCOUNT_ID/deployments` | `POST https://api.runpod.io/v2/serverless` | | List, get, update, and delete deployment | `GET /v2/serverless`, `GET`, `PATCH`, and `DELETE /v2/serverless/ENDPOINT_ID` | | Scale deployment | `PATCH /v2/serverless/ENDPOINT_ID` (`workers.min`, `workers.max`) | @@ -828,7 +827,7 @@ These are client-side variables, since Fireworks doesn't run your containers. | `FIREWORKS_API_KEY` | `RUNPOD_API_KEY` | | `OPENAI_BASE_URL` or `OPENAI_API_BASE` set to `https://api.fireworks.ai/inference/v1` | `https://api.runpod.ai/v2/ENDPOINT_ID/openai/v1` | | `OPENAI_API_KEY` set to a Fireworks key | Set to a Runpod API key | -| `ANTHROPIC_BASE_URL` set to `https://api.fireworks.ai/inference` | No equivalent route | +| `ANTHROPIC_BASE_URL` set to `https://api.fireworks.ai/inference` | No documented equivalent in the Runpod docs. The worker README lists `/openai/v1/messages`, so test a base URL of `https://api.runpod.ai/v2/ENDPOINT_ID/openai` before you rely on it. | | `~/.fireworks/auth.ini` (`firectl` credentials) | `~/.runpod/config.toml` (written by `runpodctl doctor` or `rp login`) | ### Status map diff --git a/migrate/google-cloud.mdx b/migrate/google-cloud.mdx index fa9560f80..5e25387d1 100644 --- a/migrate/google-cloud.mdx +++ b/migrate/google-cloud.mdx @@ -35,7 +35,7 @@ Google's documentation now calls Vertex AI "Gemini Enterprise Agent Platform". T | Artifact Registry | Any registry, pulled with [registry credentials](/runpodctl/reference/runpodctl-registry) | You can keep images in Artifact Registry. | | Persistent Disk, Hyperdisk boot and data disks | Container disk and volume disk | Container disk is ephemeral. Volume disk persists across stops and is deleted with the Pod. | | Filestore, Managed Lustre | [Network volume](/storage/network-volumes), including [high-performance storage](/storage/high-performance-storage) | A network volume lives in one data center. | -| Cloud Storage, Cloud Storage FUSE mounts | Keep Cloud Storage, or use a network volume or [global volume](/storage/globalvolume/overview) | Cloud Storage stays reachable from Runpod with a service account key or HMAC key. | +| Cloud Storage, Cloud Storage FUSE mounts | Keep Cloud Storage, or use a network volume or [global volume](/storage/globalvolume/overview) | Cloud Storage stays reachable from Runpod with a service account key or HMAC key. Global volumes are in beta. | | VPC, firewall rules, load balancers | [Exposed ports](/pods/configuration/expose-ports), [global networking](/pods/networking), Serverless endpoint URLs | See [What has no direct equivalent](#what-has-no-direct-equivalent). | | Secret Manager | [Runpod secrets](/pods/templates/secrets) | Reference a secret in an environment variable as `{{ RUNPOD_SECRET_name }}`. | | Regions and zones | [Data centers](/runpodctl/reference/runpodctl-datacenter) | IDs look like `US-GA-1` or `EU-RO-1`. GPU stock differs by data center. | @@ -76,7 +76,7 @@ Google's documentation now calls Vertex AI "Gemini Enterprise Agent Platform". T **On Runpod:** -- **VPC networks, firewall rules, and Private Service Connect.** Runpod has no customer-defined private network or firewall rules. Pods get [global networking](/pods/networking) (a private network between your own Pods, at `POD_ID.runpod.internal`) and explicitly [exposed ports](/pods/configuration/expose-ports). HTTP ports are served through a public HTTPS proxy URL that isn't authenticated by Runpod, so add authentication in your app. If you need a private path into your Google Cloud VPC, run a VPN or proxy that you control, or keep that workload on Google Cloud. +- **VPC networks, firewall rules, and Private Service Connect.** Runpod has no customer-defined private network or firewall rules. Pods can use [global networking](/pods/networking) (a private network between your own Pods, at `POD_ID.runpod.internal`, which you enable at deploy time on NVIDIA GPU Pods in supported data centers) and explicitly [exposed ports](/pods/configuration/expose-ports). HTTP ports are served through a public HTTPS proxy URL that isn't authenticated by Runpod, so add authentication in your app. If you need a private path into your Google Cloud VPC, run a VPN or proxy that you control, or keep that workload on Google Cloud. - **IAM service accounts, Workload Identity, and the metadata server.** A Pod or worker has no Google identity and no `metadata.google.internal`. To call Google Cloud APIs from Runpod, create a least-privilege service account, store its key as a Runpod secret, and load it at startup. Where a caller only needs one object, hand it a signed URL instead of a key. - **Customer-managed encryption keys (CMEK).** You can encrypt a Pod's volume disk at creation, with a key Runpod manages. Bring-your-own-key isn't supported, and container disks and network volumes can't be encrypted this way. - **Spot pricing through the CLI or REST API v2.** `runpodctl` and REST API v2 create on-demand Pods. Checkpoint to a volume and plan capacity with `runpodctl gpu list` rather than relying on a spot fallback. @@ -523,6 +523,9 @@ Then copy the data with one of these methods. gcloud auth activate-service-account --key-file=/tmp/key.json gcloud storage rsync gs://BUCKET/models /workspace/models --recursive + + # Remove the key file and the credential gcloud cached from it. + gcloud auth revoke --all rm /tmp/key.json ``` @@ -868,7 +871,7 @@ response = client.chat.completions.create( ) ``` -On Runpod, change the base URL, API key, and model name. This example uses the GPT OSS 120B Public Endpoint. +On Runpod, change the base URL, API key, and model name. This example uses the Qwen3 32B AWQ Public Endpoint. ```python import os @@ -876,11 +879,11 @@ import os import openai client = openai.OpenAI( - base_url="https://api.runpod.ai/v2/gpt-oss-120b/openai/v1", + base_url="https://api.runpod.ai/v2/qwen3-32b-awq/openai/v1", api_key=os.environ["RUNPOD_API_KEY"], ) response = client.chat.completions.create( - model="openai/gpt-oss-120b", + model="Qwen/Qwen3-32B-AWQ", messages=[{"role": "user", "content": "Hello"}], ) ``` @@ -1227,6 +1230,7 @@ resource "runpod_serverless_endpoint" "embedder" { name = "embedder" template_id = runpod_template.embedder.id endpoint_type = "LOAD_BALANCER" + ports = ["8080/http"] gpu_pools = ["AMPERE_24"] gpu_count = 1 @@ -1255,7 +1259,7 @@ What changed: `node_selector.accelerator` became `gpu_pools`, `max_instance_requ 3. Remove the Google resource blocks you replaced and apply again. If a resource has `deletion_protection` enabled, set it to `false` and apply before removing the block. 4. Keep the Google resources Runpod depends on: the service account, its key, the Artifact Registry repository, and any buckets. -To bring existing Runpod resources under Terraform, use `terraform import` with the Pod, endpoint, template, or volume ID. See the [provider reference](https://registry.terraform.io/providers/runpod/runpod/latest/docs) for every argument. +To bring existing Runpod resources under Terraform, use `terraform import` with the Pod, endpoint, template, or volume ID. Be careful with Pods: the API doesn't return `template_id`, `start_ssh`, `start_jupyter`, `allowed_cuda_versions`, `min_cuda_version`, `min_ram_per_gpu`, `min_vcpu_count_per_gpu`, or `data_center_ids`, so each one your configuration sets forces a replacement on the first apply after an import, which deletes the Pod and everything that isn't on a network volume. List them under `lifecycle { ignore_changes = [...] }` before you apply. See the [provider reference](https://registry.terraform.io/providers/runpod/runpod/latest/docs) for every argument. ## Reference diff --git a/migrate/hugging-face.mdx b/migrate/hugging-face.mdx index 985a55325..a3533b02c 100644 --- a/migrate/hugging-face.mdx +++ b/migrate/hugging-face.mdx @@ -607,7 +607,7 @@ Create a Serverless template for the image you pushed in Step 4, then create the -With `--template-id`, environment variables come from the template, so set them on `template create`. Each Serverless template binds to one endpoint. In REST v2, `type` and `scaling` are required, `timeout` is in milliseconds, and FlashBoot is off unless you set it. The REST example creates the endpoint without a cached model: attach one with `runpodctl serverless create --model-reference` or the **Model** field in the console, because the handler above loads from the cache. +With `--template-id`, environment variables come from the template, so set them on `template create`. Each Serverless template binds to one endpoint. In REST v2, `type` and `scaling` are required, `timeout` is in milliseconds, and FlashBoot is off unless you set it. The REST example creates the endpoint without a cached model, and the handler above loads from the cache, so its workers fail at startup with `Cached model not found` until you attach the model in the **Model** field in the console. To attach it at creation time, use the `runpodctl` tab. ### Load balancing endpoint for a custom container or TEI diff --git a/migrate/lambda.mdx b/migrate/lambda.mdx index 02bdd45c2..65957c8e0 100644 --- a/migrate/lambda.mdx +++ b/migrate/lambda.mdx @@ -71,7 +71,7 @@ Lambda sunset its hosted Inference API (the shutdown date was September 25, 2025 **Storage has a provisioned size.** A Lambda filesystem has no size to set and bills for the GiB you use. A network volume has a size from 10 GB to 4,096 GB that you choose and can grow later (contact support for more than 4 TB). A [global volume](/storage/globalvolume/overview) (beta) is the usage-billed option on Runpod. -**Billing is prepaid.** Lambda bills in one-minute increments and sends weekly invoices. Runpod bills per second against a credit balance. You need at least one hour of credit for a configuration to deploy it, and Pods stop when your balance reaches \$0. Set up [auto-pay or low balance notifications](/accounts-billing/billing) before you move production. +**Billing is prepaid.** Lambda bills in one-minute increments and sends weekly invoices. Runpod bills per second against a credit balance. You need at least one hour of credit for a configuration to deploy it. When your balance reaches \$0, Pods with a network volume are stopped and Pods without one are terminated, and a terminated Pod's data can't be recovered. Set up [auto-pay or low balance notifications](/accounts-billing/billing) before you move production. **Clusters start on demand, at a different size range.** A 1-Click Cluster is reserved in weeks, covers 16 to 512 H100 or B200 GPUs, and includes three CPU head nodes. An Instant Cluster deploys on demand with 2 to 8 nodes (16 to 64 GPUs) of B200, H200, H100, or A100, and one of the GPU nodes is the primary node. Larger clusters go through Runpod sales. Availability depends on current stock in each data center. @@ -83,7 +83,7 @@ Lambda sunset its hosted Inference API (the shutdown date was September 25, 2025 - **UDP and ICMP.** Pods accept TCP and HTTP connections. Move UDP protocols to TCP, or keep that service elsewhere. - **Source IP allowlists.** Exposed ports are reachable by anyone who has the address. Require authentication in your service, keep internal services on global networking without exposing a port, or reach them through an SSH tunnel. - **ARM instances (GH200).** Runpod images must be `linux/amd64`. Rebuild ARM images for x86-64. -- **Several filesystems on one instance.** `runpodctl pod create` and the console attach one network volume per Pod, plus one global volume if you need a second mount. REST API v2 accepts up to four network volumes in `mounts.network`, all in the Pod's data center. The simplest plan is to merge your filesystems into one network volume with subdirectories. +- **Several filesystems on one instance.** `runpodctl pod create` attaches one network volume per Pod. The console attaches one network volume plus one global volume if you need a second mount. REST API v2 accepts up to four network volumes in `mounts.network`, all in the Pod's data center, plus one global volume in `mounts.global`. The simplest plan is to merge your filesystems into one network volume with subdirectories. - **Managed Kubernetes on a cluster.** Instant Clusters come as Slurm, Ray, PyTorch training, or general application clusters. Port Kubernetes workloads to one of those, or to Pods and Serverless endpoints. - **Fixed vCPU and RAM per instance type.** Pod CPU and memory depend on the host. Use the REST v2 placement floors if your job needs a minimum. @@ -161,7 +161,7 @@ echo "== Environment"; env | cut -d= -f1 | sort EOF ``` -The snapshot is the input for your Dockerfile in Step 4. The environment section prints variable names only, so secrets stay out of the file. If you launched instances with cloud-init, find the `user_data` you passed at launch in your own scripts, because it's write-only in the API. +The snapshot is the input for your Dockerfile in Step 4. The environment section prints variable names only, so secret values stay out of that section. It runs in a non-interactive session, so also check `~/.bashrc` for variables you export there, and review the container commands in the file before you share it. If you launched instances with cloud-init, find the `user_data` you passed at launch in your own scripts, because it's write-only in the API. Finally, find the code and configuration that reference Lambda. @@ -382,6 +382,8 @@ Choose a transfer path based on where the data is. ``` The trailing slash on the source copies the directory's contents into `/workspace`. If the transfer is interrupted, run the same command again and rsync continues where it stopped. Running inside `tmux` keeps the copy alive if your SSH session drops. + + A filesystem isn't the only place data lives. Run the same command for anything you keep on the instance's root volume, for example directories under `/home/ubuntu`, because the root volume is destroyed when you terminate the instance. If your filesystem is in a region with the [Filesystem S3 Adapter](https://docs.lambda.com/public-cloud/s3-adapter-filesystems/) (Lambda lists `us-east-2`, `us-east-3`, and `us-midwest-2`) and was created in March 2025 or later, you can pull the data without a running Lambda instance. Generate an access key pair on the Filesystem S3 Adapter Keys page of the Lambda console and copy the filesystem's bucket name from the Filesystems page. @@ -858,7 +860,7 @@ Lambda routes are relative to `https://cloud.lambda.com/api/v1`. Runpod routes a | `GET /filesystems` | `GET /v2/network-volumes` | `runpodctl network-volume list` | | `POST /filesystems` | `POST /v2/network-volumes` | `runpodctl network-volume create` | | `DELETE /filesystems/{id}` | `DELETE /v2/network-volumes/{id}` | `runpodctl network-volume delete VOLUME_ID` | -| `GET` and `PUT /firewall-rules`, `/firewall-rulesets` | `ports` on `POST /v2/pods` and `PATCH /v2/pods/{id}` | `--ports` on `runpodctl pod create` and `pod update` | +| `GET` and `PUT /firewall-rules`; `GET`, `POST`, `PATCH`, and `DELETE` on `/firewall-rulesets` | `ports` on `POST /v2/pods` and `PATCH /v2/pods/{id}` | `--ports` on `runpodctl pod create` and `pod update` | | `GET /audit-events` | Audit logs in REST v2 | | | No equivalent | `POST /v2/pods/{id}/exec` (run a command without SSH) | | diff --git a/migrate/modal.mdx b/migrate/modal.mdx index f3acda3da..98fec0fef 100644 --- a/migrate/modal.mdx +++ b/migrate/modal.mdx @@ -859,7 +859,7 @@ docker push YOUR_REGISTRY/textgen:v1 runpodctl template create --name textgen-v1 --image YOUR_REGISTRY/textgen:v1 --serverless \ --container-disk-in-gb 40 \ - --env '{"HF_TOKEN":"{{ RUNPOD_SECRET_hf-token }}"}' + --env '{"HF_TOKEN":"{{ RUNPOD_SECRET_hf-token }}","HF_HUB_CACHE":"/runpod-volume/huggingface-cache/hub"}' runpodctl serverless create --name textgen --template-id TEMPLATE_ID \ --gpu-id AMPERE_24 --workers-min 1 --workers-max 4 \ @@ -867,7 +867,7 @@ runpodctl serverless create --name textgen --template-id TEMPLATE_ID \ --model-reference https://huggingface.co/Qwen/Qwen2.5-7B-Instruct:main ``` -`--gpu-id` accepts a pool ID or a device ID, and `--execution-timeout` is in seconds here. Add `--network-volume-id VOLUME_ID` to mount a network volume at `/runpod-volume`. For a private registry, save credentials with `runpodctl registry create` and pass `--registry-auth-id` when you create the template. In the console, the same flow is **Serverless**, then **New Endpoint**, then **Import from Docker Registry**. +`--gpu-id` accepts a pool ID or a device ID, and `--execution-timeout` is in seconds here. `--model-reference` attaches a cached model under `/runpod-volume/huggingface-cache/hub/`. A custom handler doesn't look there on its own, so the template sets `HF_HUB_CACHE` to that path. Without it, the handler downloads the weights again on every cold start. Add `--network-volume-id VOLUME_ID` to mount a network volume at `/runpod-volume`. For a private registry, save credentials with `runpodctl registry create` and pass `--registry-auth-id` when you create the template. In the console, the same flow is **Serverless**, then **New Endpoint**, then **Import from Docker Registry**. ### Load-balancing endpoint from a custom image @@ -951,8 +951,8 @@ A request to a Modal Web Function looks like this: ```bash curl -X POST https://WORKSPACE--classify-web.modal.run/classify \ - -H "Modal-Key: YOUR_MODAL_TOKEN_ID" \ - -H "Modal-Secret: YOUR_MODAL_TOKEN_SECRET" \ + -H "Modal-Key: YOUR_PROXY_TOKEN_ID" \ + -H "Modal-Secret: YOUR_PROXY_TOKEN_SECRET" \ -H "Content-Type: application/json" \ -d '{"text": "This works."}' ``` diff --git a/migrate/paperspace.mdx b/migrate/paperspace.mdx index 6ea4a7fb1..9a825fc0d 100644 --- a/migrate/paperspace.mdx +++ b/migrate/paperspace.mdx @@ -36,7 +36,7 @@ You can hand most of this migration to a coding agent. See [Migrate with a codin | DigitalOcean Volumes (block storage) | [Network volume](/storage/network-volumes) | A file system mount, not a raw block device. | | Paperspace datasets (versioned), DigitalOcean Spaces | Network volume with the [S3-compatible API](/storage/s3-api), or a [global volume](/storage/globalvolume/overview) | Runpod has no dataset versioning or bucket service. You can also keep reading from Spaces. | | Public IP (static or dynamic), Droplet public IPv4 | HTTP proxy URL or a mapped TCP port | Runpod assigns the public IP and external port. See [Expose ports](/pods/configuration/expose-ports). | -| Paperspace private network, DigitalOcean VPC | [Global networking](/pods/networking) | Private Pod-to-Pod DNS at `POD_ID.runpod.internal`. | +| Paperspace private network, DigitalOcean VPC | [Global networking](/pods/networking) | Private Pod-to-Pod DNS at `POD_ID.runpod.internal`. NVIDIA GPU Pods in supported data centers only. | | Paperspace secrets (`secret:NAME`) | [Secrets](/pods/templates/secrets) | Referenced as `{{ RUNPOD_SECRET_name }}` in environment variables. | | Paperspace regions (NY2, CA1, AMS1), DigitalOcean regions | Data centers | List them with `runpodctl datacenter list`. | | Paperspace hourly billing | Per-second billing | Pods bill per second for compute and storage. Serverless bills per second while workers run. | @@ -86,7 +86,7 @@ You can hand most of this migration to a coding agent. See [Migrate with a codin - **Raw block devices.** Network volumes mount as a file system. You can't partition or format them. - **Multiple probe types.** Fold readiness and startup logic into the single health path. -**On Paperspace and DigitalOcean:** +**On the Paperspace and DigitalOcean GPU products this guide covers:** - **Scale to zero for your own containers.** Paperspace `replicas` is the floor for a running Deployment, and you stop it by setting `enabled: false`. - **A managed request queue.** Queue-based endpoints give you async jobs, status polling, streaming, retries, and webhooks without running a queue yourself. @@ -171,7 +171,7 @@ Search your repositories for CLI invocations, hostnames, credentials, and Terraf ```bash grep -rnE \ - "pspace |gradient (deployments|workflows|notebooks|datasets)|api\.paperspace\.com|paperspacegradient\.com|PAPERSPACE_API_KEY|\.gradient/workflows|doctl |api\.digitalocean\.com|digitaloceanspaces\.com|inference\.do-ai\.run|dedicated-inference\.do-infra\.ai|DIGITALOCEAN_(ACCESS_)?TOKEN|MODEL_ACCESS_KEY|from pydo|digitalocean/godo|digitalocean_(droplet|volume|volume_attachment|spaces_bucket|kubernetes_cluster|kubernetes_node_pool)" \ + "pspace |gradient (deployments|workflows|notebooks|datasets)|api\.paperspace\.com|paperspacegradient\.com|PAPERSPACE_API_KEY|\.gradient/workflows|doctl |api\.digitalocean\.com|digitaloceanspaces\.com|inference\.do-ai\.run|dedicated-inference\.do-infra\.ai|DIGITALOCEAN_(ACCESS_)?TOKEN|MODEL_ACCESS_KEY|from pydo|digitalocean/godo|digitalocean_(droplet|volume|volume_attachment|spaces_bucket|kubernetes_cluster|kubernetes_node_pool)|paperspace_(machine|script|network|autoscaling_group)" \ --include="*.py" --include="*.ts" --include="*.js" --include="*.go" --include="*.sh" \ --include="*.yaml" --include="*.yml" --include="*.json" --include="*.toml" --include="*.tf" \ --include="Dockerfile*" --include="Makefile" . @@ -296,7 +296,7 @@ Translate the rest of the VM setup as follows: - **GPU drivers and CUDA.** Don't install NVIDIA drivers in the image. The host provides them. Your image supplies the CUDA user-space libraries, and you set a CUDA floor when you create the Pod (`--min-cuda-version 12.8`) so it lands on a host whose driver supports your image. - **Startup scripts and user data.** A Paperspace startup script or Droplet `user_data` (cloud-init) becomes either Dockerfile steps (for installs) or the container start command (for launching your process). -- **Docker and Docker Compose on the VM.** If the Droplet ran `docker run --gpus all your-image`, that image is your Pod image, and the `docker run` flags become template fields: `-p` becomes `ports`, `-e` becomes `env`, `-v` becomes a volume mount, and the trailing command becomes the start command. For a Compose stack, run one Pod per service and connect them with [global networking](/pods/networking), or combine the services into one image with a process supervisor. +- **Docker and Docker Compose on the VM.** If the Droplet ran `docker run --gpus all your-image`, that image is your Pod image, and the `docker run` flags become template fields: `-p` becomes `ports`, `-e` becomes `env`, `-v` becomes a volume mount, and the trailing command becomes the start command. For a Compose stack, run one Pod per service and connect them with [global networking](/pods/networking) (NVIDIA GPU Pods only), or combine the services into one image with a process supervisor. - **`systemd` services.** A container has one main process. Run your service in the foreground as the start command. - **The `paperspace` user.** Paperspace Linux machines provision a `paperspace` user with a home directory at `/home/paperspace`. Runpod's SSH examples connect as `root`, and persistent data lives in `/workspace`. Update hardcoded paths. - **ML-in-a-Box and the AI/ML-ready image.** Both are VM images with a preinstalled stack. The closest starting point is a `runpod/pytorch` image plus your own `requirements.txt`. @@ -610,7 +610,7 @@ Replace `nyc3` with your bucket's region. Running this inside a Pod writes strai ### Models -For Hugging Face models on Serverless, attach the model to the endpoint instead of copying weights. Runpod stages it on the host before the worker starts, and it appears under `/runpod-volume/huggingface-cache/hub/`. This replaces a Paperspace Deployment `git-lfs` integration that clones a Hugging Face URL into `/models`. +For Hugging Face models on Serverless, attach the model to the endpoint instead of copying weights: add `--model-reference` to the `runpodctl serverless create` command you run in [Step 6](#step-6-deploy-on-runpod), as in the example below. Runpod stages it on the host before the worker starts, and it appears under `/runpod-volume/huggingface-cache/hub/`. This replaces a Paperspace Deployment `git-lfs` integration that clones a Hugging Face URL into `/models`. ```bash runpodctl serverless create --template-id TEMPLATE_ID --gpu-id AMPERE_80 \ @@ -760,13 +760,13 @@ curl -s -X POST https://api.runpod.io/v2/serverless \ }, "workers": {"min": 0, "max": 3, "idleTimeout": 30}, "scaling": {"type": "REQUEST_COUNT", "requestCount": 4}, - "timeout": 600000 + "timeout": 300000 }' ``` -The response includes `requestUrls.base` (`https://ENDPOINT_ID.api.runpod.ai`) and `requestUrls.health`. Set both `PORT` and `PORT_HEALTH` to your server's port and list the same port in `ports`: each defaults to 80 on its own, and a mismatch leaves the worker unhealthy. `{{ RUNPOD_SECRET_hf_token }}` reads a Runpod secret named `hf_token`, which you create in the console or with `POST /v2/account/secrets`. Load-balancing endpoints scale on `REQUEST_COUNT` only. `workers.min` of 0 scales to zero. Set it to 1 or more to keep warm workers, which is the closest match to Paperspace `replicas`. In the console, the same flow is **Serverless**, then **New Endpoint**. +The response includes `requestUrls.base` (`https://ENDPOINT_ID.api.runpod.ai`) and `requestUrls.health`. Set both `PORT` and `PORT_HEALTH` to your server's port and list the same port in `ports`: each defaults to 80 on its own, and a mismatch leaves the worker unhealthy. `{{ RUNPOD_SECRET_hf_token }}` reads a Runpod secret named `hf_token`, which you create in the console or with `POST /v2/account/secrets`. This request sets `HEALTH_CHECK_PATH` to `/ping`, so it needs the image rebuilt with the `/ping` route from [Step 4](#step-4-move-your-images-and-code). An unchanged Paperspace image has no `/ping` route, and its workers never become healthy. Load-balancing endpoints scale on `REQUEST_COUNT` only. `workers.min` of 0 scales to zero. Set it to 1 or more to keep warm workers, which is the closest match to Paperspace `replicas`. In the console, the same flow is **Serverless**, then **New Endpoint**. ### Deploy a queue-based endpoint (from a Deployment or a fan-out Workflow) @@ -977,15 +977,15 @@ A Paperspace auto-shutdown timer has no setting to copy across. Pods come with ` ```bash pspace deployment delete DEPLOYMENT_ID + pspace snapshot delete SNAPSHOT_ID pspace machine stop MACHINE_ID pspace machine delete MACHINE_ID pspace shared-drive delete SHARED_DRIVE_ID - pspace snapshot delete SNAPSHOT_ID pspace template delete TEMPLATE_ID pspace public-ip release PUBLIC_IP ``` - Deleting a machine (Paperspace calls this deactivating) permanently removes the machine, its files, and its snapshots, and the machine must be off first. Delete Notebooks, Workflows, and datasets in the Paperspace console, and review your Gradient subscription plan there. Run `pspace public-ip release --help` and `pspace snapshot delete --help` for the exact arguments. + Deleting a machine (Paperspace calls this deactivating) permanently removes the machine, its files, and its snapshots, and the machine must be off first, so wait for `pspace machine get MACHINE_ID` to show it as off before you delete it. Delete Notebooks, Workflows, and datasets in the Paperspace console, and review your Gradient subscription plan there. Run `pspace public-ip release --help` and `pspace snapshot delete --help` for the exact arguments. A powered-off GPU Droplet still bills. Destroy it, and remove the volumes and snapshots that the destroy leaves behind. @@ -1003,13 +1003,13 @@ A Paperspace auto-shutdown timer has no setting to copy across. Pods come with ` Droplet backups stay until their retention period ends. Delete Spaces buckets and model access keys in the Control Panel once nothing reads them. Revoke the Paperspace API keys and DigitalOcean tokens you created for the migration. - Delete transfer Pods and stop development Pods you aren't using (`runpodctl pod list --all`). Keep endpoint minimum workers at 0 unless you need warm capacity, and set an idle timeout that fits your traffic. Review spend with `runpodctl billing pods` and `runpodctl billing serverless`, and turn on [low balance notifications](/accounts-billing/billing#low-balance-notifications): Pods stop when your balance reaches zero. + Delete transfer Pods and stop development Pods you aren't using (`runpodctl pod list --all`). Network volumes bill until you delete them, including ones the SDK creates for you such as `pipeline-artifacts` (`runpodctl network-volume list`). Keep endpoint minimum workers at 0 unless you need warm capacity, and set an idle timeout that fits your traffic. Review spend with `runpodctl billing pods` and `runpodctl billing serverless`, and turn on [low balance notifications](/accounts-billing/billing#low-balance-notifications). When your balance reaches zero, Pods with a network volume are stopped, and Pods without one are terminated and their volume disk data can't be recovered. ## Migrate your Terraform -DigitalOcean resources are managed with the `digitalocean/digitalocean` provider. The Runpod provider (`runpod/runpod`, version 2.x) is built on REST API v2 and reads `RUNPOD_API_KEY` from the environment. Paperspace resources aren't covered by the `digitalocean` provider, so recreate them from the inventory in Step 1. +DigitalOcean resources are managed with the `digitalocean/digitalocean` provider. The Runpod provider (`runpod/runpod`, version 2.x) is built on REST API v2 and reads `RUNPOD_API_KEY` from the environment. Paperspace resources aren't covered by the `digitalocean` provider, so recreate them from the inventory in Step 1. If your configuration uses the community `Paperspace/paperspace` provider, map `paperspace_machine` to `runpod_pod` the same way as `digitalocean_droplet`, and move `paperspace_script` contents into the image or the start command. | `digitalocean` resource | Runpod resource | Notes | | --- | --- | --- | @@ -1232,7 +1232,7 @@ All three APIs authenticate with `Authorization: Bearer`. Requests to deployed e | Restarting | `STARTING` | | | Shutting Down, Off | `EXITED` (`runtimeStatus: stopped`) | Volume disk is kept and billed. | | Inaccessible | `ERROR` | Check `runpodctl pod logs POD_ID`. | -| Deactivated | `TERMINATED` | | +| Deactivated (machine deleted) | `TERMINATED` | Not a listed machine state. The machine no longer exists. | Queue-based jobs report `IN_QUEUE`, `IN_PROGRESS`, `COMPLETED`, `FAILED`, `CANCELLED`, or `TIMED_OUT`. See [Job states](/serverless/endpoints/job-states). diff --git a/migrate/replicate.mdx b/migrate/replicate.mdx index e7fcbe235..079a6e172 100644 --- a/migrate/replicate.mdx +++ b/migrate/replicate.mdx @@ -28,7 +28,6 @@ This guide is for teams that call models on [Replicate](https://replicate.com/do | Hardware SKU (`gpu-t4`, `gpu-l40s`, `gpu-a100-large`, `gpu-h100`) | GPU pool (`AMPERE_16`, `ADA_48_PRO`, `AMPERE_80`, `ADA_80_PRO`) | See the [GPU mapping](#gpu-mapping). | | Cold boot | Cold start, reduced by FlashBoot, [cached models](/serverless/endpoints/model-caching), and active workers | | | Secret inputs (`cog.Secret`) | Endpoint environment variables that reference [Runpod secrets](/pods/templates/secrets) | Secrets are set on the endpoint, not sent per request. | -| Regions | [Data centers](/serverless/endpoints/endpoint-configurations#data-centers) you can allow or restrict per endpoint | Network volumes pin an endpoint to one data center. | | Billing: run time on shared hardware, or all online time for private models and deployments | Per-second billing while workers run (cold start, execution, idle timeout) | See [Serverless pricing](/serverless/pricing). Public Endpoints bill per output. | | HTTP API (`https://api.replicate.com/v1`) | Job API (`https://api.runpod.ai/v2/ENDPOINT_ID`) and [REST API v2](/api-reference-v2/overview) (`https://api.runpod.io/v2`) for managing resources | Two hosts: one to run jobs, one to manage infrastructure. | | Client libraries (`replicate` for Python and Node.js, Go, Swift) | `runpod` (Python), `runpod-sdk` (JavaScript), Go SDK, or plain HTTP | See the [operation reference](/serverless/endpoints/operation-reference). | @@ -76,7 +75,6 @@ This guide is for teams that call models on [Replicate](https://replicate.com/do | On Runpod | On Replicate | Notes | | --- | --- | --- | -| Scale to zero for private endpoints, with per-second billing only while workers run | Private models and deployments bill for setup, idle, and active time (fast-booting fine-tunes are the exception) | Set active workers to 0 on Runpod. | | Any container image, any base, any server | Cog is the packaging format | You can still run a Cog image on Runpod. | | [Load balancing endpoints](/serverless/load-balancing/overview) with your own HTTP routes | Models are called through the predictions API | Useful for custom REST APIs and WebSockets. | | Jobs that run up to 7 days | 30 minute default prediction timeout | Set `policy.executionTimeout` and `policy.ttl`. | @@ -93,7 +91,7 @@ This guide is for teams that call models on [Replicate](https://replicate.com/do ## Step 1: Take inventory of what you run on Replicate -List what you have on Replicate before you create anything on Runpod. Replicate's HTTP API covers everything you need, and the commands below use `curl` and `jq`. +List what you have on Replicate before you create anything on Runpod. Replicate's HTTP API covers most of what you need, and the commands below use `curl` and `jq`. Confirm which account or organization the token belongs to: @@ -145,7 +143,7 @@ The `Input` schema lists every input with its type, default, and constraints. Ke Then find every place your code touches Replicate: ```bash -grep -rnE "import replicate|from replicate|require\(.replicate.\)|from .replicate.|api\.replicate\.com|REPLICATE_API_TOKEN|replicate\.delivery|r8\.im/|from cog import|cog (push|build|predict|login)" \ +grep -rnE "import replicate|from replicate|require\(.replicate.\)|from .replicate.|api\.replicate\.com|REPLICATE_API_TOKEN|replicate\.delivery|r8\.im/|from cog import|cog (push|build|predict|run|login)" \ --exclude-dir=node_modules --exclude-dir=.git --exclude-dir=.venv . ``` @@ -340,7 +338,7 @@ class Predictor(BasePredictor): ``` - Current Cog documentation uses `run: "run.py:Runner"`, `BaseRunner`, and a `run()` method. `BasePredictor` and `predict()` are deprecated there but still work, and Replicate's own guides still show them. The port below is the same for both: `setup()` becomes module-level code, and `predict()` or `run()` becomes the handler. + Current Cog documentation uses `run: "run.py:Runner"`, `BaseRunner`, and a `run()` method. `BasePredictor`, `predict()`, and `cog predict` are deprecated there (`cog run -i` replaces `cog predict -i`) but still work, and Replicate's own guides still show them. The port below is the same for both: `setup()` becomes module-level code, and `predict()` or `run()` becomes the handler. Here's the same model as a Runpod worker. The handler validates input, generates the image, and returns a URL if a bucket is configured or base64 if not: @@ -588,7 +586,7 @@ Cog models carry their weights in the image or fetch them in `setup()`, so there **Weights in your Cog image.** If your weights are in the model repository or downloaded during the Cog build, they're already in your source. Bake them into the worker image as shown in Step 4, or put them on a network volume. -**Weights from Hugging Face.** Use [cached models](/serverless/endpoints/model-caching). Runpod schedules workers on hosts that already have the model and makes it available under `/runpod-volume/huggingface-cache/hub/`. Set it in the console's **Model** field, with `--model-reference` in runpodctl, or with `model=runpod.Model(...)` in the SDK. +**Weights from Hugging Face.** Use [cached models](/serverless/endpoints/model-caching). Runpod tries to start workers on hosts that already have the model and makes it available under `/runpod-volume/huggingface-cache/hub/`. Set it in the console's **Model** field, with `--model-reference` in runpodctl, or with `model=runpod.Model(...)` in the SDK. **Fine-tuned weights created on Replicate.** A Replicate training adds a new version to its destination model. Replicate's fine-tuning guide describes a **Download weights** option for trained models, and the training object (`GET /v1/trainings/TRAINING_ID`) has an `output` field. Download the weights while you still have access, then put them where your Runpod worker can read them. @@ -686,7 +684,7 @@ curl -s -X POST https://api.runpod.io/v2/serverless \ `type` and `scaling` are required. REST v2 defaults differ from the console: `flashboot` defaults to `OFF`, `timeout` to 300000 ms, and `workers.idleTimeout` to 10 seconds, so set them explicitly. The response includes the endpoint ID you use in request URLs. -In the console, go to [Serverless](https://www.console.runpod.io/serverless), click **New Endpoint**, and import from a Docker registry or a [GitHub repository](/serverless/workers/github-integration). The GitHub integration builds the image for you on each push, which replaces `cog push` in CI. +In the console, go to [Serverless](https://www.console.runpod.io/serverless), click **New Endpoint**, and import from a Docker registry or a [GitHub repository](/serverless/workers/github-integration). The GitHub integration builds the image for you each time you create a GitHub release, which replaces `cog push` in CI. ### Scaling and cold starts @@ -780,7 +778,7 @@ Both functions mount the same volume, so `generate` reads what `train` wrote, an output = replicate.run( "black-forest-labs/flux-schnell", - input={"prompt": "an iguana on the beach, pointillism"}, + input={"prompt": "an iguana on the beach, pointillism", "output_format": "png"}, ) with open("output.png", "wb") as f: @@ -797,7 +795,7 @@ Both functions mount the same volume, so `generate` reads what `train` wrote, an response = requests.post( "https://api.runpod.ai/v2/black-forest-labs-flux-1-schnell/runsync", headers={"Authorization": f"Bearer {os.environ['RUNPOD_API_KEY']}"}, - json={"input": {"prompt": "an iguana on the beach, pointillism", "width": 1024, "height": 1024}}, + json={"input": {"prompt": "an iguana on the beach, pointillism", "width": 1024, "height": 1024, "image_format": "png"}}, timeout=120, ) response.raise_for_status() @@ -839,7 +837,7 @@ Both functions mount the same volume, so `generate` reads what `train` wrote, an const replicate = new Replicate(); const [output] = await replicate.run("black-forest-labs/flux-schnell", { - input: { prompt: "an iguana on the beach, pointillism" }, + input: { prompt: "an iguana on the beach, pointillism", output_format: "png" }, }); await writeFile("output.png", output); @@ -859,7 +857,7 @@ Both functions mount the same volume, so `generate` reads what `train` wrote, an "Content-Type": "application/json", }, body: JSON.stringify({ - input: { prompt: "an iguana on the beach, pointillism", width: 1024, height: 1024 }, + input: { prompt: "an iguana on the beach, pointillism", width: 1024, height: 1024, image_format: "png" }, }), } ); @@ -895,7 +893,7 @@ Both functions mount the same volume, so `generate` reads what `train` wrote, an -H "Authorization: Bearer $REPLICATE_API_TOKEN" \ -H "Content-Type: application/json" \ -H "Prefer: wait" \ - -d '{"input": {"prompt": "an iguana on the beach, pointillism"}}' + -d '{"input": {"prompt": "an iguana on the beach, pointillism", "output_format": "png"}}' ``` After: @@ -904,7 +902,7 @@ Both functions mount the same volume, so `generate` reads what `train` wrote, an curl -s -X POST https://api.runpod.ai/v2/black-forest-labs-flux-1-schnell/runsync \ -H "Authorization: Bearer $RUNPOD_API_KEY" \ -H "Content-Type: application/json" \ - -d '{"input": {"prompt": "an iguana on the beach, pointillism", "width": 1024, "height": 1024}}' + -d '{"input": {"prompt": "an iguana on the beach, pointillism", "width": 1024, "height": 1024, "image_format": "png"}}' ``` @@ -1144,7 +1142,7 @@ Scripts that manage Replicate resources map to runpodctl or REST v2. See the [AP | Cog or Replicate | Runpod | | --- | --- | | `cog init` | `rp flash init my-app` (SDK app), or write `handler.py` and a `Dockerfile` | -| `cog predict -i key=value` | `python handler.py --test_input '{"input": {"key": "value"}}'`, or `rp flash dev main.py` | +| `cog predict -i key=value` (`cog run -i key=value` in current Cog) | `python handler.py --test_input '{"input": {"key": "value"}}'`, or `rp flash dev main.py` | | `cog build -t IMAGE` | `docker build --platform linux/amd64 -t IMAGE .` | | `cog login` | `docker login` for your registry, and `rp login` or `runpodctl doctor` for Runpod | | `cog push r8.im/OWNER/MODEL` | `docker push IMAGE`, then `runpodctl template update TEMPLATE_ID --image IMAGE`. Or `rp flash deploy`. | diff --git a/migrate/together-ai.mdx b/migrate/together-ai.mdx index 7677c9205..b21cadb8b 100644 --- a/migrate/together-ai.mdx +++ b/migrate/together-ai.mdx @@ -71,7 +71,7 @@ You can hand most of this migration to a coding agent. See [Migrate with a codin | Certified deployment profiles (model, quantization, hardware, and optimizations chosen for you) | No equivalent. Use the [vLLM GPU sizing guidance](/serverless/vllm/configuration#gpu-selection) and test. | | Traffic splitting, A/B tests, shadow experiments, and metric-gated rollouts on one endpoint | No built-in equivalent. Run two endpoints and split traffic in your own gateway or client. Changing an endpoint's image or environment rolls workers to a new release. | | `/v1/embeddings` and `/v1/rerank` on dedicated endpoints | The vLLM worker documents `/chat/completions`, `/completions`, and `/models`. For embeddings or reranking, run an embedding server on a [load balancing endpoint](/serverless/load-balancing/overview), or look for an embedding worker in the [Hub](/hub/overview). | -| OpenAI Responses API and Anthropic Messages API | Not documented for the vLLM worker. Use chat completions. | +| OpenAI Responses API and Anthropic Messages API | The Runpod docs cover chat completions, completions, and models. The [worker README](https://github.com/runpod-workers/worker-vllm#usage-openai-compatibility) also lists `/openai/v1/responses` and `/openai/v1/messages`, so test those routes with your client, or use chat completions. | | Managed fine-tuning jobs with a dataset upload and a job ID | No managed job API. Run training on a Pod. See [Fine-tunes](#fine-tunes). | | Automatic prompt caching with a discounted cached-token price | No token pricing on your own endpoint. vLLM prefix caching is a setting (`ENABLE_PREFIX_CACHING`) that affects speed, not billing. | | Geo-specific inference through `us-api.together.ai` | Set data centers on your own endpoint. | @@ -212,7 +212,6 @@ These are the Together serverless model strings that have a Public Endpoint in t | --- | --- | --- | | `moonshotai/Kimi-K3` | [Moonshot Kimi](/public-endpoints/models/moonshot-kimi), `model` set to `kimi-k3` | OpenAI-compatible: `https://api.runpod.ai/v2/moonshot-kimi/openai/v1` | | `moonshotai/Kimi-K3.1-Pro` | [Moonshot Kimi](/public-endpoints/models/moonshot-kimi), `model` set to `kimi-k3.1-pro` | Same base URL as above | -| `openai/gpt-oss-120b` | [GPT-OSS 120B](/public-endpoints/models/gpt-oss-120b). The model page sets `model` to `gpt-oss-120b`, and the [AI coding tools page](/public-endpoints/ai-coding-tools) uses `openai/gpt-oss-120b`, so call `/models` to confirm. | OpenAI-compatible: `https://api.runpod.ai/v2/gpt-oss-120b/openai/v1` | | Qwen chat models (`Qwen/Qwen3...`) | [Qwen3 32B AWQ](/public-endpoints/models/qwen3-32b), a different Qwen3 model | OpenAI-compatible: `https://api.runpod.ai/v2/qwen3-32b-awq/openai/v1`, `model` set to `Qwen/Qwen3-32B-AWQ` | | `ByteDance-Seed/Seedream-4.0` | [Seedream 4.0 T2I](/public-endpoints/models/seedream-4-t2i) and [Seedream 4.0 Edit](/public-endpoints/models/seedream-4-edit) | Native: `https://api.runpod.ai/v2/seedream-v4-t2i/runsync` | | `Qwen/Qwen-Image` | [Qwen Image](/public-endpoints/models/qwen-image) | Native: `https://api.runpod.ai/v2/qwen-image-t2i/runsync` | @@ -295,10 +294,11 @@ def handler(job): runpod.serverless.start({"handler": handler}) ``` -Anything you loaded in `setup()` moves to module level so it loads once per worker. The return value becomes the job's `output`. If `predict()` raises an exception, Sprocket marks the job failed, and an exception in a Runpod handler does the same. Three Sprocket behaviors need explicit code on Runpod: +Anything you loaded in `setup()` moves to module level so it loads once per worker. The return value becomes the job's `output`. If `predict()` raises an exception, Sprocket marks the job failed, and an exception in a Runpod handler does the same. Four Sprocket behaviors need explicit code on Runpod: - **Progress updates.** Replace `emit_info(...)` with `runpod.serverless.progress_update(job, ...)`. - **Input files.** Sprocket downloads any HTTPS URL in the payload and hands your code a local path. A Runpod handler receives the URL as sent, so download it in the handler. +- **Output files.** Sprocket uploads a returned `FileOutput(...)` to Together storage and puts its URL in the result. A Runpod handler uploads the file to your own storage and returns the URL. - **HTTP server mode.** A Sprocket worker that serves `predict_path` directly becomes a [load balancing endpoint](/serverless/load-balancing/build-a-worker): your own HTTP server with a `/ping` health check. ### Container image @@ -446,7 +446,7 @@ This is the landing spot for most Together serverless models and all dedicated e --env TOOL_CALL_PARSER=hermes ``` - `--hub-id` pulls the worker image, container disk size, and default environment from the Hub release, and `--env` overrides individual values. `--gpu-id` accepts a pool ID or a GPU type ID; set it explicitly so the endpoint runs on the hardware you sized for. `--model-reference` attaches the Hugging Face model as a cached model: the `:main` suffix is the branch, tag, or revision, and the flag requires a GPU endpoint. The command prints the endpoint ID. For a gated or private model, add `--env HF_TOKEN=YOUR_HF_TOKEN`, or store the token as a [secret](/pods/templates/secrets). + `--hub-id` pulls the worker image, container disk size, and default environment from the Hub release, and `--env` overrides individual values. `--gpu-id` accepts a pool ID or a GPU type ID; set it explicitly so the endpoint runs on the hardware you sized for. `--model-reference` attaches the Hugging Face model as a cached model: the `:main` suffix is the branch, tag, or revision, and the flag requires a GPU endpoint. The command prints the endpoint ID. For a gated or private model, store the token as a [secret](/pods/templates/secrets) named `hf_token` and add `--env HF_TOKEN='{{ RUNPOD_SECRET_hf_token }}'`, so the token itself never appears in the command or the endpoint configuration. Open the [vLLM worker](https://console.runpod.io/hub/runpod-workers/worker-vllm) in the Hub and click **Deploy**. Enter the Hugging Face model ID in the **Model** field, expand **Advanced** to set **Max Model Length** and other vLLM settings, click **Next**, then **Create Endpoint**. The full walkthrough is in the [vLLM quickstart](/serverless/vllm/get-started). @@ -492,10 +492,12 @@ After, on Runpod: runpodctl serverless update ENDPOINT_ID \ --workers-min 1 \ --workers-max 3 \ - --scaler-type REQUEST_COUNT \ - --scaler-value 4 + --scale-by requests \ + --scale-threshold 4 ``` +Older `runpodctl` builds name these flags `--scaler-type` and `--scaler-value` on `serverless update`. Run `runpodctl serverless update --help` to see which your build accepts. + With `--workers-min 1`, one worker stays loaded and bills continuously, the same trade you made with `minReplicas`. Request count scaling adds workers based on queued plus in-progress requests divided by the scaler value, and it's the recommended scaler for LLM workloads. Runpod has no inactivity timeout because it isn't needed: with `--workers-min 0`, workers stop after the idle timeout and start again on the next request. To cut cold starts without paying for an always-on worker, keep the model cached, leave FlashBoot on, and raise the idle timeout (`--idle-timeout`, in seconds) so a worker stays warm between bursts. REST API v2 rejects an idle timeout on a queue-based endpoint that scales on request count, so pair a long idle timeout with queue delay scaling. The same change through REST API v2: @@ -527,7 +529,7 @@ runpodctl serverless create \ --env ENABLE_LORA=true \ --env MAX_LORA_RANK=16 \ --env 'LORA_MODULES=[{"name": "my-adapter", "path": "YOUR_HF_USER/my-adapter"}]' \ - --env HF_TOKEN=YOUR_HF_TOKEN + --env HF_TOKEN='{{ RUNPOD_SECRET_hf_token }}' ``` `MAX_LORA_RANK` must be at least the rank you trained with (the default is 16). After the endpoint is up, call `client.models.list()` to see the names it serves, and pass the adapter's name as `model` in requests. @@ -800,12 +802,12 @@ If you use the Vercel AI SDK, replace the Together provider with [`@runpod/ai-sd | Feature | Together AI | Runpod vLLM endpoint | Runpod Public Endpoints | | --- | --- | --- | --- | -| Chat completions | Supported | Supported (`/chat/completions`) | OpenAI-compatible route documented for Qwen3 32B AWQ, Moonshot Kimi, GPT-OSS 120B, and Cogito 671B. Granite 4.0 documents the native API only. | -| Text completions | Supported (legacy) | Supported (`/completions`) | Native `prompt` input on Qwen3 32B AWQ, GPT-OSS 120B, and Cogito 671B. For the OpenAI route, see the model page. | +| Chat completions | Supported | Supported (`/chat/completions`) | OpenAI-compatible route documented for Qwen3 32B AWQ and Moonshot Kimi. Granite 4.0 documents the native API only. | +| Text completions | Supported (legacy) | Supported (`/completions`) | Native `prompt` input on Qwen3 32B AWQ. For the OpenAI route, see the model page. | | Model list | Supported | Supported (`/models`) | See the model page. | -| Streaming | Supported | Supported (`stream=True`; keep `RAW_OPENAI_OUTPUT=1`) | Supported on the OpenAI-compatible route. | -| Tool calling | Supported on models marked for function calling | Depends on the model. Set `ENABLE_AUTO_TOOL_CHOICE=true` and the matching `TOOL_CALL_PARSER`. | Documented for GPT-OSS 120B. For other models, check the model. | -| Structured outputs (`response_format`) | Supported on models marked for structured outputs (`json_schema`) | Not listed in the worker docs. vLLM guided decoding is configurable with `GUIDED_DECODING_BACKEND`. Test it with your model. | See the model page. | +| Streaming | Supported | Supported (`stream=True`) | Supported on the OpenAI-compatible route. | +| Tool calling | Supported on models marked for function calling | Depends on the model. Set `ENABLE_AUTO_TOOL_CHOICE=true` and the matching `TOOL_CALL_PARSER`. | Not covered on the model pages. Check the model. | +| Structured outputs (`response_format`) | Supported on models marked for structured outputs (`json_schema`) | Not listed in the worker docs. The request goes to vLLM, which supports structured outputs, and the backend is set with `STRUCTURED_OUTPUTS_CONFIG`. Test it with your model. | See the model page. | | Vision (image inputs) | Supported on vision models | Depends on the model | Check the model. Kimi K2.6 lists multimodal input. | | Reasoning output | `reasoning` or `reasoning_content`, depending on the model | Set `REASONING_PARSER` for the model. The field name depends on the vLLM version. | Kimi supports `reasoning_effort`. | | Log probabilities | Supported, in Together's own response shape | `MAX_LOGPROBS` sets the limit (default 20). Test the response shape against your parser. | See the model page. | @@ -813,7 +815,7 @@ If you use the Vercel AI SDK, replace the Together provider with [`@runpod/ai-sd | Rerank | Dedicated endpoints only (`/v1/rerank`) | Not among the documented routes | None listed | | Batch | Together's batch API (JSONL file) | [Batch jobs](/serverless/batch-jobs) or `/run` with a webhook | Asynchronous `/run` | | Images, speech, transcription through OpenAI SDK methods | Supported | Not applicable (text models) | Native `/run` and `/runsync` API, not the OpenAI image or audio routes | -| Responses API, Messages API | Supported | Not documented | Not documented | +| Responses API, Messages API | Supported | The [worker README](https://github.com/runpod-workers/worker-vllm#usage-openai-compatibility) lists `/openai/v1/responses` and `/openai/v1/messages`. The Runpod docs don't cover them, so test before you rely on them. | Not documented | Tool calling and structured output requests use the same OpenAI request fields on both platforms (`tools`, `tool_choice`, `response_format`), so client code doesn't change once the endpoint is configured for the model. Together's `logprobs` shape differs from OpenAI's, so code that parses it needs a rewrite either way. @@ -855,7 +857,7 @@ result = response.json() print(result["output"]["result"]) ``` -Each Public Endpoint defines its own input and output fields, so check the model page. Qwen Image returns the image URL in `output.result`, and most other image models return it in `output.image_url`. Input names such as `size`, `width`, `height`, or `aspect_ratio` also vary by model. Output URLs expire after 7 days, so download anything you need to keep. Together's video API already uses submit and poll (`client.videos.create`, then `client.videos.retrieve` until `completed`). On Runpod, submit with `/run`, poll `/status/JOB_ID` until `COMPLETED`, and read the URL from `output`. +Each Public Endpoint defines its own input and output fields, so check the model page. Qwen Image returns the image URL in `output.result`, as most image and video models do, and the Flux models return it in `output.image_url`. Input names such as `size`, `width`, `height`, or `aspect_ratio` also vary by model. Output URLs expire after 7 days, so download anything you need to keep. Together's video API already uses submit and poll (`client.videos.create`, then `client.videos.retrieve` until `completed`). On Runpod, submit with `/run`, poll `/status/JOB_ID` until `COMPLETED`, and read the URL from `output`. ### Batch jobs @@ -1045,7 +1047,7 @@ Note the two Runpod hosts: `api.runpod.ai` serves requests to endpoints, and `ap | GPU count in the config (`gpu_count`) | `--gpu-count` (`gpu.count`) | | `--min-replicas` (`minReplicas`, `min_replicas`) | `--workers-min` (`workers.min`) | | `--max-replicas` (`maxReplicas`, `max_replicas`) | `--workers-max` (`workers.max`) | -| `--scaling-metric`, `--scaling-target` | `--scale-by` and `--scale-threshold` on create, `--scaler-type` and `--scaler-value` on update (`scaling`) | +| `--scaling-metric`, `--scaling-target` | `--scale-by` and `--scale-threshold` on create and update (`scaling`) | | `--scale-down-window` | `--idle-timeout` in seconds (`workers.idleTimeout`) | | `--inactive-timeout` in minutes | Not needed. Use `--workers-min 0`. | | `--max-concurrent-requests-per-replica` | `MAX_CONCURRENCY` environment variable on the vLLM worker | diff --git a/migrate/vast-ai.mdx b/migrate/vast-ai.mdx index 07d8e3b83..0c9aa8bbe 100644 --- a/migrate/vast-ai.mdx +++ b/migrate/vast-ai.mdx @@ -84,7 +84,7 @@ On the Runpod side: - **Bidding for interruptible capacity.** Vast lets you set a bid and pauses the instance when you're outbid. Pod creation in `runpodctl` and REST API v2 has no bid or interruptible field. Run these jobs as on-demand Pods with checkpoints, as a Serverless queue endpoint that scales to zero between jobs, or as an `@app.task` function that provisions a Pod per call and terminates it when the function returns. - **Self-serve reserved discounts.** `vastai prepay instance` has no counterpart. Runpod's savings plans are deprecated and can't be purchased. For committed capacity or custom pricing, [contact sales](https://www.runpod.io/contact). - **Choosing a host.** Filters such as `reliability`, `inet_down`, `dlperf`, `static_ip`, and `machine_id` don't exist. You choose a GPU type, a cloud tier, a data center or country, a minimum CUDA version, and minimum vCPU and RAM per GPU. -- **Virtual machines and Docker inside the instance.** Vast VM instances support systemd and nested containers. Pods are containers and don't run Docker Compose. Package each service as its own image and run one Pod per service (Pods in the same account can reach each other over [global networking](/pods/networking)), or run the services as processes in one image. +- **Virtual machines and Docker inside the instance.** Vast VM instances support systemd and nested containers. Pods are containers and don't run Docker Compose. Package each service as its own image and run one Pod per service (NVIDIA GPU Pods in the same account can reach each other over [global networking](/pods/networking)), or run the services as processes in one image. - **ARM hosts.** Vast lists `cpu_arch=arm64` offers. Runpod images must be `linux/amd64`. - **Copying data off a stopped instance.** `vastai copy` and Cloud Sync work on stopped Vast instances. On Runpod, put data you need without a running Pod on a network volume and read it through the S3-compatible API. - **Scheduled Cloud Sync.** `vastai cloud copy --schedule` has no counterpart. Run `aws s3 sync` or `rclone` from cron inside the Pod, or from your own scheduler against the S3-compatible API. @@ -320,7 +320,7 @@ The before and after below launch the same vLLM server: one image, one port, one -The server is then reachable at `https://POD_ID-8000.proxy.runpod.net/v1/models`. Three things changed: there's no offer ID, the port carries a type, and the Hugging Face token comes from a [secret](/pods/templates/secrets) instead of sitting in the command. Set the CUDA floor to the version your image was built for. +Create the `hf_token` secret before you run the Runpod versions. The server is then reachable at `https://POD_ID-8000.proxy.runpod.net/v1/models`. That URL is public, so set an API key on the server (vLLM's `--api-key` argument) before you put real traffic on it. Three things changed: there's no offer ID, the port carries a type, and the Hugging Face token comes from a [secret](/pods/templates/secrets) instead of sitting in the command. Set the CUDA floor to the version your image was built for. Each launch mode translates as follows. @@ -591,7 +591,7 @@ Pick the route that matches where the data is now. cd /workspace && runpodctl receive CODE ``` - For large or resumable transfers, use `rsync` over the Pod's direct SSH port. The Pod needs `22/tcp` in its ports. Read the host and port from `runpodctl ssh info POD_ID`. + For large or resumable transfers, use `rsync` over the Pod's direct SSH port. The Pod needs `22/tcp` in its ports. Read the host and port from `runpodctl ssh info POD_ID`. The Vast instance must hold a private key that the Pod accepts: connect to the instance with SSH agent forwarding (`ssh -A`), or create a key pair on the instance and register its public key with `runpodctl ssh add-key` before you create the Pod. ```bash # On the Vast instance @@ -749,6 +749,7 @@ curl -s -X POST "https://api.runpod.io/v2/serverless" \ "type": "LOAD_BALANCER", "image": "my-org/my-server:1.0", "disk": 30, + "ports": ["8000/http"], "env": {"PORT": "8000", "PORT_HEALTH": "8000"}, "gpu": {"pools": ["ADA_24"], "count": 1}, "workers": {"min": 0, "max": 3}, @@ -996,7 +997,7 @@ def create_pod(): return response.json() if response.status_code in (400, 403): continue # couldn't be placed, or no access to this GPU type: try the next one - response.raise_for_status() # 401, 402, 422, 429: fix the problem, don't loop + response.raise_for_status() # 401, 422, 429: fix the problem, don't loop raise SystemExit("None of the candidate GPU types could be placed") @@ -1153,11 +1154,16 @@ Compare outputs against Vast for the same inputs and seeds. Measure latency for **Set a cost guard on Runpod.** Set `workers.max` and the idle timeout on each endpoint, enable low balance notifications and auto-pay on the [billing page](/accounts-billing/billing), and review `runpodctl pod list --all` and `runpodctl billing pods` after the first day. A stopped Pod still bills for its volume disk, and a network volume bills until you delete it. -**Shut down Vast.** Stopped Vast instances and volumes keep billing for storage, so destroy them once the data is verified on Runpod. Delete endpoints and workergroups first, then destroy any worker instances they leave behind. +**Shut down Vast.** Stopped Vast instances and volumes keep billing for storage, so destroy them once the data is verified on Runpod. Delete Deployments, endpoints, and workergroups first, then destroy any worker instances they leave behind. ```bash +# Deployments (beta) +vastai show deployments --raw +vastai delete deployment DEPLOYMENT_ID + # Serverless vastai show endpoints --raw | jq -r '.[].id' +vastai show workergroups --raw vastai delete workergroup WORKERGROUP_ID vastai delete endpoint ENDPOINT_ID From 5284e02a359f721b6f6391b24407c1b94f4a86d7 Mon Sep 17 00:00:00 2001 From: Luke Piette Date: Sat, 14 Nov 2026 15:07:34 -0500 Subject: [PATCH 3/3] docs(migrate): replace the deprecated runpodctl config command runpodctl v3.1.1 lists config as deprecated. Checked all 804 runpodctl invocations in the guides and playbooks against a v3.1.1 build: these two rows were the only ones using a deprecated command. Co-Authored-By: Claude Fable 5.2 --- migrate/fireworks-ai.mdx | 2 +- migrate/paperspace.mdx | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/migrate/fireworks-ai.mdx b/migrate/fireworks-ai.mdx index 1c97a7100..71079489f 100644 --- a/migrate/fireworks-ai.mdx +++ b/migrate/fireworks-ai.mdx @@ -782,7 +782,7 @@ On the Runpod side, guard cost with three settings: max workers caps concurrency | `firectl` | `runpodctl` | | --- | --- | -| `firectl signin`, `firectl set-api-key API_KEY` | `runpodctl doctor`, or `runpodctl config --apiKey YOUR_API_KEY` | +| `firectl signin`, `firectl set-api-key API_KEY` | `runpodctl doctor`, or `export RUNPOD_API_KEY=YOUR_API_KEY` | | `firectl whoami` | `runpodctl user` | | `firectl deployment create MODEL --deployment-shape default` | `runpodctl serverless create --hub-id HUB_ID --gpu-id POOL --model-reference URL` | | `firectl deployment list` | `runpodctl serverless list` | diff --git a/migrate/paperspace.mdx b/migrate/paperspace.mdx index 9a825fc0d..d6429bfed 100644 --- a/migrate/paperspace.mdx +++ b/migrate/paperspace.mdx @@ -1124,7 +1124,7 @@ Migrate in two applies: create the Runpod resources alongside the DigitalOcean o | `pspace` | `doctl` | `runpodctl` | | --- | --- | --- | -| `pspace login` | `doctl auth init` | `runpodctl doctor` or `runpodctl config --apiKey YOUR_API_KEY` | +| `pspace login` | `doctl auth init` | `runpodctl doctor` or `export RUNPOD_API_KEY=YOUR_API_KEY` | | `pspace machine list` | `doctl compute droplet list --gpus` | `runpodctl pod list --all` | | `pspace machine get MACHINE_ID` | `doctl compute droplet get DROPLET_ID` | `runpodctl pod get POD_ID` | | `pspace machine create` | `doctl compute droplet create` | `runpodctl pod create` |