concepts/api-key.yaml
id: api-key
name: API key
kind: resource
summary: >
A credential that authenticates your API, CLI and MCP calls. Each key carries
the permissions it was given at creation, and Runpod shows it only once.
product: account
is_a: null
part_of: runpod-platform
aliases: [Runpod API key]
surfaces:
console: Settings
relations:
- { type: constrained_by, target: api-rate-limit, cardinality: "1" }
rules:
- id: api-key.bearer-only
statement: >
REST v2 accepts the key only in the header `Authorization: Bearer <key>`.
A request without a valid key returns 401. A valid key that lacks access
to the resource or action returns 403.
on_violation: 401 missing or invalid key, 403 missing access
status: documented
evidence:
- { source: rest-v2-spec, ref: "components.securitySchemes.bearerAuth; components.responses.ForbiddenError", seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/api-reference-v2/overview", seen: "2026-09-28" }
- id: api-key.graphql-query-parameter
statement: >
The GraphQL API takes the key as the `api_key` query parameter. When
migrating to REST v2, move the key into the `Authorization` header so it
stops appearing in URLs and logs.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/sdks/graphql/configurations", seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpod-migrate/reference/graphql-to-v2.md }
- id: api-key.shown-once
statement: >
Runpod does not store the key, so copy it when you create it and keep it
in a password manager or secret store. You can edit a key's permissions,
disable it, re-enable it or revoke it from the Settings page.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/get-started/api-keys", seen: "2026-09-28" }
- id: api-key.permission-levels
statement: >
A key is created with **All**, **Restricted** or **Read Only** access. A
Restricted key sets access per Runpod API and per Serverless endpoint
(None, Read Only or Read/Write), so grant only what the integration needs.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/get-started/api-keys", seen: "2026-09-28" }
- id: api-key.legacy-keys
statement: >
Keys generated before November 11, 2024 have either Read/Write or Read
Only access to GraphQL, plus full access to the Serverless job API.
Replace them with Restricted keys that carry the minimum permissions.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/get-started/api-keys", seen: "2026-09-28" }
concepts/api-rate-limit.yaml
id: api-rate-limit
name: API rate limit
kind: limit
summary: >
Per-user request limits on the Runpod API, counted over minute, hour and day
windows. Exceeding one returns 429 with a `Retry-After` header.
product: account
is_a: null
part_of: runpod-platform
aliases: [rate limit, "429", RateLimit header, Retry-After]
relations: []
rules:
- id: api-rate-limit.headers
statement: >
Authenticated REST v2 responses carry `RateLimit` (remaining requests `r`
and seconds to reset `t`) and `RateLimit-Policy` (quota `q` and window
length `w`) for the minute, hour and day windows, not only on 429s. Read
them to slow down before you hit the limit. Callers exempt from rate
limits receive neither header.
status: documented
evidence:
- { source: rest-v2-spec, ref: "components.headers.RateLimit; RateLimitHeader; RateLimitPolicyHeader", seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpod-migrate/reference/unlocks.md }
- id: api-rate-limit.on-429
statement: >
A 429 names the exceeded window in `detail` and sets `Retry-After` in
seconds. Wait that long, then retry with backoff.
on_violation: 429
status: documented
see: [api-rate-limit.headers]
evidence:
- { source: rest-v2-spec, ref: components.responses.TooManyRequestsError, seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpodctl/reference/output-and-errors.md }
- id: api-rate-limit.trust-the-policy-header
statement: >
Read quotas from `RateLimit-Policy` rather than hard-coding numbers.
Limits are set per user and can differ between accounts.
status: documented
see: [api-rate-limit.headers]
evidence:
- { source: rest-v2-spec, ref: RateLimitPolicyHeader, seen: "2026-09-28" }
- id: api-rate-limit.serverless-job-routes
statement: >
A Serverless endpoint's job routes have their own limits per user and
endpoint. For example, `/run` allows 1,000 requests per 10 seconds,
`/runsync`, `/status` and `/stream` allow 2,000, and `/purge-queue` allows
2. The effective limit grows with the endpoint's running workers. A 429
means back off exponentially, not poll faster.
on_violation: 429
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/send-requests", note: "See the \"Rate limits\" and \"Dynamic rate limiting\" sections.", seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpod/golden-paths/12-serverless-streaming.md }
concepts/audit-log.yaml
id: audit-log
name: Audit log
kind: resource
summary: >
The account's audit trail: who did what to which resource, newest first. For a
member of an organization it is the organization's trail.
product: account
is_a: null
part_of: runpod-platform
aliases: [audit trail, AuditLog, audit logs]
surfaces:
rest_v2: { schema: AuditLog, paths: [/v2/audit-logs] }
mcp: [list-audit-logs]
fields:
- { name: actorId, type: string, set: read, note: null when the action is not attributable to a user }
- { name: email, type: string, set: read }
- { name: resourceType, type: string, set: read, note: "free-form, not an enum" }
- { name: resourceId, type: string, set: read }
- { name: action, type: string, set: read, note: "free-form, not an enum" }
- { name: value, type: string, set: read, note: JSON-encoded details of the change }
- { name: timestamp, type: string, set: read }
relations:
- { type: uses, target: organization, cardinality: "0..1" }
rules:
- id: audit-log.plan-gated
statement: >
An account whose plan does not include audit logs is answered `403`. State
that as the answer, not as a tool failure.
on_violation: 403
status: documented
evidence:
- { source: rest-v2-spec, ref: GET /v2/audit-logs, seen: "2026-11-02" }
- id: audit-log.organization-trail
statement: >
When the caller belongs to an organization, the trail is the
organization's; otherwise it is the trail of the account the request acts
for. Entries older than the account's retention window are not returned.
status: documented
see: [organization.membership-required]
evidence:
- { source: rest-v2-spec, ref: GET /v2/audit-logs, seen: "2026-11-02" }
- id: audit-log.actor-filter-organization-only
applies_to: ["field:actorId"]
statement: >
The `actorIds` filter works for organization accounts only; a personal or
team account that sends it is answered `400`. Filter by `actions`,
`startTime` and `endTime` on any account.
on_violation: 400
status: documented
evidence:
- { source: rest-v2-spec, ref: AuditLogActorIdsFilter, seen: "2026-11-02" }
- id: audit-log.free-form-values
applies_to: ["field:action", "field:resourceType"]
statement: >
`action` and `resourceType` are free-form strings recorded at write time,
and new values appear without an API version change. The `actions` filter
matches them exactly as recorded, so read a page first to learn the names.
status: documented
evidence:
- { source: rest-v2-spec, ref: "AuditLog.action; AuditLog.resourceType", seen: "2026-11-02" }
- { source: rest-v2-spec, ref: AuditLogActionsFilter, seen: "2026-11-02" }
concepts/billing-record.yaml
id: billing-record
name: Billing record
kind: billing
summary: >
A time bucket of spend (an hour, day, week, month or year) with the total and
each cost component broken out: pods, serverless, storage, Public Endpoints
and Instant Clusters.
product: billing
is_a: null
part_of: runpod-platform
aliases: [billing, spend, usage, billing history, UserBilling]
surfaces:
rest_v2: { schema: BillingRecord, paths: [/v2/billing, /v2/billing/endpoints, /v2/billing/clusters] }
graphql: { type: UserBilling }
mcp: [list-billing, list-endpoint-billing, list-cluster-billing]
console: Billing
fields:
- { name: startTime, type: string, set: read, note: "Inclusive, RFC 3339." }
- { name: endTime, type: string, set: read, note: "Exclusive, RFC 3339." }
- { name: totalAmount, type: number, unit: USD, set: read }
- { name: podGpuAmount, type: number, unit: USD, set: read }
- { name: podCpuAmount, type: number, unit: USD, set: read }
- { name: podDiskAmount, type: number, unit: USD, set: read }
- { name: serverlessGpuAmount, type: number, unit: USD, set: read }
- { name: serverlessCpuAmount, type: number, unit: USD, set: read }
- { name: serverlessDiskAmount, type: number, unit: USD, set: read }
- { name: serverlessFeeAmount, type: number, unit: USD, set: read, note: Always 0. }
- { name: storageStandardAmount, type: number, unit: USD, set: read }
- { name: storageHighPerformanceAmount, type: number, unit: USD, set: read }
- { name: endpointAmount, type: number, unit: USD, set: read, note: Public Endpoint spend. }
- { name: clusterGpuAmount, type: number, unit: USD, set: read }
- { name: clusterDiskAmount, type: number, unit: USD, set: read }
- { name: clusterNetworkingAmount, type: number, unit: USD, set: read }
relations: []
rules:
- id: billing-record.query-window
applies_to: ["field:startTime", "field:endTime"]
statement: >
Billing is read-only history. `bucketSize` is `hour`, `day`, `week`,
`month` or `year` (default `day`). The window defaults to the last 30 days
and snaps to bucket boundaries, with `endTime` exclusive. Use `lastN` with
`bucketSize` for the most recent buckets; the last bucket is still in
progress.
status: documented
evidence:
- { source: rest-v2-spec, ref: "GET /v2/billing parameters; BillingBucketSize", seen: "2026-09-28" }
- id: billing-record.one-call-total
applies_to: ["field:totalAmount"]
statement: >
`GET /v2/billing` returns total spend across every resource with each
component broken out, so there is no need to add up the per-resource
billing routes.
status: documented
evidence:
- { source: rest-v2-spec, ref: GET /v2/billing, seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpod-migrate/reference/unlocks.md }
- id: billing-record.component-breakdown
applies_to: ["field:podDiskAmount", "field:storageStandardAmount", "field:storageHighPerformanceAmount", "field:serverlessDiskAmount"]
statement: >
Storage spend is split into pod disk, serverless disk, and standard and
high-performance network volume storage. `endpointAmount` is Public
Endpoint spend, not serverless spend.
status: documented
see: [serverless-billing.endpoints-route-is-public-endpoints]
evidence:
- { source: rest-v2-spec, ref: BillingRecord, seen: "2026-09-28" }
- id: billing-record.fee-always-zero
applies_to: ["field:serverlessFeeAmount"]
statement: >
`serverlessFeeAmount` is always 0. Platform charges are included in the
serverless compute amounts.
status: documented
evidence:
- { source: rest-v2-spec, ref: BillingRecord.serverlessFeeAmount, seen: "2026-09-28" }
- id: billing-record.per-second-billing
statement: >
Compute and pod container and volume disk are billed per second, charges
are deducted from the account's prepaid credits as resources run, and data
transfer is free. Billing runs every 5 minutes. Two kinds of storage bill
on a coarser unit: network volumes hourly, and a Serverless worker's
container disk in 5-minute intervals.
status: documented
see: [credit-balance.low-balance-actions, container-disk.billing, network-volume-billing.price]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/accounts-billing/billing", seen: "2026-09-28" }
- id: billing-record.cost-centers
statement: >
Cost centers attribute spend to a team, project or department. Each pod,
serverless endpoint, network volume or Instant Cluster belongs to at most
one cost center, and unassigned resources are reported as uncategorized.
Cost centers are managed in the console; the REST v2 billing routes take
no cost-center filter.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/accounts-billing/cost-centers", seen: "2026-09-28" }
- { source: rest-v2-spec, ref: GET /v2/billing parameters, seen: "2026-09-28" }
concepts/cloud-tier.yaml
id: cloud-tier
name: Cloud tier
kind: catalog
summary: >
Where a pod's hardware comes from: Secure Cloud, which runs in T3/T4 data
centers and is the default, or Community Cloud, which uses vetted independent
providers at lower prices.
product: null
is_a: null
part_of: runpod-platform
aliases: [Secure Cloud, Community Cloud, cloud, cloudType, SECURE, COMMUNITY]
surfaces:
rest_v2: { schema: Cloud, paths: [] }
graphql: { type: CloudTypeEnum }
runpodctl: [pod create --cloud-type]
fields: []
relations:
- { type: constrained_by, target: gpu-availability, cardinality: "1" }
rules:
- id: cloud-tier.enum-and-default
statement: >
In REST v2, `cloud` is `SECURE` or `COMMUNITY` and defaults to `SECURE` on
pod create. GraphQL's `cloudType: ALL` has no REST v2 equivalent, so pick
one tier.
status: documented
evidence:
- { source: rest-v2-spec, ref: "Cloud; CreatePodRequest.cloud", seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpod-migrate/reference/breaking-changes.md }
- id: cloud-tier.default-to-secure
statement: >
Use Secure Cloud unless the user asks for Community Cloud. Secure Cloud
runs in T3/T4 data centers with high redundancy; Community Cloud is
cheaper but less predictable. Community Cloud does not accept new hosts,
and its existing capacity remains available.
status: documented
see: [network-volume.secure-cloud-only]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/choose-a-pod", seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/concepts.md }
- id: cloud-tier.price-and-offer-per-gpu
statement: >
Each GPU type states whether each cloud offers it and lists a price per
cloud. A `false` for a cloud means that cloud does not offer the GPU. Do
not assume Community Cloud is always cheaper; compare the two prices.
status: documented
see: [gpu-type.price-is-list-per-gpu]
evidence:
- { source: rest-v2-spec, ref: "GpuType.secure; GpuType.community; GpuType.price", seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/gpu-selection.md }
- id: cloud-tier.community-ip-changes
statement: >
A Community Cloud pod's public IP address can change when the pod is
migrated or restarted, so do not hard-code it. Secure Cloud IP addresses
are expected to stay stable.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/configuration/expose-ports", seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/gotchas.md }
- id: cloud-tier.community-absent-in-some-dcs
statement: >
Stock differs by cloud and by data center, and catalog availability
defaults to Secure Cloud. Pass `cloud=COMMUNITY` with
`include=AVAILABILITY` to see where Community Cloud has capacity before
pinning both Community Cloud and a data center.
status: documented
evidence:
- { source: rest-v2-spec, ref: "GET /v2/catalog/gpus cloud parameter; GpuCloudFilter", seen: "2026-09-28" }
concepts/container-disk.yaml
id: container-disk
name: Container disk
kind: component
summary: >
The pod's own writable filesystem, where installs, downloads and temporary
files land unless they are written to a mount. Its size is the pod's `disk`
field.
product: pods
is_a: null
part_of: pod
aliases: [ephemeral disk, container volume, local disk, containerDiskInGb]
surfaces:
rest_v2: { schema: Pod }
console: Pods
fields:
- { name: disk, type: integer, unit: GB, set: "create,update", note: Minimum 1. }
relations:
- { type: billed_by, target: pod-billing, cardinality: "1" }
rules:
- id: container-disk.what-survives
applies_to: ["field:disk"]
statement: >
The container disk is cleared when the pod stops, restarts, is reset by an
edit, or is terminated. Keep anything you need in `/workspace` on the
volume disk or on a network volume.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/storage/types", note: Container disk data is lost on stop and restart. }
- { source: rest-v2-spec, ref: ContainerConfig.disk, seen: "2026-09-28", note: Wiped on restart. }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/storage.md }
- id: container-disk.always-local
applies_to: ["field:disk"]
statement: >
The container disk is local to the machine. Where `/workspace` is
network-backed, writes to the container disk run several times faster than
writes to `/workspace`; where `/workspace` is local, the two are about the
same. Install packages, build and keep scratch files on the container disk
(for example `/root`, `/opt` or `/tmp`), and keep only what must survive a
stop or restart in `/workspace`. Size the container disk for those
installs.
status: verified
see: [pod-volume-disk.network-filesystem-backed, container-disk.what-survives]
evidence:
- { source: live-probe, ref: "ubuntu:22.04 GPU pods with a 10 GB volume disk and no network volume in EU-RO-1, EU-SE-1, US-MO-2, OC-AU-1, US-TX-4 and US-PA-1: findmnt showed / as a local overlay on every pod. Where /workspace was a fuse mount (EU-RO-1, EU-SE-1, US-MO-2), dd 512 MB oflag=direct ran 713, 443 and 568 MB/s on /workspace and 2.8, 4.1 and 2.4 GB/s on the container disk", seen: "2026-10-26" }
- id: container-disk.billing
applies_to: ["field:disk"]
statement: >
The container disk is billed per GB while the pod runs. On Serverless,
each worker's container disk is billed in 5-minute intervals.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/storage/types" }
- { source: public-docs, url: "https://docs.runpod.io/serverless/pricing" }
- id: container-disk.resize
applies_to: ["field:disk"]
statement: >
`disk` can be changed on an existing pod, but that edit resets a running
pod. Size the disk for model downloads at create time.
status: documented
see: [pod.edits-apply-on-restart]
evidence:
- { source: rest-v2-spec, ref: UpdatePodRequest.disk, seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/pods/storage/types", note: Modify storage capacity. }
concepts/container-image.yaml
id: container-image
name: Container image
kind: external
summary: >
The Docker image a pod or Serverless worker runs, pulled from Docker Hub,
GHCR, a private registry, or Runpod's registry for Hub builds.
product: templates
is_a: null
part_of: null
aliases: [image, imageName, Docker image, worker image, runpod/pytorch, runpod/base]
surfaces:
console: Templates
relations:
- { type: uses, target: registry-credential, cardinality: "0..1" }
rules:
- id: container-image.use-runpod-pytorch
statement: >
Prefer an official Runpod base when its framework and CUDA stack match the
application. Reuse that stack rather than reinstalling it, and verify the
application on the target GPU; a base image alone does not establish every
dependency's compatibility.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/templates/create-custom-template", seen: "2026-10-13" }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/building-images.md }
- id: container-image.pin-exact-tag
statement: >
Choose a published versioned tag, for example
`runpod/pytorch:1.0.2-cu1281-torch280-ubuntu2404`, and record its resolved
digest for reproducible deployment. Record dependency and model revisions
too, since pinning a base does not pin later downloads.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/workers/deploy", seen: "2026-10-13", note: Image versioning and SHA references. }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/building-images.md }
- id: container-image.build-for-amd64
statement: >
Runpod hosts are x86_64. Build with `docker build --platform=linux/amd64`,
especially on an Apple Silicon machine, or the image will not start.
status: documented
evidence:
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/building-images.md }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/docker.md }
- id: container-image.ssh-on-custom-base
statement: >
Runpod PyTorch bases use `/start.sh` for services including SSH, with
`PUBLIC_KEY` handling and startup hooks. Preserve that startup when the
pod promises those services, or implement them on a custom base;
application-only pods may intentionally omit SSH and Jupyter.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/templates/create-custom-template", seen: "2026-10-13", note: Three startup modes. }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/building-images.md }
- { source: skill, path: plugins/runpod/skills/runpod/golden-paths/22-minimal-pod-image/README.md }
- id: container-image.layer-reuse
statement: >
Extending a published dependency image preserves its existing layers;
independently installing the same package versions does not guarantee
identical layers. Put dependency installation before changing source, and
distinguish builder cache reuse from layers cached on deployment hosts.
status: documented
evidence:
- { source: public-docs, url: "https://docs.docker.com/build/cache/optimize/", seen: "2026-10-13", note: Dependency layers precede changing application source. }
- { source: public-docs, url: "https://docs.docker.com/engine/storage/drivers/", seen: "2026-10-13", note: Images share existing layers. }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/building-images.md, note: "Layer order, shared dependency bases and cache boundaries." }
- id: container-image.linux-builder-permissions
statement: >
For a local Linux Docker Engine, membership in the `docker` group grants
daemon access without `sudo` and carries root-level privileges. After
adding the user, log out and back in so the login session receives the new
group membership.
status: documented
evidence:
- { source: public-docs, url: "https://docs.docker.com/engine/install/linux-postinstall/", seen: "2026-10-14" }
- { source: skill, path: plugins/runpod/skills/runpod-companion-clis/reference/docker-setup.md }
concepts/cpu-flavor.yaml
id: cpu-flavor
name: CPU flavor
kind: catalog
summary: >
A CPU instance type, such as `cpu3c` or `cpu5m`, with a vCPU range, a fixed
amount of RAM per vCPU and per-vCPU prices. CPU pods and CPU serverless
endpoints are sized from it.
product: null
is_a: null
part_of: runpod-platform
aliases: [CpuType, CpuFlavor, CPU instance type, CPU type, cpu3c, cpu5c]
surfaces:
rest_v2: { schema: CpuType, paths: [/v2/catalog/cpus, "/v2/catalog/cpus/{id}"] }
graphql: { type: CpuFlavor }
mcp: [list-cpu-types, get-cpu-type]
fields:
- { name: id, type: string, set: read, note: "Pass as `cpu.id` on create." }
- { name: name, type: string, set: read }
- { name: group, type: string, set: read, note: CPU generation group. }
- { name: vcpu.min, type: integer, set: read }
- { name: vcpu.max, type: integer, set: read }
- { name: ramGbPerVcpu, type: number, unit: GB, set: read }
- { name: price.securePerVcpu, type: number, unit: USD per vCPU per hour, set: read }
- { name: price.serverlessPerVcpu, type: number, unit: USD per vCPU per hour, set: read }
- { name: availability, type: AvailabilityLevel, set: read, note: "Returned only with `include=AVAILABILITY` and `product`." }
- { name: dataCenters, type: "DataCenterAvailability[]", set: read, note: "Returned only with `include=AVAILABILITY` and `product`." }
relations:
- { type: constrained_by, target: gpu-availability, cardinality: "1" }
rules:
- id: cpu-flavor.memory-is-derived
applies_to: ["field:ramGbPerVcpu"]
statement: >
RAM is `ramGbPerVcpu` times the vCPU count. A CPU pod create sends only
`cpu.id` and `cpu.vcpuCount`, and the API derives the memory.
status: documented
evidence:
- { source: rest-v2-spec, ref: "CpuType.ramGbPerVcpu; CreatePodRequest", seen: "2026-09-28" }
- id: cpu-flavor.vcpu-power-of-two
applies_to: ["field:vcpu.min", "field:vcpu.max"]
statement: >
`vcpuCount` must be a power of two, at least 2, and within the flavor's
`vcpu.min` to `vcpu.max` range.
on_violation: 400
status: documented
evidence:
- { source: rest-v2-spec, ref: "BaseCpuConfig.vcpuCount; GET /v2/catalog/cpus vcpuCount", seen: "2026-09-28" }
- id: cpu-flavor.price-per-vcpu
applies_to: ["field:price.securePerVcpu", "field:price.serverlessPerVcpu"]
statement: >
Prices are per vCPU per hour. Multiply by the chosen `vcpuCount` for the
hourly rate.
status: documented
evidence:
- { source: rest-v2-spec, ref: CpuType.price, seen: "2026-09-28" }
- id: cpu-flavor.availability-per-product
applies_to: ["field:availability", "field:dataCenters"]
statement: >
CPU availability differs between `POD` and `SERVERLESS`. Request it with
`include=AVAILABILITY` and `product`. Per data center, the same figures
appear as `DataCenter.cpuAvailability`.
status: verified
see: [data-center.availability-on-request]
evidence:
- { source: rest-v2-spec, ref: "GET /v2/catalog/cpus parameters; CpuProduct", seen: "2026-09-28" }
- { source: live-probe, ref: "GET /v2/catalog/cpus?include=AVAILABILITY without product returned 400 (product is required with include=AVAILABILITY)", seen: "2026-09-28" }
- id: cpu-flavor.id-field-names
applies_to: ["field:id"]
statement: >
REST v2 takes the flavor as `cpu.id`. GraphQL selects CPU flavors through
`instanceIds` together with `computeType: CPU`.
status: documented
evidence:
- { source: rest-v2-spec, ref: BaseCpuConfig.id, seen: "2026-09-28" }
- { source: public-docs, url: "https://graphql-spec.runpod.io/", ref: "instanceIds; computeType", seen: "2026-09-28" }
concepts/cpu-pod.yaml
id: cpu-pod
name: CPU pod
kind: resource
summary: >
A pod on CPU-only hardware, priced by vCPU. It has the same networking,
storage options and lifecycle as a GPU pod, apart from the limits below.
product: pods
is_a: pod
part_of: null
aliases: [CPU-only pod, CPU instance]
surfaces:
rest_v2: { schema: Pod, paths: [/v2/pods, /v2/catalog/cpus] }
mcp: [create-pod, list-cpu-types, get-cpu-type]
runpodctl: [pod create --compute-type cpu]
console: Pods
fields:
- { name: cpu, type: CreateCpuConfig, set: create, required: true }
- { name: cpu.id, type: string, ref: cpu-flavor, set: create, required: true, note: "A CPU flavor id from `GET /v2/catalog/cpus`, for example `cpu5c`." }
- { name: cpu.vcpuCount, type: integer, set: create, required: true, note: "A power of two, at least 2." }
- { name: cpu.memory, type: integer, unit: GB, set: read, note: Derived from the flavor. }
relations:
- { type: runs_on, target: cpu-flavor, cardinality: "1" }
- { type: located_in, target: data-center, cardinality: "1" }
- { type: billed_by, target: pod-billing, cardinality: "1" }
rules:
- id: cpu-pod.flavor-and-vcpus
applies_to: ["field:cpu.id", "field:cpu.vcpuCount", "field:cpu.memory"]
statement: >
A CPU pod takes a CPU flavor id in `cpu.id`, not a GPU type id, plus a
`vcpuCount` for that flavor. Do not send memory; the API derives it from
the flavor.
status: documented
see: [cpu-flavor.vcpu-power-of-two, cpu-flavor.memory-is-derived]
evidence:
- { source: rest-v2-spec, ref: "CreatePodRequest; BaseCpuConfig; CpuConfig.memory", seen: "2026-09-28" }
- id: cpu-pod.storage-limits
applies_to: ["field:cpu"]
statement: >
CPU pods support the container disk and a network volume only. A
persistent (host-local) mount is rejected.
on_violation: 400
status: documented
see: [pod.persistent-mount-limits]
evidence:
- { source: rest-v2-spec, ref: CreatePodRequest, seen: "2026-09-28" }
- id: cpu-pod.no-global-networking
applies_to: ["field:cpu"]
statement: >
CPU pods cannot use global networking, which needs an NVIDIA GPU pod.
status: documented
see: [pod.global-networking-next-start]
evidence:
- { source: rest-v2-spec, ref: CreatePodRequest.globalNetworking, seen: "2026-09-28" }
concepts/credit-balance.yaml
id: credit-balance
name: Credit balance and spend limit
kind: billing
summary: >
The prepaid balance that all usage draws from, and the account's hourly spend
limit. When the balance reaches $0, pods stop and storage can be lost.
product: billing
is_a: null
part_of: runpod-platform
aliases: [balance, clientBalance, credits, spend limit, spendLimit, low balance, auto-pay]
surfaces:
graphql: { type: User, queries: [myself] }
runpodctl: [user]
console: Billing
fields:
- { name: clientBalance, type: number, unit: USD, set: read, note: "Read from GraphQL `myself`." }
- { name: spendLimit, type: number, unit: USD per hour, set: read, note: "Read from GraphQL `myself`." }
relations: []
rules:
- id: credit-balance.not-in-rest-v2
applies_to: ["field:clientBalance"]
statement: >
REST v2 has no balance route; `/v2/billing` returns spend history only.
Read the live balance with `runpodctl user` or GraphQL `myself`.
status: documented
evidence:
- { source: skill, path: plugins/runpod/skills/runpod-migrate/reference/graphql-to-v2.md }
- { source: rest-v2-spec, ref: paths, seen: "2026-09-28" }
- { source: other, ref: runpodctl user --help, seen: "2026-09-28" }
- id: credit-balance.low-balance-actions
applies_to: ["field:clientBalance"]
statement: >
When the balance reaches $0, all running pods stop. Pods with a network
volume keep their data on the volume; pods without one are terminated and
their data cannot be recovered. Network volumes keep accruing storage
charges, and a volume whose charges stay unpaid can be deleted.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/accounts-billing/billing", seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/pods/pricing", seen: "2026-09-28" }
- id: credit-balance.avoid-interruptions
applies_to: ["field:clientBalance"]
statement: >
Low balance notifications email the account when the balance drops below a
chosen threshold. Auto-pay reloads the balance from a saved card, at most
once per hour. Use either or both to keep workloads running.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/accounts-billing/billing", seen: "2026-09-28" }
- id: credit-balance.spend-limit
applies_to: ["field:spendLimit"]
statement: >
Accounts start with a spend limit of $80 per hour across all resources.
The limit rises automatically with account history, and support can raise
it sooner on request.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/accounts-billing/billing", seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/serverless/pricing", seen: "2026-09-28" }
- id: credit-balance.credits-non-refundable
statement: >
Credits are non-refundable, cannot be withdrawn, and can be spent only on
Runpod services.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/accounts-billing/billing", seen: "2026-09-28" }
concepts/data-center.yaml
id: data-center
name: Data center
kind: catalog
summary: >
A physical location where pods and workers run and network volumes live. Each
data center lists the GPUs, CPUs, volume tiers, networking and compliance
certifications it offers.
product: null
is_a: null
part_of: runpod-platform
aliases: [datacenter, DC, dataCenterId, location, US-TX-3, EU-RO-1]
surfaces:
rest_v2: { schema: DataCenter, paths: [/v2/catalog/datacenters, "/v2/catalog/datacenters/{id}"] }
graphql: { type: DataCenter, queries: [dataCenters] }
mcp: [list-data-centers, get-data-center]
runpodctl: [datacenter list]
fields:
- { name: id, type: string, set: read }
- { name: name, type: string, set: read }
- { name: region, type: enum, values: [NORTH_AMERICA, SOUTH_AMERICA, EUROPE, ASIA, MIDDLE_EAST, AFRICA, OCEANIA, ANTARCTICA, UNKNOWN], set: read }
- { name: networkVolumeTypes, type: "VolumeType[]", set: read }
- { name: globalNetwork, type: boolean, set: read }
- { name: compliance, type: "enum[]", values: [GDPR, ISO_IEC_27001, ISO_14001, PCI_DSS, HITRUST, SOC_1_TYPE_2, SOC_2_TYPE_2, SOC_3_TYPE_2, ITAR, FISMA_HIGH, HIPAA, RENEWABLE], set: read }
- { name: gpuAvailability, type: "CatalogResourceAvailability[]", set: read, note: only with include=GPU_AVAILABILITY }
- { name: cpuAvailability, type: "CatalogResourceAvailability[]", set: read, note: only with include=CPU_AVAILABILITY }
relations:
- { type: constrained_by, target: cloud-tier, cardinality: "1..n" }
rules:
- id: data-center.availability-on-request
applies_to: ["field:gpuAvailability", "field:cpuAvailability"]
statement: >
`gpuAvailability` and `cpuAvailability` are returned only when requested
with `include`. A missing field means "not requested" or "this data center
offers none", not "out of stock".
status: documented
evidence:
- { source: rest-v2-spec, ref: "DataCenter.gpuAvailability; DataCenter.cpuAvailability", seen: "2026-09-28" }
- id: data-center.volume-types
applies_to: ["field:networkVolumeTypes"]
statement: >
An empty `networkVolumeTypes` means no network volume can be created in
this data center.
status: documented
evidence:
- { source: rest-v2-spec, ref: DataCenter.networkVolumeTypes, seen: "2026-09-28" }
- id: data-center.global-network
applies_to: ["field:globalNetwork"]
statement: >
Only data centers with `globalNetwork: true` can host pods that use global
networking.
status: documented
evidence:
- { source: rest-v2-spec, ref: "DataCenter.globalNetwork; CreatePodRequest.globalNetworking", seen: "2026-09-28" }
- id: data-center.region-is-coarse
applies_to: ["field:region"]
statement: >
`region` is only a continental grouping. Placement, stock and storage are
all per data center, so choose `dataCenterIds`, not a region. The GraphQL
region hint (for example `"US"`) has no REST v2 equivalent.
status: documented
evidence:
- { source: rest-v2-spec, ref: "DataCenterRegion; CreatePodRequest.dataCenterIds", seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpod-migrate/reference/graphql-to-v2.md }
- id: data-center.list-filters
applies_to: ["field:region", "field:compliance", "field:networkVolumeTypes", "field:globalNetwork"]
statement: >
The list filters combine with AND across filter types. Within a type,
regions match with OR, while `compliance` and `networkVolumeTypes` match
with AND, so `compliance=GDPR,SOC_2_TYPE_2` requires both.
status: documented
evidence:
- { source: rest-v2-spec, ref: GET /v2/catalog/datacenters parameters, seen: "2026-09-28" }
- id: data-center.compliance-filter
applies_to: ["field:compliance"]
statement: >
For data residency or certification requirements, filter data centers by
`compliance` and pass the matching ids as `dataCenterIds`.
status: documented
evidence:
- { source: rest-v2-spec, ref: DataCenter.compliance, seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpod-migrate/reference/unlocks.md }
- id: data-center.s3-api-per-dc
statement: >
The S3-compatible API for network volumes is available only in some data
centers, each with its own endpoint. REST v2 `DataCenter` has no field for
it, so check the S3 API docs before creating a volume you plan to reach
over S3.
status: documented
see: [network-volume.s3-api]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/storage/s3-api", note: See the data center availability section., seen: "2026-09-28" }
- { source: rest-v2-spec, ref: DataCenter, seen: "2026-09-28" }
concepts/endpoint-autoscaling.yaml
id: endpoint-autoscaling
name: Endpoint autoscaling
kind: capability
summary: >
How an endpoint adds and removes workers. A scaler watches queue delay or
in-flight requests and keeps the worker count between `workers.min` and
`workers.max`.
product: serverless
is_a: null
part_of: serverless-endpoint
aliases: [scaler, scale to zero, min workers, max workers, idle timeout, workersMin, workersMax, workersStandby]
surfaces:
mcp: [create-endpoint, update-endpoint]
runpodctl: [serverless create, serverless update]
console: Serverless > Edit endpoint
relations:
- { type: constrained_by, target: gpu-availability, cardinality: "1" }
- { type: constrained_by, target: credit-balance, cardinality: "1" }
rules:
- id: endpoint-autoscaling.scalers
statement: >
`QUEUE_DELAY` adds workers when jobs wait longer than `scaling.queueDelay`
seconds (minimum 0.5) and applies only to `QUEUE` endpoints.
`REQUEST_COUNT` targets `scaling.requestCount` in-flight requests per
worker and is required for `LOAD_BALANCER` endpoints. Both stay within
`workers.min` and `workers.max`, and the scaler can be switched on update.
status: documented
evidence:
- { source: rest-v2-spec, ref: "EndpointScaling; QueueDelayScaling; RequestCountScaling", seen: "2026-09-28" }
- id: endpoint-autoscaling.scaler-math
statement: >
The console's default queue delay is 4 seconds. `REQUEST_COUNT` aims for
`ceil((requests in queue + requests in progress) / requestCount)` workers,
so a value of 1 scales most aggressively.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/endpoint-configurations", note: Auto-scaling type. }
- id: endpoint-autoscaling.min-max
statement: >
`workers.min` 0 lets the endpoint scale to zero, with no compute charge
once the idle timeout ends. `workers.min` above 0 keeps that many workers
running and billing, even with no traffic. `workers.max` is a hard cap: on
a `QUEUE` endpoint extra jobs wait in the queue, while a load-balanced
endpoint has no queue.
status: documented
see: [serverless-billing.active-workers-bill-idle, serverless-endpoint.lb-no-queue, endpoint-autoscaling.idle-timeout]
evidence:
- { source: skill, path: plugins/runpod/skills/runpodctl/SKILL.md, note: Decision rules and serverless facts. }
- id: endpoint-autoscaling.active-worker-means-min
statement: >
An "active worker" is one kept warm by `workers.min`, not `workers.max`.
Set `workers.min` back to 0 after development.
status: documented
see: [serverless-billing.active-workers-bill-idle]
evidence:
- { source: skill, path: plugins/runpod/skills/runpodctl/SKILL.md, note: Active worker means minimum workers. }
- { source: public-docs, url: "https://docs.runpod.io/serverless/workers/overview", note: Worker types. }
- id: endpoint-autoscaling.idle-timeout
statement: >
`workers.idleTimeout` (1 to 3600 seconds) is how long a worker stays up
after its last request, and that time is billed. It is rejected on a
`QUEUE` endpoint that scales on `REQUEST_COUNT`. A short timeout makes
batch or evaluation workloads cold-start between requests, so raise it for
them.
on_violation: 422
status: documented
see: [serverless-endpoint.defaults-by-layer]
evidence:
- { source: rest-v2-spec, ref: EndpointWorkers.idleTimeout, seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpod-migrate/reference/breaking-changes.md, note: "Explains when `idleTimeout` is rejected." }
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/endpoint-configurations", note: Idle timeout is billed. }
- id: endpoint-autoscaling.account-worker-quota
statement: >
The total `workers.max` across all of your endpoints is capped by your
account balance. The cap starts at 5 and rises with balance; the public
docs list the tiers, and support can raise it further. The cap is checked
at create and whenever `workers.max` goes up.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/workers/overview", note: Max worker limits. }
- id: endpoint-autoscaling.gpu-priority-distribution
statement: >
An endpoint lists up to three GPU types in priority order. When the first
choice is unavailable, Runpod uses the next one, with no worker minimum.
With five or more workers, Runpod spreads workers across the listed types,
most on the primary; with fewer than five, every worker uses the
highest-priority type available.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/endpoint-configurations", seen: "2026-10-06", note: GPU priority and worker distribution. }
- id: endpoint-autoscaling.extra-workers
statement: >
During traffic spikes the platform can add extra workers on hosts that
already have the image cached.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/workers/overview", note: Worker types. }
- id: endpoint-autoscaling.idle-endpoint-scale-down
statement: >
An endpoint that receives no requests for 3 days has `workers.max` lowered
to 2, and Runpod emails the owner. After 7 days without requests,
`workers.max` is set to 0. It stays there until you raise it, and any
request resets the timer.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/endpoint-configurations", note: Idle endpoint scale-down. }
- { source: public-docs, url: "https://docs.runpod.io/serverless/troubleshooting", note: Prolonged inactivity. }
concepts/endpoint-build.yaml
id: endpoint-build
name: Endpoint build
kind: component
summary: >
A container image build that Runpod runs from a commit to a GitHub repository
connected to an endpoint. A completed build produces a release.
product: serverless
is_a: null
part_of: serverless-endpoint
aliases: [build, GitHub build, GitHub integration build]
surfaces:
rest_v2: { schema: Build, paths: ["/v2/serverless/{id}/builds", "/v2/serverless/{id}/builds/{buildId}"] }
mcp: [list-endpoint-builds, get-endpoint-build]
console: Serverless > Builds
fields:
- { name: id, type: string, set: read }
- { name: status, type: BuildState, set: read }
- { name: commitHash, type: string, set: read }
- { name: branch, type: string, set: read }
- { name: imageName, type: string, ref: container-image, set: read }
- { name: error, type: string, set: read, note: "set for `FAILED` and `TEST_FAILED`" }
- { name: startedAt, type: string, set: read }
- { name: completedAt, type: string, set: read }
states:
field: status
values:
PENDING: Queued and not started.
BUILDING: Building the image.
UPLOADING: Pushing the image.
TESTING: Running tests.
COMPLETED: Finished; the image was produced.
FAILED: The build failed.
CANCELLED: The build was cancelled.
TEST_FAILED: The tests failed.
evidence:
- { source: rest-v2-spec, ref: BuildState, seen: "2026-09-28" }
relations:
- { type: requires, target: container-image, cardinality: "1" }
rules:
- id: endpoint-build.terminal-states
applies_to: ["field:status", "field:error"]
statement: >
`COMPLETED`, `FAILED`, `CANCELLED` and `TEST_FAILED` are terminal.
`PENDING`, `BUILDING`, `UPLOADING` and `TESTING` mean the build is still
running. `error` explains a `FAILED` or `TEST_FAILED` build.
status: documented
evidence:
- { source: rest-v2-spec, ref: "BuildState; Build.error", seen: "2026-09-28" }
- id: endpoint-build.feeds-release
statement: >
A completed build produces a release with `source` set to `GIT_BUILD` and
its `buildId`. Workers move to it as they are replaced.
status: documented
see: [endpoint-release.source, endpoint-release.rollout]
evidence:
- { source: rest-v2-spec, ref: Release.buildId, seen: "2026-09-28" }
- id: endpoint-build.listing
statement: >
The build list is newest first, with a default limit of 100. Fetch an
older build directly by its id.
status: documented
evidence:
- { source: rest-v2-spec, ref: "GET /v2/serverless/{id}/builds; GET /v2/serverless/{id}/builds/{buildId}", seen: "2026-09-28" }
- id: endpoint-build.github-targets
statement: >
The managed GitHub integration builds workers for queue-based and
load-balanced Serverless endpoints. A connected GitHub account and App
control repository access, and the setup selects a branch and Dockerfile
path.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/workers/github-integration", seen: "2026-10-10" }
- { source: skill, path: plugins/runpod/skills/runpod-build-template/reference/github-integration.md, note: Documented behavior table. }
- id: endpoint-build.github-budgets
statement: >
The managed GitHub route limits the Docker build step to 30 minutes, the
total clone/build/upload/test process to 160 minutes, and image size to 80
GB. These are limits of that route; diagnose the failing phase before
changing model delivery or selecting another builder.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/workers/github-integration", seen: "2026-10-10" }
- { source: skill, path: plugins/runpod/skills/runpod-build-template/reference/github-integration.md, note: Documented behavior table. }
- id: endpoint-build.github-public-base
statement: >
The managed GitHub builder does not support private base images. Use an
authorized public base or an external builder and registry when the
project requires a private base.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/workers/github-integration", seen: "2026-10-10" }
- id: endpoint-build.github-build-gpu
statement: >
The managed GitHub builder does not provide GPU access during a build. A
build that requires a GPU needs another builder; device-independent CUDA
compilation does not itself require GPU access.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/workers/github-integration", seen: "2026-10-10" }
- { source: skill, path: plugins/runpod/skills/runpod-build-template/reference/github-integration.md, note: Documented behavior table. }
- id: endpoint-build.github-image-export
statement: >
Images produced by the managed GitHub integration cannot be pulled or
executed on other platforms. Use an external registry route when the
project needs a portable image or local testing of that exact artifact.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/workers/github-integration", seen: "2026-10-10" }
- id: endpoint-build.github-update-trigger
statement: >
Updating an existing GitHub integration requires a new GitHub release;
pushing a commit alone does not complete deployment. Verify the build,
release and worker image after publishing the release.
status: documented
see: [endpoint-build.feeds-release, endpoint-release.rollout]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/workers/github-integration", seen: "2026-10-10" }
- { source: skill, path: plugins/runpod/skills/runpod-build-template/reference/github-integration.md, note: Documented behavior table. }
- id: endpoint-build.github-console-rollback
statement: >
An endpoint deployed through GitHub integration can roll back to a
previous build from the console's Builds tab without a new GitHub release.
A previous build must exist in deployment history; deploying a later
GitHub release supersedes the rollback.
status: documented
see: [endpoint-build.github-update-trigger]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/workers/github-integration", seen: "2026-10-14", note: Roll back to a previous build. }
- id: endpoint-build.github-account-connection
statement: >
A Runpod account supports one GitHub connection, and connections are not
shared. Check the owning account and repository access before planning a
team or organization build.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/workers/github-integration", seen: "2026-10-10" }
concepts/endpoint-release.yaml
id: endpoint-release
name: Endpoint release
kind: component
summary: >
A versioned snapshot of an endpoint's configuration. Each change to the
endpoint's container creates one, and workers move to it as they are replaced.
product: serverless
is_a: null
part_of: serverless-endpoint
aliases: [release, rollout, rolling release, rolling update, endpoint version]
surfaces:
rest_v2: { schema: Release, paths: ["/v2/serverless/{id}/releases"] }
mcp: [list-endpoint-releases]
console: Serverless > Releases
fields:
- { name: id, type: string, set: read }
- { name: version, type: integer, set: read, note: endpoint configuration version this release produced }
- { name: source, type: enum, values: [GIT_BUILD, MANUAL], set: read }
- { name: buildId, type: string, ref: endpoint-build, set: read, note: "set when `source` is `GIT_BUILD`" }
- { name: workerCount, type: integer, set: read, note: workers running this version }
- { name: diff, type: "ReleaseDiffEntry[]", set: read, note: configuration fields that changed }
- { name: createdAt, type: string, set: read }
relations:
- { type: uses, target: endpoint-build, cardinality: "0..1" }
- { type: uses, target: template, cardinality: "0..1" }
rules:
- id: endpoint-release.rollout
statement: >
A container change (`image`, `env`, `args`, `disk`, `ports`, `registry`,
or an update to the bound template) creates a new release and starts a
rolling release. Idle workers on the old version are replaced right away;
busy workers finish their current job first, so the endpoint keeps serving
requests. Track progress with the rollout summary and each worker's
`isStale`.
status: documented
see: [worker.status-vocabulary]
evidence:
- { source: rest-v2-spec, ref: "PATCH /v2/serverless/{id}; GET /v2/serverless/{id}/releases; RolloutSummary", seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/rolling-releases", note: How idle and running workers are replaced. }
- id: endpoint-release.version-overlap
statement: >
During a rollout some workers run the old version and some the new one.
For strict version consistency, set `workers.max` to 0, wait for the
workers to stop, update the endpoint, then restore `workers.max`. Use
versioned image tags instead of `:latest`, because existing workers can
keep serving a cached `:latest` image.
status: documented
see: [serverless-endpoint.max-zero-stops-workers]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/rolling-releases", note: Best practices. }
- id: endpoint-release.source
applies_to: ["field:source", "field:buildId"]
statement: >
`source` is `GIT_BUILD` for a release produced by a GitHub build, with
`buildId` set, or `MANUAL` for a configuration change, with `buildId`
null.
status: documented
evidence:
- { source: rest-v2-spec, ref: "Release.source; Release.buildId; ReleaseSource", seen: "2026-09-28" }
- id: endpoint-release.rollout-summary-page-independent
statement: >
The release list is paginated newest first, with a default limit of 1000.
Its rollout summary always describes the endpoint's current state,
whichever page is requested.
status: documented
evidence:
- { source: rest-v2-spec, ref: "GET /v2/serverless/{id}/releases", seen: "2026-09-28" }
concepts/exposed-port.yaml
id: exposed-port
name: Exposed port
kind: component
summary: >
A container port opened to the outside. HTTP ports are served through the
Runpod proxy at `https://<pod-id>-<port>.proxy.runpod.net`; TCP ports get a
public IP and an external port. Ports that are not exposed cannot be reached
from outside the pod.
product: pods
is_a: null
part_of: pod
aliases: [ports, port mappings, proxy URL, HTTP proxy, TCP port, public IP]
surfaces:
rest_v2: { schema: Pod }
runpodctl: [pod create --ports, pod update --ports]
console: Pods (Connect menu)
fields:
- { name: ports, type: "string[]", set: "create,update", note: "Each entry is port/protocol, for example `8888/http` or `22/tcp`." }
- { name: runtime.ports, type: "PodRuntimePort[]", set: read, note: "The live mapping: `private`, `public`, `type` and `ip`." }
relations:
- { type: constrained_by, target: cloud-tier, cardinality: "1" }
rules:
- id: exposed-port.http-proxy
applies_to: ["field:ports"]
statement: >
An HTTP port N is served at `https://<pod-id>-<N>.proxy.runpod.net`, over
HTTPS only. The service must listen on `0.0.0.0`, not `localhost`. A pod
can expose at most 10 HTTP ports.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/configuration/expose-ports" }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/networking.md }
- id: exposed-port.proxy-100s-timeout
applies_to: ["field:ports"]
statement: >
Requests through the HTTP proxy time out after 100 seconds with a `524`
error. For longer work, return a job id and let the client poll, or use a
TCP port.
on_violation: 524
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/configuration/expose-ports", note: Proxy limitations and behavior. }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/gotchas.md }
- id: exposed-port.proxy-is-public
applies_to: ["field:ports"]
statement: >
A proxy URL is public and unauthenticated; a hard-to-guess pod id is not
access control. Tell the user when you hand out a proxy URL, and add
authentication inside the service if it handles anything sensitive.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/configuration/expose-ports", note: Public accessibility. }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/networking.md }
- id: exposed-port.mappings-appear-late
applies_to: ["field:runtime.ports"]
statement: >
`runtime.ports` shows a public IP and external port only once the pod is
running and the mapping is ready. A null `public` or `ip` right after
start is expected; read the pod again, or check the console's **Connect**
menu.
status: documented
evidence:
- { source: rest-v2-spec, ref: "PodRuntimePort.public; Pod.runtime", seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/gotchas.md }
- id: exposed-port.tcp-port-changes
applies_to: ["field:runtime.ports"]
statement: >
The external TCP port differs from the internal one and is reassigned on
every reset and on every stop and start, so read it again after each
restart.
status: documented
see: [cloud-tier.community-ip-changes]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/configuration/expose-ports", note: TCP access via public IP. }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/gotchas.md }
- id: exposed-port.symmetric-ports
applies_to: ["field:ports"]
statement: >
To get the same internal and external TCP port, request a port number
above 70000. The assigned port is available inside the pod as an
environment variable such as `RUNPOD_TCP_PORT_70000`.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/configuration/expose-ports", note: Symmetrical port mapping. }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/networking.md }
- id: exposed-port.proxy-502-not-ready
applies_to: ["field:ports"]
statement: >
A proxy `502` or `404` on an HTTP port usually means the service behind it
is still starting or has crashed, not that the pod is gone. Check the pod
logs and retry.
status: documented
see: [pod.running-is-not-ready]
evidence:
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/gotchas.md }
- { source: public-docs, url: "https://docs.runpod.io/pods/troubleshooting/troubleshooting-502-errors" }
- id: exposed-port.no-udp
applies_to: ["field:ports"]
statement: >
Only `http` and `tcp` ports work; pods do not support UDP. Some official
templates list `22/udp`, which has no effect.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/configuration/expose-ports", note: TCP access via public IP. }
- { source: skill, path: plugins/runpod/skills/runpod-templates/reference/pytorch.md, note: "The template's ports include 22/udp." }
concepts/file-transfer.yaml
id: file-transfer
name: File transfer
kind: capability
summary: >
Ways to move datasets into Runpod and results out: `runpodctl send` and
`receive`, `runpodctl network-volume upload` and `download`, SCP or rsync over
SSH to a pod, and the S3-compatible API on network volumes.
product: null
is_a: null
part_of: runpod-platform
aliases: [runpodctl send, runpodctl receive, network-volume upload, network-volume download, croc, S3 upload, s3api, cloud sync, datasets in, checkpoints out]
surfaces:
mcp: []
runpodctl: [send, receive, network-volume upload, network-volume download]
console: Pods (cloud sync), Settings (S3 API keys)
fields: []
relations:
- { type: uses, target: network-volume, cardinality: "0..1" }
- { type: uses, target: pod-ssh-access, cardinality: "0..1" }
rules:
- id: file-transfer.no-pod-file-api
statement: >
REST v2 and the Runpod MCP server have no route that moves files to or
from a pod. Use SCP or rsync over `ssh.direct`, `runpodctl send` and
`receive`, or the S3-compatible API on a network volume.
status: documented
see: [pod-ssh-access.proxy-vs-direct]
evidence:
- { source: rest-v2-spec, ref: paths under /v2, seen: "2026-09-28", note: No path uploads or downloads files. }
- { source: public-docs, url: "https://docs.runpod.io/pods/configuration/use-ssh" }
- id: file-transfer.send-receive
statement: >
`runpodctl send <path>` prints a connection code and waits; run `runpodctl
receive <code>` on the other machine. It needs no API key and no pod, and
works between any two machines. Each `send` generates a new code unless
you pass `--code`, so capture the first line of its output and check that
both sides exit 0. Transfers are already compressed, so there is no need
to archive first.
status: documented
evidence:
- { source: skill, path: plugins/runpod/skills/runpodctl/SKILL.md, note: File Transfer section. }
- { source: public-docs, url: "https://docs.runpod.io/runpodctl/reference/runpodctl-send" }
- { source: other, ref: runpodctl send --help (runpodctl 2.14.0), seen: "2026-09-28" }
- id: file-transfer.s3-api-per-data-center
statement: >
The S3-compatible API reaches a network volume without a running pod. Only
some data centers offer it, each at its own endpoint,
`https://s3api-<dc>.runpod.io/`. Pass that endpoint with `--endpoint-url`,
the data center id with `--region`, and the volume id as the bucket. S3
API keys are separate from Runpod API keys and are created in the console
under Settings.
status: documented
see: [network-volume.s3-api]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/storage/s3-api" }
- { source: skill, path: plugins/runpod/skills/runpod-companion-clis/SKILL.md }
- id: file-transfer.runpodctl-volume-transfer
statement: >
In runpodctl 3.1.0 and later, `runpodctl network-volume upload` and
`runpodctl network-volume download` move a file or a folder to or from a
network volume through its S3-compatible API, with no pod running. They
need S3 API keys in addition to the Runpod API key, which looks up the
volume's data center unless `--data-center-id` is passed. A transfer that
stops partway exits with the code `transfer_incomplete`, and running the
same command again resumes it, skipping files that are already done. A
full volume exits with `volume_full`: grow the volume, then run the
command again.
status: documented
see: [file-transfer.s3-api-per-data-center]
evidence:
- { source: other, url: "https://github.com/runpod/runpodctl/blob/main/README.md", seen: "2026-11-06", note: network volume files section and the error code table. }
- { source: skill, path: plugins/runpod/skills/runpodctl/reference/command-reference.md, note: Network volume upload and download (v3.1.0+). }
concepts/flash.yaml
id: flash
name: Flash
kind: tool
summary: >
Code-first serverless in the Runpod Python SDK (`runpod` 2.0). You define a
`runpod.App` with queue, API and task resources, iterate with `rp flash dev`
against remote workers, and ship with `rp flash deploy`. The older standalone
`runpod-flash` package is deprecated.
product: flash
is_a: null
part_of: runpod-platform
aliases: [Runpod Flash, runpod-flash, flash CLI, rp flash, Runpod SDK apps, Apps SDK]
relations:
- { type: runs_on, target: serverless-endpoint, cardinality: "1..n" }
- { type: uses, target: network-volume, cardinality: "0..n" }
- { type: uses, target: model-store, cardinality: "0..n" }
- { type: requires, target: api-key, cardinality: "1" }
rules:
- id: flash.part-of-runpod-sdk
statement: >
Flash ships in the `runpod` package from version 2.0, with the `rp` CLI
(`rp flash init`, `dev`, `deploy`, `app`, `env`, `undeploy`). The CLI is
also installed as `runpod`. Install `runpod>=2.0.0`.
status: verified
evidence:
- { source: public-docs, url: "https://docs.runpod.io/sdk/cli", seen: "2026-10-16" }
- { source: live-probe, ref: "pip install runpod==2.0.0rc1; rp --help and rp flash --help", seen: "2026-10-16" }
- id: flash.runpod-flash-deprecated
statement: >
The standalone `runpod-flash` package is deprecated and gets no fixes
after October 29, 2026. Endpoints it deployed keep running. Write new code
on the Runpod SDK and migrate existing projects.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/sdk/migrate-from-flash", seen: "2026-10-16" }
- { source: skill, path: plugins/runpod/skills/runpod-flash/reference/migrate-from-runpod-flash.md }
- id: flash.legacy-fallback
statement: >
`runpod-flash` still installs alongside the Runpod SDK: its CLI is
`flash`, separate from `rp`, and `runpod_flash` imports with `runpod` 2.x
installed. To fall back from the SDK, give it its own environment with the
project's requirements and `runpod<2`, so the SDK project keeps 2.x;
`flash dev` and `flash deploy` import the project's modules. Create that
environment outside the project directory so a deploy does not package it;
`rp flash deploy` skips only `.venv/`, `venv/` and `env/`. For cleanup
alone, `uv tool install runpod-flash --with "runpod<2"` is enough. Use it
when an SDK build, deploy or worker step fails in the SDK rather than in
your code. It gets no fixes after October 29, 2026.
status: verified
see: [flash.runpod-flash-deprecated]
evidence:
- { source: live-probe, ref: "one venv with runpod 2.0.0rc2 + runpod-flash 1.21.0: import runpod_flash ok, flash --help and rp --version both run; uv tool install runpod-flash --with 'runpod<2' resolved runpod 1.12.0 + runpod-flash 1.21.0 and flash --help exited 0; uv venv + uv pip install runpod-flash 'runpod<2' -r requirements.txt (numpy) gave runpod 1.12.0, numpy importable, flash --help ok", seen: "2026-10-22" }
- { source: public-docs, url: "https://docs.runpod.io/sdk/migrate-from-flash", seen: "2026-10-16" }
- { source: other, url: "https://github.com/runpod/runpod-python/blob/fc4a6c73c3915625d65df63416db835d0b927b8f/runpod/apps/deploy.py#L35", seen: "2026-10-23", note: "DEFAULT_IGNORES skips .venv/, venv/ and env/, not other virtualenv names." }
- id: flash.not-flashboot
statement: >
Flash is the code-first SDK and CLI. FlashBoot is a separate endpoint
setting for faster restarts; Flash exposes it through the `flashboot`
parameter but does not depend on it.
status: documented
see: [flashboot.flash-sdk-default-on]
evidence:
- { source: skill, path: plugins/runpod/skills/runpod-flash/SKILL.md }
- { source: skill, path: plugins/runpod/skills/runpod-flash/reference/api.md, note: The flashboot parameter of the decorators. }
- id: flash.resource-types
statement: >
An app has three resource types. `@app.queue` on a function creates a
queue-based endpoint. `@app.api` on a class creates a load-balanced
endpoint whose `@runpod.get` and `@runpod.post` methods are routes, with
setup in a `@runpod.init` method. `@app.task` on a function runs one Pod
per call and terminates it when the function returns. `gpu=` and `cpu=`
cannot be used together.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/sdk/overview", seen: "2026-10-16" }
- { source: other, url: "https://github.com/runpod/runpod-python/blob/v2.0.0rc1/runpod/apps/spec.py", seen: "2026-10-16", note: gpu and cpu are mutually exclusive. }
- id: flash.dev-session
statement: >
`rp flash dev FILE` creates temporary endpoints, runs the
`@runpod.local_entrypoint` on your machine, streams the workers' output
and runs again after each file change. Ending the session deletes the
endpoints. It runs no local HTTP server. `--once` runs the entrypoint
once, cleans up and exits non-zero on failure.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/sdk/apps/dev-sessions", seen: "2026-10-16" }
- { source: other, url: "https://github.com/runpod/runpod-python/blob/v2.0.0rc1/runpod/rp_cli/main.py", seen: "2026-10-16", note: With stdin closed the session re-runs on file changes only. }
- id: flash.same-module-in-dev-and-deploy
statement: >
Dev sessions and deploys run the whole module source on the worker, so
module-level imports, constants and helpers behave the same in both.
Deploy packages the project directory, but a dev session ships only the
module's source, so helpers in sibling local modules import only in
deploy. An app run with dev sessions lives in one file.
status: documented
evidence:
- { source: other, url: "https://github.com/runpod/runpod-python/blob/v2.0.0rc1/runpod/apps/serialization.py", seen: "2026-10-16", note: get_function_source returns the full module source. }
- { source: other, url: "https://github.com/runpod/runpod-python/blob/v2.0.0rc1/runpod/apps/deploy.py", seen: "2026-10-19", note: package_project walks the project directory into the artifact. }
- { source: public-docs, url: "https://docs.runpod.io/sdk/migrate-from-flash", seen: "2026-10-16", note: Known differences lists projects split across multiple local files. }
- id: flash.json-queue-arguments
statement: >
Queue job arguments are plain JSON, in a dev session as well as deployed,
so a non-JSON argument fails before the job is sent. A deployed queue
function receives the job's `input` object as keyword arguments.
status: documented
see: [job.input-envelope]
evidence:
- { source: other, url: "https://github.com/runpod/runpod-python/blob/v2.0.0rc1/runpod/apps/targets.py", seen: "2026-10-16", note: "require_json_args runs on live queue calls; deployed targets send {\"input\": kwargs}." }
- id: flash.api-route-body
statement: >
A POST route on an `@app.api` class receives the request's JSON body as
its one argument, so callers send the object itself with no wrapper key. A
GET route takes no argument.
status: documented
evidence:
- { source: other, url: "https://github.com/runpod/runpod-python/blob/v2.0.0rc1/examples/apps/web_service.py", seen: "2026-10-16", note: "Counter.post(\"/bump\", {\"by\": 3}) reaches bump(self, body) as body[\"by\"]." }
- id: flash.artifact-limits
statement: >
`rp flash deploy` packages an artifact of up to 1500 MB. On the built-in
GPU images it leaves out `torch`, `torchvision`, `torchaudio` and
`triton`. Dev sessions and tasks need local Python 3.10 to 3.14. Deploy
falls back to Python 3.12 with a warning on other local versions, and
`--python-version` picks the worker's Python for deploys.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/sdk/apps/deploy", seen: "2026-10-16" }
- { source: other, url: "https://github.com/runpod/runpod-python/blob/fc4a6c73c3915625d65df63416db835d0b927b8f/runpod/rp_cli/main.py#L190", seen: "2026-10-22", note: "On an unsupported local interpreter, deploy warns and builds for DEFAULT_PYTHON_VERSION." }
- { source: skill, path: plugins/runpod/skills/runpod-flash/reference/setup-and-cli.md }
- id: flash.sdk-defaults
statement: >
The SDK uses its own defaults: `workers` (0, 3), `idle_timeout` 60
seconds, `execution_timeout_ms` 0 (no limit), `flashboot` `True`, and
`container_disk_gb` 30 on GPU or 10 on CPU. They differ from the
platform's defaults, so set them explicitly when they matter.
status: documented
see: [serverless-endpoint.defaults-by-layer]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/sdk/configuration/parameters", seen: "2026-10-16" }
- { source: skill, path: plugins/runpod/skills/runpod-flash/reference/api.md, note: Parameters table. }
- id: flash.volume-mounts
statement: >
Queue and API resources mount one volume, at `/runpod-volume`. Tasks mount
up to one network volume and one global volume at distinct paths. Read
`volume.path` inside the remote function. Undeploying or ending a dev
session leaves the volumes, and they keep billing.
status: documented
see: [serverless-endpoint.network-volume-pins-dc]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/sdk/configuration/storage", seen: "2026-10-16" }
- id: flash.cli-mediated
statement: >
Deploy and tear down through the CLI rather than raw REST or GraphQL
calls: `rp flash deploy`, `rp flash undeploy --app APP --env ENV` for one
environment, and `rp flash app delete APP --yes` for every environment.
Without `--yes` the delete prompts for confirmation and aborts when there
is no terminal.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/sdk/cli", seen: "2026-10-16" }
- { source: skill, path: plugins/runpod/skills/runpod-flash/reference/setup-and-cli.md }
- { source: other, url: "https://github.com/runpod/runpod-python/blob/fc4a6c73c3915625d65df63416db835d0b927b8f/runpod/rp_cli/main.py#L718", seen: "2026-10-22", note: "app delete takes --yes/-y; otherwise click.confirm(..., abort=True)." }
- id: flash.resource-name-min-length
statement: >
`rp flash deploy` rejects queue and API resources whose name is shorter
than 3 characters. The name defaults to the function or class name, and
dev sessions do not check it, so a short name passes `rp flash dev` and
fails at deploy. Pass `name=` or rename the function or class.
status: documented
evidence:
- { source: other, url: "https://github.com/runpod/runpod-python/blob/fc4a6c73c3915625d65df63416db835d0b927b8f/runpod/apps/deploy.py#L133", seen: "2026-10-22", note: queue and api names must be at least 3 characters }
- id: flash.api-key-precedence
statement: >
A set `RUNPOD_API_KEY` overrides the key saved by `rp login` in
`~/.runpod/config.toml`, so a stale exported key causes 401 errors while
the saved login is valid.
status: documented
evidence:
- { source: other, url: "https://github.com/runpod/runpod-python/blob/v2.0.0rc1/runpod/apps/utils/network.py", seen: "2026-10-16", note: RUNPOD_API_KEY is read before the stored credentials. }
- { source: skill, path: plugins/runpod/skills/runpod-flash/SKILL.md, note: Gotcha 12. }
- id: flash.client-limits
statement: >
Payloads are limited to 10 MB, so pass URLs or write to a volume. Jobs get
no execution timeout by default, but `.remote()` stops waiting after 300 s
on a queue and raises `TimeoutError` while the job keeps running. For
longer calls, `.spawn()` and `job.result(timeout=...)`. The endpoint
client's `run_sync` takes a `timeout` in seconds.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/sdk/apps/dev-sessions", seen: "2026-10-16", note: remote() and job.wait() impose no execution timeout by default. }
- { source: other, url: "https://github.com/runpod/runpod-python/blob/v2.0.0rc1/runpod/apps/targets.py", seen: "2026-10-19", note: "DEFAULT_TIMEOUT_SECONDS = 300.0 bounds invoke() and job result waits; _wait_terminal raises TimeoutError at the deadline." }
- { source: skill, path: plugins/runpod/skills/runpod-flash/SKILL.md, note: Gotcha 7. }
- id: flash.torch-family-from-gpu-image
statement: >
From `runpod` 2.0.0, builds leave `torch`, `torchvision`, `torchaudio` and
`triton` out of the artifact whether a requirement lists them or a
dependency pulls them in (`peft`, `accelerate`, `sentence-transformers`),
together with the wheels only they need. GPU workers on the built-in
images use the image's `torch 2.9.1+cu128`, and version pins on these
packages are ignored. In an app that mixes CPU and GPU resources, a CPU
resource whose dependencies need torch installs it at cold start, so it
needs container disk for it.
status: verified
evidence:
- { source: live-probe, ref: "rp flash deploy of a gpu queue function with dependencies=[\"peft\"]: artifact 80.7 MB with no torch (645.6 MB with torch 2.14.1 before the fix); the worker returned torch 2.9.1+cu128 from site-packages with cuda True", seen: "2026-10-16" }
- { source: other, url: "https://github.com/runpod/runpod-python/blob/fc4a6c73c3915625d65df63416db835d0b927b8f/runpod/apps/build.py#L322", seen: "2026-10-22", note: Removes transitive torch-family wheels after vendoring and scopes the cold-start install list to CPU resources. }
- id: flash.gpu-device-names-pin-devices
statement: >
From `runpod` 2.0.0, GPU device names in `gpu=` pin those devices.
Endpoints select GPUs by pool, so the SDK selects each device's pool and
excludes the pool's other devices, reading membership from the live GPU
catalog. A pool id allows every device in the pool, including MIG slices.
status: verified
see: [gpu-pool.pin-a-card]
evidence:
- { source: live-probe, ref: "gpu=[\"NVIDIA GeForce RTX 4090\", \"NVIDIA L4\", \"NVIDIA RTX A5000\"] stored gpuIds ADA_24,AMPERE_24,-NVIDIA GeForce RTX 3090,-NVIDIA RTX PRO 6000 Blackwell Server Edition MIG 1g.24gb; four parallel jobs all reported NVIDIA GeForce RTX 4090", seen: "2026-10-16" }
- { source: live-probe, ref: "GET https://api.runpod.io/v2/catalog/gpus lists NVIDIA RTX PRO 6000 Blackwell Server Edition MIG 1g.24gb with pool AMPERE_24", seen: "2026-10-16" }
- id: flash.worker-api-key-every-endpoint
statement: >
`rp flash deploy` and `rp flash dev` add the account API key
(`RUNPOD_API_KEY`) to every endpoint's template env, including endpoints
whose code never calls another resource. Anyone who can read the
endpoint's template can read the key.
status: verified
evidence:
- { source: other, url: "https://github.com/runpod/runpod-python/blob/fc4a6c73c3915625d65df63416db835d0b927b8f/runpod/apps/deploy.py#L315", seen: "2026-10-22", note: "Deploy sets RUNPOD_API_KEY in every template env unless the resource's env already sets it." }
- { source: other, url: "https://github.com/runpod/runpod-python/blob/fc4a6c73c3915625d65df63416db835d0b927b8f/runpod/apps/dev.py#L146", seen: "2026-10-22", note: "Dev sessions add the client's key to every dev endpoint." }
- id: flash.dev-session-stop-signals
statement: >
From `runpod` 2.0.0, a dev session deletes its endpoints when it receives
SIGINT or SIGTERM, including a background session started with SIGINT
ignored. A second signal stops waiting for a running entrypoint. `kill -9`
skips the cleanup.
status: verified
see: [flash.dev-session]
evidence:
- { source: live-probe, ref: "rp flash dev main.py < /dev/null & from a script, then kill -TERM and separately kill -INT: both logged 'cleaning up 1 deleted' and left no endpoints", seen: "2026-10-16" }
- { source: other, url: "https://github.com/runpod/runpod-python/blob/fc4a6c73c3915625d65df63416db835d0b927b8f/runpod/rp_cli/main.py#L498", seen: "2026-10-22", note: rp flash dev handles SIGINT and SIGTERM and ignores further signals during teardown. }
- id: flash.dev-rerun-template-hold
statement: >
In `runpod` 2.1.0, a dev session deletes its endpoints on exit but not
their templates, and each template name comes from the app and resource
names. The platform keeps a deleted endpoint's template for up to about 90
minutes, so starting the same app again in that window fails with
"endpoint template names must be unique". Wait, or give the `runpod.App` a
new name.
status: verified
see: [flash.dev-session]
evidence:
- { source: live-probe, ref: "runpod 2.1.0: a second rp flash dev queue_gpu.py --once after a passing one failed with 'endpoint template names must be unique'; GraphQL deleteTemplate on the leftover dev-gpu-sum-gpu_sum-* template returned 'Template is still held by deleted endpoint ..., pending cleanup ... within about 90 minutes of the endpoint delete'; the same app under a new runpod.App name started normally", seen: "2026-11-04" }
- id: flash.legacy-dev-ships-body-only
statement: >
In the legacy `runpod-flash` package, `flash dev` ships only the decorated
function's body to the worker, so imports and helpers go inside it, while
`flash deploy` imports the whole module and can hide the mistake.
status: documented
evidence:
- { source: skill, path: plugins/runpod/skills/runpod-flash/reference/runpod-flash-legacy.md, note: Gotcha 1. }
- id: flash.legacy-defaults
statement: >
Legacy `runpod-flash` endpoints default to `workers` (0, 1), a 64 GB
container disk and `min_cuda_version` 12.8, because the `runpod/flash`
image needs CUDA 12.8; on older hosts that image crash-loops.
status: documented
evidence:
- { source: skill, path: plugins/runpod/skills/runpod-flash/reference/runpod-flash-legacy.md, note: Defaults section. }
concepts/flashboot.yaml
id: flashboot
name: FlashBoot
kind: capability
summary: >
An endpoint setting that keeps a worker's state on its host after the worker
scales down, so it can start again faster than after a fresh cold start.
product: serverless
is_a: null
part_of: serverless-endpoint
aliases: [FlashBoot, Priority FlashBoot, flashBootType]
surfaces:
mcp: [create-endpoint, update-endpoint]
runpodctl: [serverless create]
console: Serverless > Edit endpoint > FlashBoot
relations:
- { type: requires, target: worker, cardinality: "1" }
rules:
- id: flashboot.modes
statement: >
`flashboot` is `OFF`, `FLASHBOOT` or `PRIORITY_FLASHBOOT`, not a boolean.
REST v2 create defaults to `OFF`, while new endpoints made in the console
have FlashBoot on. An endpoint created over REST without it simply
cold-starts more slowly.
status: documented
evidence:
- { source: rest-v2-spec, ref: "FlashBoot; CreateEndpointRequest.flashboot", seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/endpoint-configurations", note: FlashBoot section. }
- { source: skill, path: plugins/runpod/skills/runpod-migrate/reference/breaking-changes.md, note: "Explains that `flashboot` changed from a boolean to an enum." }
- id: flashboot.best-effort
statement: >
FlashBoot revives a worker from state retained after it scaled down. When
no retained state is available, the worker cold-starts normally, so
FlashBoot shortens starts without guaranteeing them. It helps most on
endpoints with steady traffic, where workers often cycle between active
and idle.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/endpoint-configurations", note: FlashBoot section. }
- id: flashboot.flash-sdk-default-on
statement: >
The Flash SDK exposes FlashBoot as the `flashboot` parameter of
`@app.queue` and `@app.api` (and of the legacy `Endpoint`), and it
defaults to `True`. FlashBoot itself is an endpoint setting, not part of
Flash.
status: documented
see: [flash.not-flashboot]
evidence:
- { source: skill, path: plugins/runpod/skills/runpod-flash/reference/api.md, note: Parameters table. }
- { source: public-docs, url: "https://docs.runpod.io/sdk/configuration/parameters", seen: "2026-10-16" }
concepts/global-networking.yaml
id: global-networking
name: Global networking
kind: capability
summary: >
A private network between an account's pods. Each enabled pod gets a private
IP and an internal DNS name, `<podId>.runpod.internal`, so services can talk
without public ports.
product: pods
is_a: null
part_of: runpod-platform
aliases: [".runpod.internal", globalNetwork, private networking, pod-to-pod networking]
surfaces:
rest_v2: { schema: Pod }
mcp: [create-pod, update-pod, get-pod]
console: Pods (deploy options)
fields:
- { name: globalNetworking, type: boolean (write) / PodGlobalNetworking (read), set: "create,update" }
- { name: globalNetworking.ip, type: string, set: read, note: Present only when enabled. }
- { name: globalNetworking.internalDns, type: string, set: read, note: "`<podId>.runpod.internal`. Present only when enabled." }
relations:
- { type: constrained_by, target: data-center, cardinality: "1" }
- { type: constrained_by, target: gpu-type, cardinality: "1" }
rules:
- id: global-networking.internal-dns
applies_to: ["field:globalNetworking.ip", "field:globalNetworking.internalDns"]
statement: >
Reach another enabled pod at its `globalNetworking.internalDns`
(`<podId>.runpod.internal`) or `globalNetworking.ip`, including pods in
other data centers. Only pods in the same account can reach each other.
Bind services to `0.0.0.0`, and use this network instead of public ports
for internal services such as databases.
status: documented
see: [pod.global-networking-next-start, data-center.global-network]
evidence:
- { source: rest-v2-spec, ref: "PodGlobalNetworking; CreatePodRequest.globalNetworking", seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/pods/networking" }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/networking.md }
- id: global-networking.limits
applies_to: ["field:globalNetworking"]
statement: >
Global networking works on NVIDIA GPU pods only, in the data centers whose
`globalNetwork` is `true`, at about 100 Mbps between pods.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/networking" }
- { source: rest-v2-spec, ref: "CreatePodRequest.globalNetworking; DataCenter.globalNetwork", seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/networking.md }
concepts/global-volume.yaml
id: global-volume
name: Global volume
kind: resource
summary: >
Persistent, usage-billed storage that is not tied to a data center, so pods
and GPU Serverless endpoints in any data center can mount it. It suits
read-mostly files such as model weights.
product: storage
is_a: null
part_of: runpod-platform
aliases: [global storage, GlobalVolume, global volumes]
surfaces:
rest_v2: { schema: GlobalVolume, paths: [/v2/global-volumes, "/v2/global-volumes/{id}"] }
mcp: [create-global-volume, list-global-volumes, get-global-volume, update-global-volume, delete-global-volume]
console: Storage
fields:
- { name: id, type: string, set: read }
- { name: name, type: string, set: "create,update", required: true, note: not unique }
relations:
- { type: constrained_by, target: credit-balance, cardinality: "1" }
rules:
- id: global-volume.any-data-center
statement: >
A global volume has no data center. Compute in any supported data center
can mount it, so it does not narrow GPU choice the way a network volume
does.
status: documented
see: [network-volume.same-dc-only]
evidence:
- { source: rest-v2-spec, ref: POST /v2/global-volumes, seen: "2026-11-02" }
- { source: public-docs, url: "https://docs.runpod.io/storage/globalvolume/overview", seen: "2026-11-02" }
- id: global-volume.name-only-create
applies_to: ["field:name"]
statement: >
A create takes only a `name`; there is no size or data center to choose,
and storage grows with the data. An account holds a capped number of
global volumes, and a create past the cap is answered `400`.
on_violation: 400
status: verified
evidence:
- { source: rest-v2-spec, ref: CreateGlobalVolumeRequest.name, seen: "2026-11-02" }
- { source: rest-v2-spec, ref: POST /v2/global-volumes, seen: "2026-11-02" }
- { source: live-probe, ref: POST /v2/global-volumes with only name, seen: "2026-10-29" }
- id: global-volume.rename-only
applies_to: ["field:name"]
statement: >
An update can only change `name`; an empty update body is rejected.
status: documented
evidence:
- { source: rest-v2-spec, ref: UpdateGlobalVolumeRequest, seen: "2026-11-02" }
- id: global-volume.pod-mount
statement: >
A pod mounts at most one global volume, under `mounts.global`, on GPU and
CPU pods; CPU pods take one through the API only, since the console
attaches it to GPU pods only. It combines with a network volume but not
with a pod volume disk (`400`), and the mount is fixed at create: only its
`path` can change later.
on_violation: 400
status: verified
see: [pod.mount-kind-fixed]
evidence:
- { source: rest-v2-spec, ref: Mounts.global, seen: "2026-11-02" }
- { source: public-docs, url: "https://docs.runpod.io/storage/globalvolume/overview", note: "See \"Limitations\".", seen: "2026-11-02" }
- { source: live-probe, ref: "POST /v2/pods, CPU pod with mounts.global", note: "Mounted; a delete of the volume was then refused with 409.", seen: "2026-10-29" }
- { source: live-probe, ref: "POST /v2/pods, CPU pod with mounts.global", seen: "2026-11-02" }
- id: global-volume.pod-default-path
statement: >
A CPU pod created through REST v2 or MCP with a global mount, no `path`
and no network volume mounts it at `/workspace-global`; for a GPU pod, or
a pod that also mounts a network volume, the spec names no default. The
console's deploy flow mounts it at `/workspace` when no network volume is
attached. Read `mounts.global[0].path` back rather than assuming.
status: verified
evidence:
- { source: rest-v2-spec, ref: GlobalMount.path, seen: "2026-11-02" }
- { source: live-probe, ref: "POST /v2/pods with mounts.global and no path, then GET /v2/pods/{id}", note: "One CPU pod (cpu3c, 2 vCPU), no network volume, no path; the create response and the read both returned mounts.global[0].path /workspace-global.", seen: "2026-11-02" }
- { source: public-docs, url: "https://docs.runpod.io/storage/globalvolume/globalvolume-pods", note: "See \"Mount paths\" for the console deploy flow.", seen: "2026-11-02" }
- id: global-volume.endpoint-mount
statement: >
A Serverless endpoint mounts at most one global volume, under
`mounts.global`, on GPU endpoints only; a CPU endpoint is answered `400`.
Workers see it at `/runpod-volume`, and an endpoint carries network
volumes or a global volume, not both.
on_violation: 400
status: verified
see: [serverless-endpoint.volumes-via-mounts-network]
evidence:
- { source: rest-v2-spec, ref: EndpointMountsInput.global, seen: "2026-11-02" }
- { source: rest-v2-spec, ref: EndpointMount, seen: "2026-11-02" }
- { source: live-probe, ref: POST /v2/serverless on a CPU endpoint with mounts.global, note: Answered 400., seen: "2026-10-29" }
- id: global-volume.delete-blocked-while-mounted
statement: >
Deleting a global volume removes its stored data permanently. The delete
is refused with `409` while any pod, stopped pods included, or endpoint
still mounts it, and nothing is detached or terminated for you.
on_violation: 409
status: verified
evidence:
- { source: rest-v2-spec, ref: "DELETE /v2/global-volumes/{id}", seen: "2026-11-02" }
- { source: live-probe, ref: "DELETE /v2/global-volumes/{id} while mounted, then after the pod was deleted", note: "409 then 204.", seen: "2026-10-29" }
- id: global-volume.object-backed
statement: >
A global volume is object-backed: no file locking, atomic rename, hard
links or permission bits. A template that installs a Python environment
into `/workspace`, such as the official ComfyUI template, cannot use it as
its primary volume; mount it at a second path for model files.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/storage/globalvolume/overview", note: "See \"Limitations\".", seen: "2026-11-02" }
- id: global-volume.eventual-consistency
statement: >
Changes made to a global volume after a pod or worker starts may not be
visible inside it, and when two writers write the same file at once the
last write wins. Keep write-heavy work on a network volume.
status: documented
see: [network-volume.concurrent-writes]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/storage/globalvolume/overview", note: "See \"Limitations\".", seen: "2026-11-02" }
- { source: public-docs, url: "https://docs.runpod.io/storage/overview", seen: "2026-11-02" }
- id: global-volume.billing
statement: >
A global volume bills for stored data plus per-request fees, with Class A
requests (writes, creates, lists) priced above Class B requests (reads,
metadata lookups). State that shape before creating one.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/storage/globalvolume/overview", note: "See \"Request costs\".", seen: "2026-11-02" }
- id: global-volume.low-balance-deletion
statement: >
When the account balance reaches $0, a global volume is flagged and
permanently deleted after 15 days. Turn on low-balance notifications.
status: documented
see: [credit-balance.low-balance-actions]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/storage/globalvolume/overview", seen: "2026-11-02" }
concepts/gpu-availability.yaml
id: gpu-availability
name: GPU availability
kind: catalog
summary: >
How much stock a GPU type has in each data center for a given product (pod,
cluster or serverless), reported as `NONE`, `LOW`, `MEDIUM` or `HIGH`. CPU
flavor stock is reported the same way; see the CPU flavor concept.
product: null
is_a: null
part_of: runpod-platform
aliases: [stock, stockStatus, availability, capacity]
surfaces:
rest_v2: { schema: AvailabilityLevel, paths: [/v2/catalog/gpus, /v2/catalog/cpus, /v2/catalog/datacenters] }
graphql: { type: GpuAvailability, queries: [gpuTypes] }
mcp: [get-capacity, list-gpu-types, list-data-centers]
runpodctl: [gpu list, datacenter list]
fields: []
relations:
- { type: requires, target: gpu-type, cardinality: "1" }
- { type: requires, target: data-center, cardinality: "1" }
rules:
- id: gpu-availability.per-type-per-dc-per-product
statement: >
Stock is reported per GPU type per data center, and it differs by product
(`POD`, `CLUSTER` or `SERVERLESS`): the same GPU type can be scarce for
pods and plentiful for serverless. Check stock for the product you are
about to create, in the data center you intend to use.
status: documented
see: [gpu-type.availability-needs-product]
evidence:
- { source: rest-v2-spec, ref: "GET /v2/catalog/gpus product; GpuType.dataCenters; DataCenter.gpuAvailability", seen: "2026-09-28" }
- id: gpu-availability.levels-and-case
statement: >
REST v2 levels are `NONE`, `LOW`, `MEDIUM` and `HIGH`, in upper case.
GraphQL's `stockStatus` uses `High`, `Medium`, `Low` and `None`, so string
comparisons carried over from GraphQL stop matching without any error.
status: documented
evidence:
- { source: rest-v2-spec, ref: AvailabilityLevel, seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpod-migrate/reference/breaking-changes.md }
- id: gpu-availability.cuda-filter
statement: >
The GPU catalog accepts `cudaVersions` or `minCudaVersion` together with
`include=AVAILABILITY`. Use them to see which CUDA versions have stock
before creating a pod with a CUDA constraint.
status: documented
see: [gpu-type.cuda-versions-with-capacity]
evidence:
- { source: rest-v2-spec, ref: "GET /v2/catalog/gpus cudaVersions, minCudaVersion", seen: "2026-09-28" }
- id: gpu-availability.cli-best-status
statement: >
In `runpodctl gpu list`, the top-level `stockStatus` is the best status
across all data centers, and GPUs with no stock are hidden by default.
When a pod must run in a specific data center, read the per-data-center
breakdown and pass `--include-unavailable`.
status: documented
evidence:
- { source: skill, path: plugins/runpod/skills/runpodctl/SKILL.md }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/gpu-selection.md }
concepts/gpu-pool.yaml
id: gpu-pool
name: GPU pool
kind: catalog
summary: >
A serverless GPU tier, such as `ADA_24`, that groups GPU types with similar
memory. Endpoints select pools rather than individual GPU types, and
serverless prices are set per pool.
product: serverless
is_a: null
part_of: runpod-platform
aliases: [GPU group, GPU tier, pool id, GpuGroup, AMPERE_24, ADA_24, ADA_80_PRO]
surfaces:
rest_v2: { paths: [/v2/catalog/gpus] }
mcp: [list-gpu-types, create-endpoint, set-endpoint-gpus]
runpodctl: [gpu list]
fields:
- { name: id, type: string, set: read, note: "pool id, read from `GpuType.pool` in `GET /v2/catalog/gpus`" }
relations:
- { type: constrained_by, target: gpu-availability, cardinality: "1" }
rules:
- id: gpu-pool.pool-vs-type-ids
applies_to: ["field:id"]
statement: >
Pool ids (`ADA_24`) and GPU type ids (`NVIDIA GeForce RTX 4090`) are
different. Read a GPU type's pool from the `pool` field of `GET
/v2/catalog/gpus`, where null means the type is in no serverless pool.
Endpoints take pools; pods take GPU types.
status: documented
evidence:
- { source: rest-v2-spec, ref: "GpuType.pool; BaseEndpointGpuConfig.pools", seen: "2026-09-28" }
- id: gpu-pool.pricing
statement: >
The serverless list price per GPU per hour comes from the pool
(`GpuType.price.serverless`). Multiply it by `gpu.count` for the
per-worker rate. Negotiated discounts are not shown, and the price is
absent when the pool is not available to the caller.
status: documented
evidence:
- { source: rest-v2-spec, ref: GpuType.price.serverless, seen: "2026-09-28" }
- id: gpu-pool.pin-a-card
statement: >
To run on one specific card, select the pool that contains it and list the
pool's other GPU types in `gpu.excludedTypes`. At least one type must
remain.
status: documented
see: [serverless-endpoint.pools-not-types]
evidence:
- { source: rest-v2-spec, ref: POST /v2/serverless (Checking what you can deploy), seen: "2026-09-28" }
concepts/gpu-type.yaml
id: gpu-type
name: GPU type
kind: catalog
summary: >
A GPU hardware model, such as NVIDIA RTX A6000, with its VRAM, the clouds that
offer it, its list prices, and the most GPUs one pod can use.
product: null
is_a: null
part_of: runpod-platform
aliases: [GPU, gpuTypeId, GpuType, GPU model]
surfaces:
rest_v2: { schema: GpuType, paths: [/v2/catalog/gpus, "/v2/catalog/gpus/{id}"] }
graphql: { type: GpuType, queries: [gpuTypes] }
mcp: [list-gpu-types, get-gpu-type]
runpodctl: [gpu list]
fields:
- { name: id, type: string, set: read, note: "Pass as `gpu.id` on pod create." }
- { name: name, type: string, set: read }
- { name: pool, type: string, ref: gpu-pool, set: read, note: "Pass to serverless endpoints; null when the GPU is not in a serverless pool." }
- { name: manufacturer, type: enum, values: [NVIDIA, AMD, UNKNOWN], set: read }
- { name: memory, type: integer, unit: GB, set: read, note: VRAM per GPU. }
- { name: secure, type: boolean, set: read, note: Offered on Secure Cloud. }
- { name: community, type: boolean, set: read, note: Offered on Community Cloud. }
- { name: price.secure, type: number, unit: USD per GPU per hour, set: read }
- { name: price.community, type: number, unit: USD per GPU per hour, set: read }
- { name: price.serverless, type: number, unit: USD per GPU per hour, set: read }
- { name: maxCount.secure, type: integer, set: read }
- { name: maxCount.community, type: integer, set: read }
- { name: availability, type: AvailabilityLevel, set: read, note: "Returned only with `include=AVAILABILITY` and `product`." }
- { name: dataCenters, type: "DataCenterAvailability[]", set: read, note: "Returned only with `include=AVAILABILITY` and `product`." }
- { name: cudaVersions, type: "CudaVersionAvailability[]", set: read, note: "Returned only with `include=AVAILABILITY`." }
relations:
- { type: uses, target: gpu-pool, cardinality: "0..1" }
- { type: constrained_by, target: gpu-availability, cardinality: "1" }
- { type: constrained_by, target: cloud-tier, cardinality: "1..n" }
rules:
- id: gpu-type.price-is-list-per-gpu
applies_to: ["field:price.secure", "field:price.community", "field:price.serverless"]
statement: >
`price` is the hourly list price for one GPU, quoted per cloud for pods
and as the pool rate for serverless. Multiply it by `gpu.count` for the
rate of a pod or worker. The rate actually billed for a pod is the pod's
own `cost` field.
status: documented
evidence:
- { source: rest-v2-spec, ref: GpuType.price, seen: "2026-09-28" }
- id: gpu-type.max-count-not-stock
applies_to: ["field:maxCount.secure", "field:maxCount.community"]
statement: >
`maxCount` is the most GPUs one pod of this type can have in that cloud,
which is the GPU count of the largest machine. It is a ceiling, not stock.
Read `availability` for what is free.
status: documented
see: [gpu-type.availability-needs-product]
evidence:
- { source: rest-v2-spec, ref: GpuType.maxCount, seen: "2026-09-28" }
- id: gpu-type.availability-needs-product
applies_to: ["field:availability", "field:dataCenters", "field:cudaVersions"]
statement: >
Availability is returned only with `include=AVAILABILITY`, which requires
`product` (`POD`, `CLUSTER` or `SERVERLESS`). Sending either parameter
without the other returns 400. A missing `dataCenters` list means the
requested configuration is unavailable everywhere.
on_violation: 400
status: documented
evidence:
- { source: rest-v2-spec, ref: "GET /v2/catalog/gpus parameters; GpuType.dataCenters", seen: "2026-09-28" }
- id: gpu-type.cuda-versions-with-capacity
applies_to: ["field:cudaVersions"]
statement: >
`cudaVersions` lists the CUDA versions that have free capacity for a GPU
type, because machines of the same type can run different drivers. A pod
pinned to a version marked `available: false` fails on capacity. A GPU
type whose machines report no CUDA version, such as an AMD GPU, has no
`cudaVersions` at all.
status: documented
see: [machine.cuda-is-per-machine, gpu-pool.pool-vs-type-ids]
evidence:
- { source: rest-v2-spec, ref: "GpuType.cudaVersions; CudaVersionAvailability", seen: "2026-09-28" }
concepts/hub-repo.yaml
id: hub-repo
name: Hub repo
kind: catalog
summary: >
A GitHub repository listed on the Runpod Hub, configured by `hub.json` and a
`Dockerfile`. Each GitHub release is built into an image that you can deploy
in one click.
product: hub
is_a: null
part_of: runpod-platform
aliases: [Hub, Runpod Hub, hub listing, hub release, hubReleaseId, hub.json]
surfaces:
mcp: [list-hub-repos, deploy-hub-repo]
runpodctl: [hub list, hub search, hub get, serverless create --hub-id]
console: Hub
relations:
- { type: requires, target: container-image, cardinality: "1" }
rules:
- id: hub-repo.releases-not-commits
statement: >
The Hub indexes GitHub releases, not commits. Creating a new release
rebuilds and retests the listing. A new repo shows as Pending while it
builds and is tested, then Runpod reviews it before it is published.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/hub/publishing-guide", seen: "2026-09-28" }
- id: hub-repo.hub-json-config
statement: >
`.runpod/hub.json` sets the listing's metadata and deployment defaults:
`runsOn` (`GPU` or `CPU`), `gpuIds` (pool ids), `gpuCount`,
`containerDiskInGb`, `allowedCudaVersions`, `env` and `presets`. Endpoint
settings such as timeouts and worker counts are set on the endpoint after
it is deployed.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/hub/publishing-guide", note: "See the `hub.json` reference.", seen: "2026-09-28" }
- id: hub-repo.deploy-through-hub
statement: >
Deploy a Hub repo with `deploy-hub-repo` or `runpodctl serverless create
--hub-id <id>`. The Hub path resolves the listing, takes its built image
and config (GPU ids, container disk, env), and creates the endpoint from
them. On that path `--gpu-id` takes a GPU name such as `NVIDIA GeForce RTX
4090` and stores the matching pool (`ADA_24`) in runpodctl 2.14.0; older
builds needed the pool id itself. Omit `--gpu-id` to use the listing's GPU
config.
status: verified
conflict: false
evidence:
- { source: skill, path: plugins/runpod/skills/runpodctl/SKILL.md, note: "See \"Create from hub\"." }
- { source: skill, path: plugins/runpod/skills/runpodctl/evals/hub-deploy-serverless.eval.md }
- { source: skill, path: plugins/runpod/skills/runpod/golden-paths/20-model-caching-endpoint.md, note: "Passes --gpu-id \"NVIDIA GeForce RTX 4090\" with --hub-id." }
- { source: live-probe, ref: "runpodctl 2.14.0: serverless create --hub-id <vLLM listing> --gpu-id \"NVIDIA GeForce RTX 4090\" created the endpoint with gpu.pools [ADA_24]", seen: "2026-09-28" }
- id: hub-repo.creator-revenue
statement: >
Maintainers of published repos earn a share of the compute that users run
from their listings, paid to their Runpod credit balance. The share is
tiered by monthly compute hours, up to 7%.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/hub/revenue-sharing", seen: "2026-09-28" }
- id: hub-repo.authoring-files
statement: >
A Hub project includes `.runpod/hub.json`, `.runpod/tests.json`, worker
code, a Dockerfile and a README. The metadata and deployment defaults must
agree with the worker's actual input, model, hardware and storage
requirements.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/hub/publishing-guide", seen: "2026-10-10" }
- { source: skill, path: plugins/runpod/skills/runpod-build-template/reference/hub-publishing.md, note: Author the listing without a user-supplied checklist. }
- id: hub-repo.runpod-directory-precedence
statement: >
Common project files under `.runpod` take precedence over their root
copies for Hub builds. Inspect both locations so the listing builds the
intended code and context.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/hub/publishing-guide", seen: "2026-10-10" }
- id: hub-repo.test-input-contract
statement: >
The `input` in `.runpod/tests.json` is the raw job payload, without a
second Runpod submission envelope. Configure a realistic timeout and test
hardware, and retain a semantic workload check because the guide's HTTP
200 criterion alone does not establish useful output.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/hub/publishing-guide", seen: "2026-10-10" }
- { source: skill, path: plugins/runpod/skills/runpod-build-template/reference/hub-publishing.md, note: Author the listing without a user-supplied checklist. }
concepts/instant-cluster.yaml
id: instant-cluster
name: Instant Cluster
kind: resource
summary: >
A group of identical GPU pods in one data center, connected by a private
high-bandwidth network and set up for distributed training, Slurm, Ray or
other multi-node workloads.
product: clusters
is_a: null
part_of: runpod-platform
aliases: [cluster, Instant Clusters, cluster node, Slurm cluster, Ray cluster]
surfaces:
rest_v2: { schema: Cluster, paths: [/v2/clusters, "/v2/clusters/{id}", "/v2/clusters/{id}/pods"] }
graphql: { mutations: [createCluster, deleteCluster] }
mcp: [create-cluster, list-clusters, get-cluster, update-cluster, delete-cluster, list-cluster-pods]
runpodctl: []
console: Instant Clusters
fields:
- { name: name, type: string, set: "create,update", required: true }
- { name: type, type: enum, values: [APPLICATION, TRAINING, SLURM, RAY], set: create, required: true }
- { name: compute.gpuTypeId, type: string, ref: gpu-type, set: create, required: true }
- { name: compute.gpuCountPerPod, type: integer, set: create, required: true, note: "1 to the GPU type's `maxCount`." }
- { name: compute.podCount, type: integer, set: create, required: true, note: "2 to 250." }
- { name: templateId, type: string, ref: template, set: create }
- { name: dataCenterIds, type: "string[]", ref: data-center, set: create, note: Placement preference. }
- { name: dataCenterId, type: string, ref: data-center, set: read }
- { name: "mounts.network[].volumeId", type: string, ref: network-volume, set: create }
- { name: startSsh, type: boolean, set: create }
- { name: pods, type: ClusterPodsSummary, set: read, note: "Counts only. List members with `GET /v2/clusters/{id}/pods`." }
- { name: primary, type: ClusterPrimary, set: read }
- { name: network, type: ClusterNetwork, set: read }
relations:
- { type: runs_on, target: pod, cardinality: "1..n" }
- { type: runs_on, target: gpu-type, cardinality: "1" }
- { type: located_in, target: data-center, cardinality: "1" }
- { type: uses, target: network-volume, cardinality: "0..1" }
- { type: uses, target: template, cardinality: "0..1" }
- { type: billed_by, target: billing-record, cardinality: "1" }
rules:
- id: instant-cluster.fixed-at-create
applies_to: ["field:type", "field:compute.gpuTypeId", "field:compute.gpuCountPerPod", "field:compute.podCount"]
statement: >
`PATCH /v2/clusters/{id}` only renames a cluster. The type, GPU type, GPUs
per pod and container settings are fixed at creation; to change them,
delete the cluster and create a new one.
status: documented
evidence:
- { source: rest-v2-spec, ref: "PATCH /v2/clusters/{id}; UpdateClusterRequest", seen: "2026-09-28" }
- id: instant-cluster.scale-in-console
applies_to: ["field:compute.podCount"]
statement: >
Add pods to a running cluster from the console with **Scale cluster**; new
pods use the same GPU type, template and network storage. REST v2 has no
route that adds pods. To shrink a cluster, terminate the pods you no
longer need from the console. Clusters on reserved hardware cannot be
scaled this way.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/instant-clusters/scale-clusters" }
- { source: rest-v2-spec, ref: paths under /v2/clusters, seen: "2026-09-28", note: No path adds pods to an existing cluster. }
- id: instant-cluster.homogeneous-single-dc
applies_to: ["field:compute.podCount", "field:compute.gpuCountPerPod", "field:dataCenterIds"]
statement: >
Every member pod is identical: `podCount` pods (2 to 250), each with
`gpuCountPerPod` GPUs of `gpuTypeId`, up to the GPU type's `maxCount`. The
whole cluster is placed in one data center.
status: documented
evidence:
- { source: rest-v2-spec, ref: "ClusterCompute; CreateClusterRequest.dataCenterIds", seen: "2026-09-28" }
- id: instant-cluster.private-image-via-template
applies_to: ["field:templateId"]
statement: >
A template is the only way to run a private image on a cluster; the create
body has no registry field. `templateId` cannot be combined with `image`,
`args`, `entrypoint`, `cmd`, `disk`, `env`, `ports` or `mounts`, and it
must name a pod template, not a serverless one.
on_violation: 400
status: documented
evidence:
- { source: rest-v2-spec, ref: "CreateClusterRequest; CreateClusterRequest.templateId", seen: "2026-09-28" }
- id: instant-cluster.members-managed-by-cluster
applies_to: ["field:pods"]
statement: >
Member pods are left out of `GET /v2/pods` unless you pass
`includeClusterPods=true`, and the pod endpoints cannot change or delete
them. `DELETE /v2/clusters/{id}` terminates every member pod.
status: documented
evidence:
- { source: rest-v2-spec, ref: "Pod.cluster; DELETE /v2/clusters/{id}", seen: "2026-09-28" }
- id: instant-cluster.primary-node
applies_to: ["field:primary"]
statement: >
The primary node has `NODE_RANK` 0 and is the Slurm controller on Slurm
clusters. Every member pod gets `NODE_RANK`, `PRIMARY_ADDR` and
`PRIMARY_PORT` (also exposed as `MASTER_ADDR` and `MASTER_PORT`).
`primary.sshEndpoint` appears only once the primary pod is `RUNNING` with
`22/tcp` exposed.
status: documented
evidence:
- { source: rest-v2-spec, ref: "PodCluster.rank; ClusterPrimary.sshEndpoint", seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/instant-clusters/configuration" }
- { source: public-docs, url: "https://docs.runpod.io/instant-clusters/slurm-clusters" }
- id: instant-cluster.nccl-interface
statement: >
Set `NCCL_SOCKET_IFNAME=ens1` so NCCL uses the cluster's high-bandwidth
interfaces (`ens1` to `ens8`). Traffic over the external `eth0` interface
makes distributed training time out or hang.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/instant-clusters/configuration" }
- { source: public-docs, url: "https://docs.runpod.io/instant-clusters/pytorch" }
- id: instant-cluster.shared-volume-same-dc
applies_to: ["field:mounts.network[].volumeId"]
statement: >
A network volume shared by the cluster must be in the cluster's data
center.
status: documented
see: [pod.network-volume-same-dc]
evidence:
- { source: rest-v2-spec, ref: NetworkMount.volumeId, seen: "2026-09-28" }
concepts/job.yaml
id: job
name: Serverless job
kind: component
summary: >
One request submitted to a queue-based endpoint. It waits in the queue, runs
on one worker, and its result is kept for a short time after it finishes.
product: serverless
is_a: null
part_of: serverless-endpoint
aliases: [request, request ID, job ID]
surfaces:
mcp: [run-endpoint, runsync-endpoint, get-job-status, stream-job, cancel-job, retry-job, purge-endpoint-queue]
runpodctl: [serverless run, serverless status]
console: Serverless > Requests
fields:
- { name: id, type: string, set: read, note: "returned by `/run` or `/runsync`" }
- { name: input, type: object, set: create, required: true }
- { name: status, type: enum, values: [IN_QUEUE, IN_PROGRESS, COMPLETED, FAILED, CANCELLED, TIMED_OUT], set: read }
- { name: policy.executionTimeout, type: integer, unit: ms, default: 600000, set: create, note: "5 seconds to 7 days" }
- { name: policy.ttl, type: integer, unit: ms, default: 86400000, set: create, note: "10 seconds to 7 days" }
- { name: policy.lowPriority, type: boolean, default: false, set: create }
- { name: webhook, type: string, set: create }
- { name: output, type: object, set: read }
states:
field: status
values:
IN_QUEUE: Waiting for a worker.
IN_PROGRESS: Picked up by a worker and running.
COMPLETED: Finished and returned a result.
FAILED: The handler raised an error.
CANCELLED: Cancelled through `/cancel`.
TIMED_OUT: Expired before a worker picked it up, or the worker did not report back in time.
transitions:
- { action: cancel, from: [IN_QUEUE, IN_PROGRESS], to: CANCELLED }
- { action: retry, from: [FAILED, TIMED_OUT], to: IN_QUEUE }
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/job-states" }
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/operation-reference", note: The /cancel and /retry operations. }
- { source: rest-v2-spec, ref: "EndpointRequestUrls.cancel; EndpointRequestUrls.retry", seen: "2026-09-28" }
relations:
- { type: runs_on, target: worker, cardinality: "0..1" }
- { type: billed_by, target: serverless-billing, cardinality: "1" }
rules:
- id: job.status-vocabulary
applies_to: ["field:status"]
statement: >
The job API reports `IN_QUEUE`, then `IN_PROGRESS`, and ends as
`COMPLETED`, `FAILED`, `CANCELLED` or `TIMED_OUT`.
status: documented
evidence:
- { source: skill, path: plugins/runpod/skills/runpod/golden-paths/12-serverless-streaming.md, note: Observed jobs report IN_PROGRESS. }
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/job-states", seen: "2026-10-06", note: Lists exactly these six states. }
- id: job.run-vs-runsync
statement: >
`/run` returns a job id in `IN_QUEUE` right away and keeps the result for
30 minutes. `/runsync` waits up to 90 seconds by default (adjustable with
`?wait=` from 1000 to 300000 ms), returns while the job keeps running if
it takes longer, and keeps the result for 1 minute. Use `/run` and poll
`/status` for the first call to a cold endpoint.
status: documented
see: [api-rate-limit.serverless-job-routes]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/operation-reference", note: The /runsync operation. }
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/send-requests", note: Result retention. }
- { source: rest-v2-spec, ref: EndpointRequestUrls.runSync, seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpodctl/reference/command-reference.md }
- id: job.input-envelope
applies_to: ["field:input"]
statement: >
The request body is always `{"input": {...}}`, and `input` must not be
empty. `/run` accepts payloads up to 10 MB and `/runsync` up to 20 MB, so
pass large inputs as URLs.
status: documented
evidence:
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/endpoint-workflows.md }
- { source: skill, path: plugins/runpod/skills/runpod-flash/reference/api.md, note: Raw HTTP to a deployed queue resource. }
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/send-requests" }
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/operation-reference", seen: "2026-10-14", note: Maximum payload sizes for /run and /runsync. }
- id: job.cancel
applies_to: ["action:cancel"]
statement: >
`/cancel` stops an in-progress job or removes a queued one. A job
cancelled before a worker picks it up is not billed.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/operation-reference", note: The /cancel operation. }
- id: job.ttl-and-timeout
applies_to: ["field:policy.ttl", "field:policy.executionTimeout"]
statement: >
`ttl` (default 24 hours, maximum 7 days) starts at submission and covers
both queue and run time. When it expires, the job is deleted in any state
and `/status` returns 404. `executionTimeout` limits only run time, so set
`ttl` to at least the expected queue time plus `executionTimeout`.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/send-requests", note: Execution policies. }
- id: job.low-priority
applies_to: ["field:policy.lowPriority"]
statement: >
A job submitted with `policy.lowPriority` set to true does not trigger
worker scaling.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/send-requests", note: Execution policies. }
- id: job.streaming-and-webhooks
applies_to: ["field:webhook"]
statement: >
`/stream` works only when the handler is a generator, and a client
disconnect does not stop the worker. A `webhook` receives the result when
the job reaches a terminal state. It must return 200; otherwise Runpod
retries up to 2 more times, 10 seconds apart.
status: documented
evidence:
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/endpoint-workflows.md, note: Streaming section. }
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/send-requests", note: Webhook notifications. }
- id: job.api-host
statement: >
Jobs are not part of the REST v2 control API. They go to the serverless
API host (`https://api.runpod.ai/v2/<endpoint-id>/run` and similar). Read
the exact URLs from the endpoint's `requestUrls` instead of building them.
status: documented
evidence:
- { source: rest-v2-spec, ref: EndpointRequestUrls, seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpod-migrate/reference/rest-v1-to-v2.md, note: The requestUrls field. }
- id: job.purge-queue
statement: >
`/purge-queue` removes every pending job and leaves in-progress jobs
running.
status: documented
evidence:
- { source: rest-v2-spec, ref: EndpointRequestUrls.purgeQueue, seen: "2026-09-28" }
concepts/log-stream.yaml
id: log-stream
name: Log stream (pod and worker logs)
kind: capability
summary: >
A pod's or serverless worker's container output and system events, streamed
live with a short backfill of recent lines.
product: null
is_a: null
part_of: runpod-platform
aliases: [pod logs, worker logs, container logs, system logs, SSE log stream]
surfaces:
rest_v2: { paths: ["/v2/pods/{id}/logs", "/v2/serverless/{id}/workers/{workerId}/logs"] }
mcp: [stream-pod-logs, stream-worker-logs]
runpodctl: [pod logs, serverless logs]
console: Pods and Serverless (Logs tab)
fields: []
relations:
- { type: uses, target: pod, cardinality: "0..1" }
- { type: uses, target: worker, cardinality: "0..1" }
rules:
- id: log-stream.sse-parameters
statement: >
Logs stream as Server-Sent Events of `{source, line, ts}`. `source` is
`container` or `system`; omit it to get both. `tail` backfills recent
lines (default 100, maximum 5000, `0` for live only), and `since` takes an
RFC 3339 time. `Last-Event-ID` resumes a dropped stream and overrides
`since` and `tail`.
status: documented
evidence:
- { source: rest-v2-spec, ref: "GET /v2/pods/{id}/logs; LogSource", seen: "2026-09-28" }
- id: log-stream.system-source-for-stalls
statement: >
The `system` source reports pod lifecycle events: image pull, container
create and container start. A stalled deploy usually shows there as
repeated pull progress or a create that never reaches start, so read it
first when a pod stays in `STARTING`.
status: documented
evidence:
- { source: rest-v2-spec, ref: LogSource, seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/pods/manage-pods" }
- { source: other, ref: runpodctl pod logs --help (runpodctl 2.14.0), seen: "2026-09-28" }
- id: log-stream.worker-log-retention
statement: >
Worker logs live only on the worker's machine and are removed when the
worker terminates. Endpoint logs are kept for 90 days. To keep logs
longer, write them to a network volume or an external logging service.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/development/logs" }
- { source: public-docs, url: "https://docs.runpod.io/serverless/development/write-logs" }
concepts/machine.yaml
id: machine
name: Machine
kind: infrastructure
summary: >
A physical server in a data center that hosts pods and workers. A pod is
assigned to one machine when it is first deployed and stays tied to it, even
while it is stopped.
product: null
is_a: null
part_of: data-center
aliases: [host, host machine, physical machine, server]
relations: []
rules:
- id: machine.pod-is-bound
statement: >
A pod runs on exactly one machine. It is assigned when the pod is first
deployed, stays assigned while the pod is stopped, and resumes on the same
machine. Moving to another machine means a new pod, through migration or a
fresh deploy, with a new pod id.
status: documented
see: [pod.stopped-pod-keeps-host]
evidence:
- { source: rest-v2-spec, ref: Pod.cudaVersion, seen: "2026-09-28", note: A stopped pod keeps its machine assignment and resumes onto the same host. }
- { source: public-docs, url: "https://docs.runpod.io/pods/troubleshooting/zero-gpus", seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/pods/troubleshooting/pod-migration", seen: "2026-09-28", note: Pod ids are tied to physical machines. }
- id: machine.gpus-are-per-machine
statement: >
A running pod reserves GPUs on its machine for as long as it runs.
Stopping the pod releases those GPUs, and another customer can rent them
while the pod is stopped.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/troubleshooting/zero-gpus", seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/pods/troubleshooting/pod-migration", seen: "2026-09-28" }
- id: machine.largest-machine-caps-gpu-count
statement: >
Because a pod runs on one machine, the most GPUs one pod can use is the
GPU count of the largest machine of that type. `maxCount` on the GPU type
reports that ceiling for each cloud.
status: documented
see: [gpu-type.max-count-not-stock]
evidence:
- { source: rest-v2-spec, ref: GpuType.maxCount, seen: "2026-09-28" }
- id: machine.host-local-storage
statement: >
The pod's volume disk is tied to its machine. The data stays while the pod
is stopped, can become unavailable or be lost if the machine fails, and is
not billed while the machine is unavailable. Use a network volume for data
that must outlive the machine.
status: documented
see: [pod-volume-disk.deprecated-prefer-network]
evidence:
- { source: rest-v2-spec, ref: PersistentMount, seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/accounts-billing/billing", seen: "2026-09-28", note: Storage is not charged while the host machine is unavailable. }
- id: machine.cuda-is-per-machine
statement: >
The CUDA version is a property of the machine's driver, not of the GPU
type. Machines with the same GPU type can report different versions, and
`cudaVersion` on a pod shows what its machine reports.
status: documented
see: [pod.cuda-discover-first]
evidence:
- { source: rest-v2-spec, ref: "Pod.cudaVersion; GpuType.cudaVersions", seen: "2026-09-28" }
concepts/model-store.yaml
id: model-store
name: Model Store
kind: capability
summary: >
Runpod-managed caching of model weights on hosts, so Serverless workers start
with the model already present. Attach a Hugging Face model by URL, or upload
your own model to the Model Repository.
product: storage
is_a: null
part_of: runpod-platform
aliases: [cached models, model cache, model weights, modelReferences, Model Repository]
surfaces:
runpodctl: [model add, model list, model remove, serverless create --model-reference, serverless update --model-reference, serverless model-status]
relations:
- { type: uses, target: serverless-endpoint, cardinality: "0..n" }
- { type: uses, target: network-volume, cardinality: "0..1" }
rules:
- id: model-store.attach-by-reference
statement: >
Attach a Hugging Face model to an endpoint by its full URL with a ref, for
example `https://huggingface.co/<org>/<model>:main`. Public, gated and
private models all work; gated and private ones need a Hugging Face token.
Workers find the model in the standard Hugging Face cache under
`/runpod-volume/huggingface-cache/hub/`.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/model-caching", seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpodctl/reference/model-caching.md }
- id: model-store.gpu-endpoints-only
statement: >
`runpodctl serverless create --model-reference` works with `--template-id`
and `--hub-id` on GPU endpoints only (`--compute-type GPU`).
status: documented
evidence:
- { source: skill, path: plugins/runpod/skills/runpodctl/reference/model-caching.md }
- id: model-store.no-download-billing
statement: >
Model download time is not billed, whether or not the cache hits. On a
miss, worker start is held until the download finishes, so jobs wait in
the queue and the endpoint can look stuck while it downloads.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/model-caching", seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpodctl/reference/model-caching.md }
- id: model-store.best-effort-locality
statement: >
The scheduler prefers hosts that already hold the model, but residency is
best-effort. Cache capacity depends on each machine's disk, so two workers
on one endpoint can cold-start at different speeds. For a
latency-sensitive endpoint, pre-load the weights on a network volume
instead.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/model-caching", seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpodctl/reference/model-caching.md }
- id: model-store.not-on-hugging-face
statement: >
For a private model that is not on Hugging Face, upload it to the Model
Repository with `runpodctl model add`, or bake it into the image.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/model-caching", seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpodctl/reference/model-caching.md }
- id: model-store.deployment-routes
statement: >
Hugging Face model caching can be configured for Docker, GitHub and Hub
Serverless deployments. Choosing the managed GitHub builder does not
require baking eligible model weights into its image.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/model-caching", seen: "2026-10-10" }
- { source: skill, path: plugins/runpod/skills/runpod-build-template/reference/model-delivery.md, note: Delivery choices and documented cache constraints. }
- id: model-store.one-model-per-endpoint
statement: >
An endpoint supports one cached model. An adapter and a separate base can
require different repositories, so account for every required component
instead of assuming caching one also supplies the other.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/model-caching", seen: "2026-10-10" }
- { source: skill, path: plugins/runpod/skills/runpod-build-template/reference/model-delivery.md, note: Delivery choices and documented cache constraints. }
- id: model-store.all-quantization-variants
statement: >
When a Hugging Face repository contains several quantization variants,
Model Store downloads all of them. Account for the whole repository's
download size and cache-miss startup time, not just the variant the
application loads.
status: documented
see: [model-store.no-download-billing]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/model-caching", seen: "2026-10-14", note: Cached-model limitations. }
- id: model-store.custom-loader-integration
statement: >
A custom worker must configure its loader's cache directory or pass the
matching local snapshot; using a Hugging Face-compatible library alone
does not establish cache discovery. Match the repository and revision,
validate the loader's handling of cache symlinks, and keep writes in
designated writable storage rather than the provider cache.
status: documented
see: [model-store.attach-by-reference]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/model-caching", seen: "2026-10-13" }
- { source: public-docs, url: "https://docs.runpod.io/serverless/development/huggingface-models", seen: "2026-10-13" }
- { source: skill, path: plugins/runpod/skills/runpod-build-template/reference/model-delivery.md, note: Connect configuration to the loader. }
- id: model-store.flash-model-parameter
statement: >
In a Flash SDK app, attach a Hugging Face model with
`model=runpod.Model("owner/name")` on `@app.queue` or `@app.api`. The
platform stages the weights on the host before the worker starts, and
`Model.path` gives the staged directory. Gated models need `HF_TOKEN` on
the same resource.
status: documented
see: [model-store.attach-by-reference, model-store.best-effort-locality]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/sdk/overview", seen: "2026-10-16" }
- { source: other, url: "https://github.com/runpod/runpod-python/blob/v2.0.0rc1/runpod/apps/model.py", seen: "2026-10-16" }
- { source: skill, path: plugins/runpod/skills/runpod-flash/reference/api.md, note: Models and secrets. }
- id: model-store.flash-attach-after-deploy
statement: >
For a legacy `runpod-flash` endpoint, attach the model after `flash
deploy` with `runpodctl serverless update <endpoint-id> --model-reference
<url>:<ref>`, which prints the saved `modelReferences`. Later `flash
deploy` runs keep the reference. Point `HF_HUB_CACHE` at
`/runpod-volume/huggingface-cache/hub` so the load finds it.
status: verified
see: [model-store.attach-by-reference, model-store.best-effort-locality]
evidence:
- { source: live-probe, ref: "runpodctl 2.14.0 serverless update --model-reference on a runpod-flash 1.20.1 endpoint, then flash deploy with a changed env; GraphQL myself.endpoints.modelReferences kept the reference", seen: "2026-10-14" }
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/model-caching", seen: "2026-10-14", note: Cached models are under /runpod-volume/huggingface-cache/hub/. }
- { source: skill, path: plugins/runpod/skills/runpodctl/reference/model-caching.md }
concepts/network-volume-billing.yaml
id: network-volume-billing
name: Network volume billing
kind: billing
summary: >
What network volumes cost over time, split into standard and high-performance
storage, with one record per volume per time bucket.
product: billing
is_a: billing-record
part_of: null
aliases: [storage billing, volume cost]
surfaces:
rest_v2: { schema: NetworkVolumeBillingRecord, paths: [/v2/billing/network-volumes] }
mcp: [list-network-volume-billing]
runpodctl: [billing network-volume]
fields:
- { name: networkVolumeId, type: string, ref: network-volume, set: read }
- { name: startTime, type: string, set: read }
- { name: endTime, type: string, set: read }
- { name: totalAmount, type: number, unit: USD, set: read }
- { name: standardAmount, type: number, unit: USD, set: read }
- { name: highPerformanceAmount, type: number, unit: USD, set: read }
relations:
- { type: requires, target: network-volume, cardinality: "1" }
rules:
- id: network-volume-billing.per-volume-per-bucket
applies_to: ["field:networkVolumeId", "field:standardAmount", "field:highPerformanceAmount"]
statement: >
Records are one per volume per bucket, split into standard and
high-performance storage. Pass `networkVolumeId` to narrow the results to
one volume. The path is hyphenated (`/v2/billing/network-volumes`), unlike
REST v1's `/billing/networkvolumes`.
status: documented
evidence:
- { source: rest-v2-spec, ref: "NetworkVolumeBillingRecord; GET /v2/billing/network-volumes", seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpod-migrate/reference/rest-v1-to-v2.md }
- id: network-volume-billing.price
applies_to: ["field:totalAmount"]
statement: >
Network volume storage is billed hourly per GB of allocated size from
creation until deletion, whether or not anything is attached. The per-GB
rate is lower for the part of a volume above 1 TB. Delete volumes you no
longer need.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/pricing", seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/accounts-billing/billing", seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/storage.md }
- { source: rest-v2-spec, ref: POST /v2/network-volumes }
concepts/network-volume.yaml
id: network-volume
name: Network volume
kind: resource
summary: >
Persistent storage that lives in one data center, outlives any pod or worker,
and mounts into pods, Serverless endpoints and Instant Clusters in that data
center. It is not the volume disk, which lives on a single pod's machine.
product: storage
is_a: null
part_of: runpod-platform
aliases: [volume, network storage, NV, NetworkVolume]
surfaces:
rest_v2: { schema: NetworkVolume, paths: [/v2/network-volumes, "/v2/network-volumes/{id}"] }
graphql: { type: NetworkVolume }
mcp: [create-network-volume, list-network-volumes, get-network-volume, update-network-volume, delete-network-volume]
runpodctl: [network-volume create, network-volume list, network-volume get, network-volume update, network-volume delete]
console: Storage
fields:
- { name: id, type: string, set: read, note: also the S3 bucket name }
- { name: name, type: string, set: "create,update", required: true, note: not unique }
- { name: size, type: integer, unit: GB, set: "create,update", required: true, note: "10-4096 on create" }
- { name: dataCenter, type: string, ref: data-center, set: create, required: true }
- { name: type, type: enum, values: [STANDARD, HIGH_PERFORMANCE], set: create }
relations:
- { type: located_in, target: data-center, cardinality: "1" }
- { type: constrained_by, target: cloud-tier, cardinality: "1" }
- { type: billed_by, target: network-volume-billing, cardinality: "1" }
- { type: constrained_by, target: credit-balance, cardinality: "1" }
rules:
- id: network-volume.data-center-level-not-machine-level
statement: >
A network volume lives at the data center level: any pod or endpoint in
its data center can attach it, it can move between machines, and it keeps
its data after the pod is deleted. The volume disk is different; it lives
on one pod's machine and only that pod can use it.
status: documented
see: [pod-volume-disk.machine-level-not-data-center-level, network-volume.same-dc-only]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/storage/types", seen: "2026-09-28", note: "A network volume can be attached to multiple pods, moved between machines, and kept after the pod is deleted." }
- { source: public-docs, url: "https://docs.runpod.io/storage/network-volumes", seen: "2026-09-28" }
- id: network-volume.size-bounds
applies_to: ["field:size"]
statement: >
`size` is 10 to 4096 GB at creation. For a larger volume, contact Runpod
support.
on_violation: 422
status: documented
evidence:
- { source: rest-v2-spec, ref: CreateNetworkVolumeRequest.size, seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/storage/network-volumes", seen: "2026-09-28" }
- id: network-volume.size-grow-only
applies_to: ["field:size"]
statement: >
`size` can only grow. Storage cannot be reduced after creation, so start
with what you need and expand later.
status: documented
evidence:
- { source: rest-v2-spec, ref: UpdateNetworkVolumeRequest.size, seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/storage/network-volumes", seen: "2026-09-28" }
- id: network-volume.placement-immutable
applies_to: ["field:dataCenter", "field:type"]
statement: >
`dataCenter` and `type` are fixed at creation. To move data to another
data center, create a new volume there and copy the data.
status: documented
see: [pod.network-volume-same-dc, pod.mount-kind-fixed]
evidence:
- { source: rest-v2-spec, ref: "NetworkVolume.dataCenter; NetworkVolume.type", seen: "2026-09-28" }
- id: network-volume.type-per-dc
applies_to: ["field:type", "field:dataCenter"]
statement: >
`type` must be a tier the data center lists in `networkVolumeTypes`. When
`type` is omitted, the data center's default tier is used.
`HIGH_PERFORMANCE` gives up to 3x the throughput and 4x the IOPS of
`STANDARD`, at a higher price per GB, in select data centers.
status: documented
see: [data-center.volume-types]
evidence:
- { source: rest-v2-spec, ref: "CreateNetworkVolumeRequest.type; DataCenter.networkVolumeTypes", seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/storage/high-performance-storage", seen: "2026-09-28" }
- id: network-volume.same-dc-only
applies_to: ["field:dataCenter"]
statement: >
Only compute in the volume's own data center can mount it, which limits
GPU choice to what that data center offers. Volumes in different data
centers do not sync.
status: documented
see: [pod.network-volume-same-dc]
evidence:
- { source: rest-v2-spec, ref: NetworkMount.volumeId, seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/storage/network-volumes", seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/storage.md }
- id: network-volume.serverless-one-per-dc
statement: >
A Serverless endpoint can attach several network volumes, at most one per
data center. Each worker gets exactly one volume, the one in its data
center, so attaching volumes in several data centers widens GPU choice.
Copy the data to each volume yourself, since volumes do not sync.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/storage/network-volumes", note: "See \"Attach multiple volumes\".", seen: "2026-09-28" }
- id: network-volume.mount-path
statement: >
On a pod created through REST v2, the network mount path has no default
and must be sent. Pods conventionally use `/workspace`. Serverless workers
see the volume at `/runpod-volume`.
status: documented
evidence:
- { source: rest-v2-spec, ref: NetworkMount.path, seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/storage/network-volumes", seen: "2026-09-28" }
- id: network-volume.concurrent-writes
statement: >
Several workers or pods can have the same volume mounted, but writing to
it from more than one at a time can corrupt data. Coordinate concurrent
writes in your application.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/storage/network-volumes", seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/storage.md }
- id: network-volume.secure-cloud-only
statement: >
Network volumes can be attached only to Secure Cloud pods.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/storage/network-volumes", seen: "2026-09-28" }
- id: network-volume.allocated-size-only
applies_to: ["field:size"]
statement: >
`size` is the allocated size. The API returns no usage figure, so track
how full a volume is from inside a pod or worker.
status: documented
see: [network-volume-billing.price]
evidence:
- { source: rest-v2-spec, ref: NetworkVolume, note: The schema has no usage field., seen: "2026-09-28" }
- id: network-volume.low-balance-deletion
statement: >
When the account balance reaches $0, pods with an attached network volume
stop and the data stays on the volume, but storage charges keep accruing.
If the balance stays at $0, the volume can be terminated and its data
cannot be recovered. Turn on low-balance notifications.
status: documented
see: [network-volume.delete-irreversible, credit-balance.low-balance-actions]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/storage/network-volumes", seen: "2026-09-28" }
- id: network-volume.no-status-field
statement: >
A REST v2 volume has no status field. It is usable once the create call
returns, so there is nothing to poll.
status: documented
evidence:
- { source: rest-v2-spec, ref: NetworkVolume, seen: "2026-09-28" }
- id: network-volume.delete-irreversible
statement: >
Deleting a volume is permanent and its data cannot be recovered. Copy
anything you need, for example over the S3-compatible API, before you
delete it.
status: documented
see: [network-volume.s3-api]
evidence:
- { source: rest-v2-spec, ref: "DELETE /v2/network-volumes/{id}", seen: "2026-09-28" }
- id: network-volume.s3-api
applies_to: ["field:id", "field:dataCenter"]
statement: >
In supported data centers, a volume is reachable through Runpod's
S3-compatible API without running a pod, with the volume `id` as the
bucket and an S3 API key created in the console.
status: documented
see: [file-transfer.s3-api-per-data-center]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/storage/s3-api", seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/storage.md }
concepts/organization.yaml
id: organization
name: Organization
kind: resource
summary: >
The organization the caller belongs to, with its members and the role each
holds. REST v2 and MCP read it; they do not change it. A team account is not
an organization: these reads answer it `404`.
product: account
is_a: null
part_of: runpod-platform
aliases: [org, Organization]
surfaces:
rest_v2: { schema: Organization, paths: [/v2/organization, /v2/organization/users, "/v2/organization/users/{id}"] }
mcp: [get-organization, list-organization-users, get-organization-user]
fields:
- { name: id, type: string, set: read }
- { name: name, type: string, set: read }
rules:
- id: organization.membership-required
statement: >
The organization reads return `404` when the caller does not belong to an
organization, and `403` when organizations are unavailable to the caller.
State either as the answer, not as a tool failure.
on_violation: 404 not a member, 403 unavailable
status: documented
evidence:
- { source: rest-v2-spec, ref: GET /v2/organization, seen: "2026-11-02" }
- { source: rest-v2-spec, ref: GET /v2/organization/users, seen: "2026-11-02" }
- id: organization.roster-readable-by-members
statement: >
Any member can read the full member list, which takes no pagination, and
each member's role. A user who is not a member, including one from another
organization, returns `404` on the single-member read.
status: documented
evidence:
- { source: rest-v2-spec, ref: GET /v2/organization/users, seen: "2026-11-02" }
- { source: rest-v2-spec, ref: "GET /v2/organization/users/{id}", seen: "2026-11-02" }
- id: organization.roles
statement: >
A member's role is a built-in Runpod role (`builtin:admin`, `builtin:dev`,
`builtin:billing` or `builtin:basic`) or a custom role the organization
defined under its own id.
status: documented
evidence:
- { source: rest-v2-spec, ref: "OrganizationRoleRef.id; OrganizationRoleType", seen: "2026-11-02" }
concepts/pod-billing.yaml
id: pod-billing
name: Pod billing
kind: billing
summary: >
What GPU and CPU pods cost over time: compute while a pod runs, plus disk
storage, including while it is stopped. History is reported per pod per time
bucket.
product: billing
is_a: billing-record
part_of: null
aliases: [pod cost, costPerHr, adjustedCostPerHr]
surfaces:
rest_v2: { schema: PodBillingRecord, paths: [/v2/billing/pods] }
mcp: [list-pod-billing]
runpodctl: [billing pods]
fields:
- { name: podId, type: string, ref: pod, set: read }
- { name: startTime, type: string, set: read }
- { name: endTime, type: string, set: read }
- { name: totalAmount, type: number, unit: USD, set: read }
- { name: gpuAmount, type: number, unit: USD, set: read }
- { name: cpuAmount, type: number, unit: USD, set: read }
- { name: diskAmount, type: number, unit: USD, set: read }
relations:
- { type: requires, target: pod, cardinality: "1" }
rules:
- id: pod-billing.per-pod-per-bucket
applies_to: ["field:podId"]
statement: >
Records cover GPU and CPU pods alike, one record per pod per bucket. Pass
`podId` to narrow the results to one pod.
status: documented
evidence:
- { source: rest-v2-spec, ref: "PodBillingRecord.podId; GET /v2/billing/pods", seen: "2026-09-28" }
- id: pod-billing.live-rate-is-pod-cost
applies_to: ["field:totalAmount"]
statement: >
This route is history. A pod's current hourly compute rate is its `cost`
field, which reads 0 while the pod is stopped even though its disk is
still billed.
status: documented
see: [pod.cost-is-compute-only]
evidence:
- { source: rest-v2-spec, ref: "Pod.cost; GpuType.price", seen: "2026-09-28" }
- id: pod-billing.stopped-disk-rate
applies_to: ["field:diskAmount"]
statement: >
A stopped pod keeps paying for its volume disk, at a higher per-GB rate
than while it runs; container disk is not billed while the pod is stopped.
Terminate the pod, rather than stopping it, to end all charges.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/pricing", seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/storage.md }
- id: pod-billing.per-second
statement: >
Pods are billed per second for compute and for container and volume disk.
Hourly rates express the same price in a larger unit for readability.
There is no charge for storage while the pod's host machine is
unavailable.
status: documented
see: [network-volume-billing.price]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/overview", seen: "2026-10-06", note: Says pods are billed by the second. }
- { source: public-docs, url: "https://docs.runpod.io/pods/pricing", seen: "2026-10-06", note: Says pods are billed per second. }
- { source: public-docs, url: "https://docs.runpod.io/accounts-billing/billing", seen: "2026-10-06" }
- id: pod-billing.deploy-needs-an-hour
statement: >
Deploying an on-demand pod requires a balance that covers at least one
hour of the selected configuration. Add funds or pick a cheaper
configuration if the deploy is refused.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/pricing", seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/accounts-billing/billing", seen: "2026-09-28" }
concepts/pod-command-exec.yaml
id: pod-command-exec
name: Command execution inside a pod
kind: capability
summary: >
Running a command in a pod without opening an interactive shell. Customers and
agents do this over SSH.
product: pods
is_a: null
part_of: pod
aliases: [exec, runpodctl exec, pod commands, SSH exec]
surfaces:
mcp: []
runpodctl: [exec python]
fields: []
relations:
- { type: requires, target: pod-ssh-access, cardinality: "1" }
rules:
- id: pod-command-exec.use-ssh
statement: >
REST v2 and the Runpod MCP server have no route that runs a command in a
pod. Run commands over SSH with `ssh <user>@<host> -p <port> "<command>"`
from the pod's `ssh` block, and prefer `ssh.direct`.
status: documented
see: [pod-ssh-access.proxy-vs-direct]
evidence:
- { source: rest-v2-spec, ref: paths under /v2/pods, seen: "2026-09-28", note: "The pod paths cover CRUD, actions and logs only." }
- { source: public-docs, url: "https://docs.runpod.io/pods/configuration/use-ssh" }
- id: pod-command-exec.runpodctl-exec-deprecated
statement: >
`runpodctl exec python` runs a local Python file on a pod over SSH. It is
deprecated in favor of plain `ssh`, and it needs the same SSH access.
status: documented
evidence:
- { source: other, ref: runpodctl exec --help (runpodctl 2.14.0), seen: "2026-09-28" }
concepts/pod-deployment.yaml
id: pod-deployment
name: How pod deployment works
kind: process
summary: >
What happens from creating a pod to restarting it. A pod is placed on one
machine in a data center and stays tied to that machine. Stopping it releases
the machine's GPUs to other customers, so a later start can find no GPU on
that machine even when the data center has stock.
product: pods
is_a: null
part_of: pod
aliases: [pod placement, deploying a pod, pod scheduling, zero GPU pods]
steps:
- id: request
title: Request a pod
description: >
You choose a GPU type and count (or a CPU flavor), a cloud tier, and
optionally a list of data centers. Attaching a network volume limits the
pod to the volume's data center.
concepts: [pod, gpu-type, cloud-tier, data-center, network-volume, global-volume]
rules: [pod.gpu-xor-cpu, cloud-tier.default-to-secure, pod.network-volume-same-dc, pod-deployment.stock-before-volume, global-volume.any-data-center]
- id: place
title: Runpod places it on a machine
description: >
Runpod looks for one machine, in an allowed data center, with that many
free GPUs of exactly the type you asked for. It does not substitute
another GPU type. If no machine qualifies, the create call fails.
concepts: [machine, data-center, gpu-availability, gpu-type]
rules: [pod.no-capacity-fallback, machine.largest-machine-caps-gpu-count, pod-deployment.deploy-when-available]
- id: run
title: The pod runs on that machine
description: >
The pod's GPUs on that machine are reserved for it while it runs, and
compute is billed. Create returns before the pod is ready, so poll its
status and then check the service inside it.
concepts: [pod, machine, pod-billing]
rules: [pod.create-is-async, pod.running-is-not-ready, machine.gpus-are-per-machine]
- id: stop
title: Stopping releases the GPUs
description: >
Stopping the pod releases its GPUs, and another customer can rent them.
The pod stays tied to the same machine. Its volume disk stays on the
machine and is still billed, and its container disk is cleared.
concepts: [pod, machine, pod-volume-disk, container-disk, pod-billing]
rules: [pod.stopped-pod-keeps-host, container-disk.what-survives, pod.cost-is-compute-only]
- id: restart
title: Restarting needs a GPU on the same machine
description: >
Starting the pod again resumes it on its original machine. If that
machine's GPUs were rented while the pod was stopped, the pod cannot get a
GPU, even when the data center or region reports stock for the same GPU
type on other machines.
concepts: [pod, machine, gpu-availability]
rules: [machine.pod-is-bound, pod-deployment.stock-is-not-your-machine]
- id: recover
title: Recover when the machine has no free GPU
description: >
Wait and retry, start the pod with zero GPUs to copy data off, migrate it,
or terminate it and deploy a new pod. Migration creates a new pod on
another machine and moves the pod's data to it; terminating and
redeploying leaves the volume disk behind. Either way the new pod has a
new id and address, and data on a network volume can be attached to any
pod in its data center.
concepts: [pod-migration, network-volume, pod-volume-disk]
rules: [pod-migration.options, pod-migration.new-id-and-address, pod-deployment.keep-data-portable]
rules:
- id: pod-deployment.stock-is-not-your-machine
statement: >
Catalog availability describes free GPUs across a data center's machines,
but a stopped pod can only resume on its own machine. High stock for your
GPU type does not mean your stopped pod can start with a GPU.
status: documented
see: [gpu-availability.per-type-per-dc-per-product]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/troubleshooting/pod-migration", seen: "2026-09-28", note: "A migration prompt does not mean the GPU type is unavailable on Runpod, only on the pod's machine." }
- { source: public-docs, url: "https://docs.runpod.io/pods/troubleshooting/zero-gpus", seen: "2026-09-28" }
- id: pod-deployment.keep-data-portable
statement: >
Keep data you need on a network volume. Then a pod that cannot get a GPU
on its machine can be terminated and redeployed on any machine with free
GPUs in the volume's data center, with the data attached.
status: documented
see: [pod.network-volume-same-dc]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/troubleshooting/zero-gpus", seen: "2026-09-28", note: "See \"How do I prevent this?\"." }
- id: pod-deployment.stock-before-volume
statement: >
A network volume limits the pod to the volume's data center, so read GPU
stock per data center first and create the volume once, in a data center
that has the GPU type. After a create refused for stock, read stock again
and retry once in the volume's data center if it shows stock; otherwise
the next step is the user's choice, never a new volume per attempt.
Without a volume, list every data center with stock in `dataCenterIds` and
Runpod places the pod in one of them. When no data center that offers
network storage has the GPU type, a global volume, which has no data
center, is the fallback for checkpoints.
status: documented
see: [pod.network-volume-same-dc, pod.no-capacity-fallback, gpu-availability.per-type-per-dc-per-product, global-volume.any-data-center, global-volume.eventual-consistency, global-volume.object-backed]
evidence:
- { source: rest-v2-spec, ref: "CreatePodRequest.dataCenterIds; NetworkMount.volumeId", seen: "2026-10-30" }
- { source: public-docs, url: "https://docs.runpod.io/storage/globalvolume/globalvolume-pods", seen: "2026-11-02" }
- { source: skill, path: plugins/runpod/skills/runpod-migrate/reference/breaking-changes.md, note: "`dataCenterIds` is enforced: a create is placed only in a listed data center." }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/storage.md, note: "See \"Global volume (any data center)\": the checkpoint fallback." }
- id: pod-deployment.deploy-when-available
statement: >
When the GPU type you pick is out of capacity, the console offers Deploy
when available in place of Deploy Pod. It queues the pod and deploys it
automatically once that GPU frees up within the time window you choose. If
the account lacks the funds at that moment, no pod is deployed.
status: documented
see: [gpu-availability.per-type-per-dc-per-product]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/get-started", seen: "2026-10-30" }
concepts/pod-migration.yaml
id: pod-migration
name: Pod migration
kind: capability
summary: >
When a stopped pod's GPU has been rented by someone else, Runpod can create a
new pod with the same configuration on another machine and move the pod's data
to it. The new pod has a new id, IP address and proxy URL.
product: pods
is_a: null
part_of: pod
aliases: [migrate pod, automatic pod migration]
surfaces:
console: Pods
relations:
- { type: requires, target: pod, cardinality: "1" }
- { type: uses, target: network-volume, cardinality: "0..1" }
rules:
- id: pod-migration.not-in-rest-v2
statement: >
REST v2 has no migrate action; the pod actions are `start`, `stop`,
`restart` and `terminate`. Migration is offered in the console when a
stopped pod cannot get its GPU back.
status: documented
evidence:
- { source: rest-v2-spec, ref: PodAction, seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/pods/troubleshooting/pod-migration" }
- id: pod-migration.new-id-and-address
statement: >
A migrated pod gets a new pod id, IP address and proxy URL. Update any API
calls, proxy URLs, firewall rules or VPN settings that use the old id or
IP.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/troubleshooting/pod-migration", note: Important considerations for migration. }
- id: pod-migration.options
statement: >
When the GPU is taken, you can wait and retry, start the pod with CPUs
only to reach its data, or migrate to a new machine. Keeping data on a
network volume avoids the choice, because a new pod anywhere can attach
it.
status: documented
see: [container-disk.what-survives, pod.stopped-pod-keeps-host]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/troubleshooting/pod-migration", note: Your options when GPUs are unavailable. }
concepts/pod-ssh-access.yaml
id: pod-ssh-access
name: Pod SSH and web terminal access
kind: capability
summary: >
Shell access to a running pod, either through Runpod's SSH proxy, which needs
no public IP, or directly to the pod's SSH daemon over its published `22/tcp`
port.
product: pods
is_a: null
part_of: pod
aliases: [SSH, web terminal, ssh.runpod.io, startSsh]
surfaces:
rest_v2: { schema: Pod }
graphql: { type: Pod, mutations: [podFindAndDeployOnDemand] }
mcp: [create-pod, get-pod, get-ssh-keys, update-ssh-keys]
runpodctl: [ssh info, pod get]
console: Pods (Connect tab)
fields:
- { name: startSsh, type: boolean, set: create, note: "Write-only. GET never returns it, and PATCH cannot change it." }
- { name: ports, type: "string[]", set: "create,update", note: "Include `22/tcp` for direct SSH." }
- { name: ssh.proxy, type: PodSshEndpoint, set: read, note: Null until the pod is assigned a machine. }
- { name: ssh.direct, type: PodSshEndpoint, set: read, note: "Null unless `22/tcp` is in `ports` and the running pod has a public port for it." }
relations:
- { type: requires, target: ssh-key, cardinality: "1..n" }
- { type: uses, target: exposed-port, cardinality: "0..1" }
rules:
- id: pod-ssh-access.start-ssh-needs-keys
applies_to: ["field:startSsh"]
statement: >
Set `startSsh` on create to inject the account's registered SSH public
keys into the pod as `PUBLIC_KEY`. A pod created without it has no SSH
access, whatever its `ssh` block reports, and with no keys registered the
flag does nothing. Only images that read `PUBLIC_KEY` start an SSH daemon;
all official Runpod images do.
status: verified
evidence:
- { source: rest-v2-spec, ref: "CreatePodRequest.startSsh; PodSsh", seen: "2026-09-28" }
- { source: live-probe, ref: "POST /v2/pods with startSsh: true on runpod/base:1.0.2-ubuntu2204 (CPU pod)", seen: "2026-09-28", note: "The pod's env held `PUBLIC_KEY`, `authorized_keys` held every registered key, and SSH over `ssh.direct` succeeded.\n" }
- id: pod-ssh-access.proxy-vs-direct
applies_to: ["field:ssh.proxy", "field:ssh.direct", "field:ports"]
statement: >
The proxy connection needs no public IP or exposed port but carries an
interactive shell only. SCP, SFTP, rsync and port forwarding need
`ssh.direct`, which requires `22/tcp` in `ports` and is null while the pod
is provisioning or stopped. Prefer `ssh.direct` when it is available.
status: documented
evidence:
- { source: rest-v2-spec, ref: PodSsh, seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/pods/configuration/use-ssh" }
- { source: skill, path: plugins/runpod/skills/runpod/golden-paths/06-dev-pod.md }
- id: pod-ssh-access.web-terminal-is-session-based
statement: >
The browser web terminal is tied to its session and is not suited to
long-running work. Use SSH for long-running processes, and run them under
`tmux` or a similar tool so they survive a dropped connection.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/connect-to-a-pod" }
concepts/pod-volume-disk.yaml
id: pod-volume-disk
name: Pod volume disk
kind: component
summary: >
Persistent storage scoped to one pod and tied to its machine, mounted at
`/workspace` by default. Only that pod can use it. It survives stops and
restarts and is deleted when the pod is terminated. It is not a network
volume, which lives at the data center level.
product: pods
is_a: null
part_of: pod
aliases: [volume disk, persistent mount, /workspace disk, pod volume, local volume, encrypted volume, volumeInGb]
surfaces:
rest_v2: { schema: Pod }
runpodctl: [pod create --volume-in-gb, pod create --volume-mount-path]
console: Pods
fields:
- { name: mounts.persistent.size, type: integer, unit: GB, set: create, note: Minimum 10. }
- { name: mounts.persistent.path, type: string, set: "create,update", note: "Usually `/workspace`." }
relations:
- { type: located_in, target: machine, cardinality: "1" }
- { type: billed_by, target: pod-billing, cardinality: "1" }
rules:
- id: pod-volume-disk.machine-level-not-data-center-level
applies_to: ["field:mounts.persistent.size"]
statement: >
The volume disk belongs to one pod and is tied to that pod's machine; only
that pod can use it. A network volume is the data-center-level storage
that any pod or endpoint in the data center can attach. A pod mounts one
or the other at `/workspace`, never both.
status: documented
see: [network-volume.data-center-level-not-machine-level, machine.host-local-storage]
evidence:
- { source: rest-v2-spec, ref: "PersistentMount; Mounts", seen: "2026-09-28", note: "The persistent mount is pinned to the pod's host machine, and it cannot be combined with a network mount." }
- { source: public-docs, url: "https://docs.runpod.io/pods/storage/types", seen: "2026-09-28", note: "A network volume replaces the volume disk at `/workspace`." }
- { source: public-docs, url: "https://docs.runpod.io/pods/troubleshooting/zero-gpus", seen: "2026-09-28", note: "A stopped pod's volume storage remains on the physical machine." }
- id: pod-volume-disk.network-filesystem-backed
applies_to: ["field:mounts.persistent.size"]
statement: >
Whether `/workspace` is local depends on the data center. In a data center
that offers network volumes, expect `/workspace` to be network-backed
storage, with writes several times slower than on the container disk. In a
data center without network volumes it can go either way: some give a
local `/workspace` as fast as the container disk, others still use
network-backed storage. Check from inside the pod with `findmnt
/workspace`: a `fuse` mount is network-backed, an `xfs` filesystem on a
device is local. Either way, the container disk is local, so you don't
need to pick a data center for disk speed.
status: verified
see: [container-disk.always-local]
evidence:
- { source: live-probe, ref: "ubuntu:22.04 GPU pods with a 10 GB volume disk and no network volume. findmnt showed / as a local overlay and /workspace as a fuse mount in EU-RO-1 (offers network volumes) and in EU-SE-1 and US-MO-2 (no network volumes). dd 512 MB oflag=direct: /workspace 713, 443 and 568 MB/s; container disk 2.8, 4.1 and 2.4 GB/s", seen: "2026-10-26" }
- { source: live-probe, ref: "Same probe in OC-AU-1, US-TX-4 and US-PA-1 (no network volumes): findmnt showed /workspace as a local xfs block device. dd 512 MB oflag=direct: /workspace 5.0, 4.3 and 0.58 GB/s; container disk 5.4, 4.3 and 0.56 GB/s", seen: "2026-10-26" }
- id: pod-volume-disk.lifetime-and-billing
applies_to: ["field:mounts.persistent.size"]
statement: >
The volume disk survives stops and restarts and is deleted when the pod is
terminated. It is billed while the pod runs and, at a higher rate, while
the pod is stopped; billing ends when the pod is terminated.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/storage/types" }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/storage.md }
- id: pod-volume-disk.deprecated-prefer-network
applies_to: ["field:mounts.persistent.size"]
statement: >
The volume disk is tied to one host and is lost if that host fails. REST
v2 marks the persistent mount as deprecated; use a network volume for data
you cannot recreate.
status: documented
see: [pod.persistent-mount-limits]
evidence:
- { source: rest-v2-spec, ref: PersistentMount, seen: "2026-09-28" }
- id: pod-volume-disk.grow-only
applies_to: ["field:mounts.persistent.size"]
statement: >
The volume disk can be increased but not decreased. Editing a running pod
to resize it resets the pod.
status: documented
see: [pod.edits-apply-on-restart]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/storage/types", note: Modify storage capacity. }
- id: pod-volume-disk.encryption
statement: >
The volume disk can be encrypted at rest by selecting **Encrypt volume**
when you create the pod in the console. The key cannot be retrieved, you
cannot bring your own key, and encryption does not apply to the container
disk or network volumes. REST v2 has no encryption field.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/storage/types", note: Encrypted volumes. }
- { source: rest-v2-spec, ref: PersistentMount, seen: "2026-09-28", note: Only size and path. }
- id: pod-volume-disk.none-on-serverless
statement: >
Serverless workers have no volume disk. Their storage options are the
container disk, network volumes and S3-compatible storage.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/storage/overview" }
concepts/pod.yaml
id: pod
name: Pod
kind: resource
summary: >
A GPU or CPU container that you rent on demand and manage directly. You can
start, stop, restart and terminate it, connect over SSH, and expose ports.
product: pods
is_a: null
part_of: runpod-platform
aliases: [GPU pod, instance, rented machine, pod status, pod state, desiredStatus]
surfaces:
rest_v2: { schema: Pod, paths: [/v2/pods, "/v2/pods/{id}", "/v2/pods/{id}/action", "/v2/pods/{id}/logs"] }
graphql: { type: Pod, mutations: [podFindAndDeployOnDemand, podEditJob, podStop, podResume, podTerminate] }
mcp: [create-pod, list-pods, get-pod, update-pod, pod-action, delete-pod, stream-pod-logs]
runpodctl: [pod create, pod list, pod get, pod update, pod start, pod stop, pod restart, pod reset, pod delete, pod logs]
console: Pods
fields:
- { name: name, type: string, set: "create,update", required: true }
- { name: gpu, type: CreateGpuConfig, set: create, note: Present only on GPU pods. }
- { name: gpu.id, type: string, ref: gpu-type, set: create, note: One exact GPU type. The API does not fall back to another. }
- { name: gpu.count, type: integer, default: 1, set: create }
- { name: gpu.allowedCudaVersions, type: "string[]", set: create }
- { name: gpu.minCudaVersion, type: string, set: create }
- { name: cpu, type: CreateCpuConfig, set: create, note: Present only on CPU pods. }
- { name: image, type: string, set: "create,update" }
- { name: templateId, type: string, ref: template, set: "create,update", note: Applied once. The pod keeps no link to the template. }
- { name: args, type: string, set: "create,update", note: "The container command. `entrypoint` and `cmd` are encoded into it." }
- { name: env, type: "map<string,string>", set: "create,update" }
- { name: ports, type: "string[]", set: "create,update", note: "Each entry is port/protocol, for example `8888/http` or `22/tcp`." }
- { name: registry, type: string, ref: registry-credential, set: "create,update", note: Credential id for a private image. }
- { name: disk, type: integer, unit: GB, set: "create,update", note: Container disk size. }
- { name: "mounts.network[].volumeId", type: string, ref: network-volume, set: create }
- { name: "mounts.network[].path", type: string, set: "create,update" }
- { name: mounts.persistent.size, type: integer, unit: GB, set: create }
- { name: cloud, type: enum, values: [SECURE, COMMUNITY], default: SECURE, set: create }
- { name: dataCenterIds, type: "string[]", ref: data-center, set: create, note: Placement preference. }
- { name: dataCenterId, type: string, ref: data-center, set: read, note: Assigned by the scheduler. }
- { name: startSsh, type: boolean, default: false, set: create, note: Never returned by GET. }
- { name: startJupyter, type: boolean, default: false, set: create, note: Never returned by GET. }
- { name: globalNetworking, type: boolean, set: "create,update" }
- { name: locked, type: boolean, set: update }
- { name: cost, type: number, unit: USD/hour, set: read }
- { name: cudaVersion, type: string, set: read, note: The CUDA version the host reports. Null means unknown. }
- { name: runtime, type: PodRuntime, set: read, note: "Null unless the pod is `RUNNING`." }
- { name: ssh, type: PodSsh, set: read }
- { name: cluster, type: PodCluster, set: read, note: Present only on Instant Cluster member pods. }
- { name: status, type: PodStatus, set: read }
- { name: actions, type: "PodAction[]", set: read, note: The actions valid in the current status. }
states:
field: status
values:
PROVISIONING: The pod is being allocated.
STARTING: The container is starting.
RUNNING: The container is up.
EXITED: The pod is stopped. Compute is released and the volume disk is kept.
ERROR: The container is in an unrecoverable error state.
TERMINATED: The pod has been permanently deleted.
transitions:
- { action: start, from: [EXITED, ERROR], to: RUNNING }
- { action: stop, from: [PROVISIONING, STARTING, RUNNING], to: EXITED }
- { action: restart, from: [RUNNING], to: RUNNING }
- { action: terminate, from: [PROVISIONING, STARTING, RUNNING, EXITED, ERROR], to: TERMINATED }
on_invalid_action: 409
evidence:
- { source: rest-v2-spec, ref: "POST /v2/pods/{id}/action", seen: "2026-09-28" }
relations:
- { type: runs_on, target: machine, cardinality: "1" }
- { type: runs_on, target: gpu-type, cardinality: "0..1" }
- { type: runs_on, target: cpu-flavor, cardinality: "0..1" }
- { type: located_in, target: data-center, cardinality: "1" }
- { type: uses, target: network-volume, cardinality: "0..1" }
- { type: uses, target: template, cardinality: "0..1" }
- { type: uses, target: registry-credential, cardinality: "0..1" }
- { type: requires, target: container-image, cardinality: "1" }
- { type: constrained_by, target: cloud-tier, cardinality: "1" }
- { type: constrained_by, target: gpu-availability, cardinality: "0..1" }
- { type: billed_by, target: pod-billing, cardinality: "1" }
- { type: constrained_by, target: credit-balance, cardinality: "1" }
rules:
- id: pod.gpu-xor-cpu
applies_to: ["field:gpu", "field:cpu"]
statement: >
Set exactly one of `gpu` or `cpu` when you create a pod. Leaving out `gpu`
does not create a CPU pod.
status: documented
evidence:
- { source: rest-v2-spec, ref: "CreatePodRequest; POST /v2/pods", seen: "2026-09-28" }
- id: pod.image-or-template
applies_to: ["field:image", "field:templateId"]
statement: >
`image` is required unless `templateId` is set. Fields in the request body
override the template's values.
status: documented
evidence:
- { source: rest-v2-spec, ref: POST /v2/pods, seen: "2026-09-28" }
- id: pod.create-is-async
applies_to: ["state:PROVISIONING", "state:RUNNING"]
statement: >
Create returns `201` while the pod is still `PROVISIONING`. Poll the pod
until `status` is `RUNNING` before you use it.
status: documented
evidence:
- { source: rest-v2-spec, ref: POST /v2/pods, seen: "2026-09-28" }
- id: pod.no-capacity-fallback
applies_to: ["field:gpu.id", "field:dataCenterIds"]
statement: >
Create places exactly the GPU type you ask for. It does not search for
capacity or fall back to another GPU. Read catalog availability first,
then try your candidates in order; a `400` means that candidate could not
be placed, so move to the next one.
on_violation: 400
status: documented
evidence:
- { source: rest-v2-spec, ref: POST /v2/pods (Checking what you can deploy), seen: "2026-09-28" }
- id: pod.network-volume-same-dc
applies_to: ["field:mounts.network[].volumeId", "field:dataCenterIds"]
statement: >
A network volume attaches only to a pod in the volume's data center. Set
`dataCenterIds` to the volume's `dataCenter`, or leave it empty.
status: documented
evidence:
- { source: rest-v2-spec, ref: NetworkMount.volumeId, seen: "2026-09-28" }
- id: pod.mount-kind-fixed
applies_to: ["field:mounts.network[].volumeId", "field:mounts.persistent.size"]
statement: >
The mount kind (network, persistent or none) is fixed when the pod is
created. A network volume cannot be swapped or detached later; only the
mount path can change. To use a different volume, create a new pod.
on_violation: 400
status: documented
evidence:
- { source: rest-v2-spec, ref: Mounts, seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/pods/storage/types", note: Network volumes must be attached at creation and cannot be detached. }
- id: pod.persistent-mount-limits
applies_to: ["field:mounts.persistent.size", "field:cpu"]
statement: >
A persistent (host-local) mount cannot be combined with a network mount,
is not allowed on CPU pods, has a 10 GB minimum, and is lost if the host
fails.
on_violation: 400
status: documented
evidence:
- { source: rest-v2-spec, ref: "Mounts; PersistentMount", seen: "2026-09-28" }
- id: pod.cuda-constraints
applies_to: ["field:gpu.allowedCudaVersions", "field:gpu.minCudaVersion"]
statement: >
`gpu.allowedCudaVersions` matches exactly, so a version that no machine
reports returns a capacity error instead of a fallback. It cannot be sent
together with a non-empty `gpu.minCudaVersion`. `gpu.minCudaVersion`
compares numerically, so 12.11 is above 12.2.
on_violation: 400 when both are sent
status: documented
evidence:
- { source: rest-v2-spec, ref: CreateGpuConfig, seen: "2026-09-28" }
- id: pod.cuda-discover-first
applies_to: ["field:gpu.allowedCudaVersions", "field:cudaVersion"]
statement: >
Before you deploy, check which CUDA versions a GPU type's hosts report
with `GET /v2/catalog/gpus?include=AVAILABILITY&product=POD`
(`cudaVersions`). After deploy, `cudaVersion` shows what the pod's host
reports.
status: documented
evidence:
- { source: rest-v2-spec, ref: "CreateGpuConfig.allowedCudaVersions; Pod.cudaVersion", seen: "2026-09-28" }
- id: pod.image-cuda-must-match-host
applies_to: ["field:image", "field:gpu.minCudaVersion"]
statement: >
An image built for a newer CUDA version than the host supports fails to
start, often with an "OCI runtime create failed" error. Set
`gpu.minCudaVersion` or `gpu.allowedCudaVersions` to match the image.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/manage-pods", note: CUDA compatibility note in the create steps. }
- { source: rest-v2-spec, ref: CreateGpuConfig, seen: "2026-09-28" }
- id: pod.visible-cores-exceed-allocated-vcpus
applies_to: ["field:gpu"]
statement: >
Inside a GPU pod, `nproc`, `os.cpu_count()` and the CPU affinity mask can
report every core on the machine, more than the vCPUs the pod rents. The
pod can only use its allocated share of CPU time. Libraries that size
their thread pools from the visible core count, such as PyTorch, OpenMP,
MKL, BLAS, data loader workers and video encoders, start too many threads
and get throttled, which makes CPU-bound work slower. Read the real limit
from the cgroup (`/sys/fs/cgroup/cpu.max`, or `cpu.cfs_quota_us` divided
by `cpu.cfs_period_us` under `/sys/fs/cgroup/cpu/`), then cap threads to
it: set `OMP_NUM_THREADS`, `MKL_NUM_THREADS` and similar, call
`torch.set_num_threads()`, and size worker pools to the vCPU count,
divided across processes when you run several. The throttle counters in
the cgroup's `cpu.stat` confirm the problem.
status: verified
evidence:
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/gotchas.md }
- { source: live-probe, ref: "runpod/pytorch:1.0.3-cu1281-torch291-ubuntu2404 on an RTX 2000 Ada secure cloud pod: nproc, os.cpu_count() and the affinity mask reported 48, the cgroup CPU limit was 5.1 cores, and torch.get_num_threads() defaulted to 24. 30 CPU 2048x2048 matmuls took 2.0 s at 24 threads (throttled in every period, about 33 s of throttled time) and 1.15 s at 5 threads (no throttling)", seen: "2026-10-26" }
- { source: live-probe, ref: "Same image on an RTX 2000 Ada secure cloud pod with cgroup v2: nproc reported 128 and cpu.max 2720000 100000 (27.2 cores). The cgroup-reading snippet in reference/gotchas.md returned 27 in bash and Python, and torch.get_num_threads() read 27 after it ran. On a CPU pod (ubuntu:22.04, cpu.max max, nproc 2) it returned 2", seen: "2026-10-26" }
- id: pod.cost-is-compute-only
applies_to: ["field:cost", "state:EXITED"]
statement: >
`cost` reads 0.0 when a pod is `EXITED`, but a stopped pod is still billed
for its volume disk. Terminate the pod when you no longer need its data.
status: documented
see: [credit-balance.low-balance-actions]
evidence:
- { source: rest-v2-spec, ref: Pod.cost, seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/pods/manage-pods", note: Stop a Pod section. }
- id: pod.global-networking-next-start
applies_to: ["field:globalNetworking"]
statement: >
Global networking needs an NVIDIA GPU and a data center whose
`globalNetwork` is true. Changing it takes effect on the next start or
restart, not while the pod runs.
status: documented
see: [data-center.global-network]
evidence:
- { source: rest-v2-spec, ref: "CreatePodRequest.globalNetworking; UpdatePodRequest.globalNetworking", seen: "2026-09-28" }
- id: pod.edits-apply-on-restart
applies_to: ["field:image", "field:args", "field:env", "field:ports", "field:disk", "field:registry"]
statement: >
Editing a running pod's `image`, `args`, `env`, `ports`, `disk` or
`registry` resets the pod, which erases everything outside `/workspace` or
a network volume. Set ports and environment variables at create time when
you can.
status: documented
see: [container-disk.what-survives]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/manage-pods", note: Update a Pod warning. }
- { source: rest-v2-spec, ref: "PATCH /v2/pods/{id}", seen: "2026-09-28", note: "Some changes apply immediately and others on the next start, as noted per field." }
- id: pod.env-pid1-only
applies_to: ["field:env"]
statement: >
Environment variables set at create time reach the container's main
process but not SSH login shells. Pass them explicitly in commands you run
over SSH, or export them in the shell profile.
status: verified
evidence:
- { source: live-probe, ref: "Pod created with startSsh; PUBLIC_KEY present in /proc/1/environ, absent from the SSH session environment", seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/on-pod-setup.md }
- id: pod.lock-blocks-changes
applies_to: ["field:locked"]
statement: >
A locked pod cannot be stopped or reset until you set `locked` to false.
It keeps running, and billing, while it is locked.
status: documented
evidence:
- { source: rest-v2-spec, ref: "Pod.locked; UpdatePodRequest.locked", seen: "2026-09-28" }
- id: pod.scheduled-stop
statement: >
REST v2 has no field that stops a pod after a set time. `runpodctl pod
create` takes `--stop-after` and `--terminate-after` for that. For a pod
created another way, run `runpodctl pod stop $RUNPOD_POD_ID` from inside
it, for example `sleep 2h; runpodctl pod stop $RUNPOD_POD_ID &`.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/manage-pods", note: "Stop a Pod section, scheduled stop example." }
- { source: rest-v2-spec, ref: "CreatePodRequest; UpdatePodRequest", seen: "2026-09-28", note: No stop or terminate timer fields. }
- { source: skill, path: plugins/runpod/skills/runpodctl/SKILL.md, note: pod create --stop-after and --terminate-after }
- id: pod.stopped-pod-keeps-host
applies_to: ["action:start", "state:EXITED", "field:cudaVersion"]
statement: >
A stopped pod stays tied to its machine and resumes on the same host. If
another user has taken the GPU, you can wait, start the pod with zero GPUs
to copy data off, or migrate it to a new machine. Keep important data on a
network volume so you can redeploy on any machine.
status: documented
see: [pod-migration.options, machine.pod-is-bound]
evidence:
- { source: rest-v2-spec, ref: Pod.cudaVersion, seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/pods/troubleshooting/zero-gpus" }
- id: pod.terminate-irreversible
applies_to: ["action:terminate", "state:TERMINATED"]
statement: >
Terminating a pod cannot be undone. It destroys the container and any
persistent mount; a network volume is only detached and keeps its data.
The pod no longer appears in the pod list.
status: documented
evidence:
- { source: rest-v2-spec, ref: "DELETE /v2/pods/{id}", seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/pods/manage-pods", note: Terminate a Pod warning. }
- id: pod.running-is-not-ready
applies_to: ["state:RUNNING", "field:runtime"]
statement: >
`RUNNING` means the container is up, not that the services inside it are
ready. A custom image may still be pulling, and servers such as JupyterLab
can take a few more minutes to start. Check the service itself before you
send it work.
status: documented
evidence:
- { source: rest-v2-spec, ref: PodStatus, seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/pods/configuration/expose-ports", note: Warning under Access your service. }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/gotchas.md }
- id: pod.status-vocabulary
applies_to: ["field:status"]
statement: >
REST v2 status values (`PROVISIONING`, `STARTING`, `RUNNING`, `EXITED`,
`ERROR`, `TERMINATED`) differ from GraphQL `desiredStatus` values
(`CREATED`, `RUNNING`, `RESTARTING`, `EXITED`, `PAUSED`, `DEAD`,
`TERMINATED`). Translate between them when you use both APIs.
status: documented
evidence:
- { source: rest-v2-spec, ref: PodStatus, seen: "2026-09-28" }
- { source: public-docs, url: "https://graphql-spec.runpod.io/#definition-PodStatus" }
- id: pod.cluster-pods-managed-elsewhere
applies_to: ["field:cluster"]
statement: >
The pod endpoints cannot update or delete a pod that belongs to an Instant
Cluster, and the pod list leaves such pods out unless you pass
`includeClusterPods=true`. Manage them through `/v2/clusters/{id}`.
status: documented
see: [instant-cluster.members-managed-by-cluster]
evidence:
- { source: rest-v2-spec, ref: "Pod.cluster; GET /v2/pods; PATCH /v2/pods/{id}", seen: "2026-09-28" }
concepts/public-endpoint.yaml
id: public-endpoint
name: Public Endpoint
kind: catalog
summary: >
A model API hosted by Runpod, for image, video, audio or text, that you call
with your API key and pay for per output, with nothing to deploy.
product: public-endpoints
is_a: null
part_of: runpod-platform
aliases: [public endpoints, pay-per-use model API]
surfaces:
mcp: [list-public-endpoints, run-endpoint, runsync-endpoint, get-job-status]
console: Public Endpoints
relations:
- { type: requires, target: api-key, cardinality: "1" }
- { type: billed_by, target: serverless-billing, cardinality: "1" }
- { type: constrained_by, target: api-rate-limit, cardinality: "1" }
rules:
- id: public-endpoint.no-deploy
statement: >
There is nothing to create. List the available models, then call
`https://api.runpod.ai/v2/<model-slug>/run` or `/runsync` with your API
key. Requests use the same job API as your own endpoints.
status: documented
see: [job.run-vs-runsync]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/public-endpoints/overview" }
- { source: public-docs, url: "https://docs.runpod.io/public-endpoints/requests" }
- { source: skill, path: plugins/runpod/skills/runpod-mcp/SKILL.md, note: Public endpoints section. }
- id: public-endpoint.pay-per-output
statement: >
Pricing is per output (per megapixel, per second of video, per million
tokens or per thousand characters), not per GPU time. Failed generations
are not charged.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/public-endpoints/overview", note: See the pricing section. }
- id: public-endpoint.spend-in-billing
statement: >
Public Endpoint spend appears in `GET /v2/billing/endpoints`.
status: documented
see: [serverless-billing.endpoints-route-is-public-endpoints]
evidence:
- { source: rest-v2-spec, ref: "EndpointBillingRecord; GET /v2/billing/endpoints", seen: "2026-09-28" }
- id: public-endpoint.output-urls-expire
statement: >
Output URLs (`image_url`, `video_url` and `audio_url`) expire after 7
days. Download anything you need to keep.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/public-endpoints/requests" }
concepts/README.md
# Concept file format
Each concept is one YAML file named `<id>.yaml`. To add one, follow the
`add-concept` skill in `.claude/skills/add-concept/SKILL.md` at the repo root. Every file is
public-facing: everything in it must be safe to show a customer and backed by a
public source.
The exact schema is `ontology/tools/src/schema.ts`. `pnpm validate` (run it in
`ontology/tools/`) checks every file against that schema, against the other files and against the
REST v2 spec.
```yaml
id: pod # kebab-case; matches the file name
name: Pod
kind: resource # see "Kinds"
summary: One or two customer-facing sentences.
product: pods # the console product it sits under, or null
is_a: null # kind-of parent (cpu-pod is_a pod)
part_of: null # the whole it belongs to (worker part_of serverless-endpoint)
aliases: [GPU pod, instance]
surfaces: # the same thing, named on every surface
rest_v2: { schema: Pod, paths: [/v2/pods, "/v2/pods/{id}"] }
graphql: { type: Pod, queries: [], mutations: [podFindAndDeployOnDemand] }
mcp: [create-pod, get-pod]
runpodctl: [pod create, pod get]
console: Pods
fields: # REST v2 names are canonical
- { name: "mounts.network[].volumeId", type: string, ref: network-volume, set: create }
states: # only for concepts with a lifecycle
field: status
values: { RUNNING: The container is running., EXITED: The pod is stopped. }
transitions:
- { action: stop, from: [RUNNING], to: EXITED }
on_invalid_action: 409
evidence: [...]
relations:
- { type: located_in, target: data-center, cardinality: "1" }
steps: # only for kind: process, at least two, in order
- id: stop
title: Stopping releases the GPUs
description: What happens in this step.
concepts: [pod, machine] # concepts the step involves
rules: [pod.stopped-pod-keeps-host] # existing rules that govern it
rules:
- id: pod.network-volume-same-dc # <concept-id>.<slug>, unique across all files
applies_to: ["field:mounts.network[].volumeId", field:dataCenterIds] # omit for concept-wide rules
statement: One to three sentences an agent can act on.
on_violation: 400 # optional: what the caller observes
status: documented
conflict: false
see: [network-volume.placement-immutable] # related rules on other concepts
evidence:
- { source: rest-v2-spec, ref: NetworkMount.volumeId, seen: 2026-09-28 }
- { source: public-docs, url: "https://docs.runpod.io/storage/network-volumes", note: Why it matters. }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/storage.md }
```
Run `pnpm format` in `ontology/tools/` after editing. It rewrites every file in this style
(key order, folded text wrapped at 80 columns, one-line list entries, quoting where YAML needs
it) without changing the data, and CI runs `pnpm format:check`.
## Kinds
| Kind | Meaning | Examples |
|---|---|---|
| `platform` | The root | runpod-platform |
| `resource` | Has its own id and CRUD operations in the API | pod, serverless-endpoint, template, network-volume |
| `component` | Part of a resource, with no CRUD of its own | worker, job, container-disk, exposed-port |
| `catalog` | Read-only reference data | gpu-type, data-center, cloud-tier, public-endpoint |
| `capability` | Something you do with a resource | pod-ssh-access, log-stream, flashboot |
| `billing` | Charges and credit | billing-record, pod-billing, savings-plan |
| `limit` | A cap or quota | api-rate-limit |
| `tool` | A Runpod SDK or CLI the customer runs | flash |
| `external` | Referenced by Runpod but not owned by it | container-image |
| `infrastructure` | Runpod hardware a resource runs on, not managed directly | machine |
| `process` | How something works, as ordered steps across concepts | pod-deployment |
## Relations
The hierarchy lives in `is_a` and `part_of`. `relations` uses a closed set of types:
| Type | Meaning |
|---|---|
| `requires` | Cannot exist without the target |
| `uses` | Optionally references the target |
| `located_in` | Pinned to the target |
| `runs_on` | Executes on the target |
| `billed_by` | Its charges appear under the target |
| `constrained_by` | Capped by a limit or a catalog entry |
A field with `ref:` is an edge too, called `field_ref` in the SQLite `edges`
view. A process links to every concept its steps involve with an `involves`
edge.
## Processes
A `process` explains how something works over time, such as what happens when
a pod is deployed, stopped and restarted. Its `steps` are in order. Each step
names the concepts it involves and the existing rules that govern it, so the
facts stay on their concepts and the process ties them into a sequence. Put a
rule on the process itself only when it is about the flow as a whole.
## Status, evidence and conflicts
| status | Meaning |
|---|---|
| `documented` | Stated in the REST v2 spec, the public docs or a public skill |
| `verified` | Observed on a live account |
Each evidence entry has a `source` and the keys a reader needs to find it.
`note` adds context but never stands alone. `seen` is the date the source was
checked.
| source | What it is | Requires |
|---|---|---|
| `rest-v2-spec` | The REST v2 OpenAPI spec (`https://api.runpod.io/v2/openapi.json`, vendored at `testdata/runpod-migrate/v2-openapi.json`) | `ref` |
| `public-docs` | docs.runpod.io and the public GraphQL reference | `url` |
| `skill` | A skill file in this repo, cited by `path` from the repo root | `path` |
| `live-probe` | A request made against a live account | `ref`, `seen` |
| `other` | Anything else public, such as a CLI's `--help` output | `ref` or `url` |
A `url`, and any URL in a rule's `statement` or `on_violation`, must be https on a
public source host: docs.runpod.io, docs.docker.com, api.runpod.io,
graphql-spec.runpod.io, runpod.io, www.runpod.io, huggingface.co, one of the
public repositories `runpod/runpod-plugins-official`, `runpod/runpodctl`,
`runpod/runpod-python`, `runpod/runpod-mcp`, `runpod/docs`, `runpod/containers`
and `runpod/flash` on github.com, or a
Runpod service (console.runpod.io, mcp.runpod.io, cli.runpod.net, api.runpod.ai,
`<endpoint-id>.api.runpod.ai`, rest.runpod.io, `<pod-id>-<port>.proxy.runpod.net`,
`s3api-<dc>.runpod.io`). In a statement, a placeholder such as `<pod-id>` or
`${POD_ID}` stands for one name. The lists are `PUBLIC_HOSTS`,
`PUBLIC_HOST_PATTERNS` and `PUBLIC_GITHUB_REPOS` in
`ontology/tools/src/validate.ts`; add a host there when citing a new public
source.
Set `conflict: true` only when two public sources disagree. The statement gives
both readings and tells the agent how to act, and the evidence cites both
sources. A live probe settles it.
## Writing style
- Write plain present tense in complete sentences. A statement is one to three sentences; a summary is one or two.
- Say what holds and, where it helps, what to do: "Set `dataCenterIds` to the volume's data center."
- Put code identifiers in backticks: fields, enum values, paths, environment variables, commands and headers.
- Write "Runpod". Capitalize "Pods" and "Serverless" only as product names. "Pod", "endpoint", "worker", "template" and "network volume" are lower-case generic terms.
- Keep statements durable. Avoid "currently", "today", "not yet" and similar words. When a fact may change, the evidence's `seen` date records when it was checked.
- Prefer positive claims. A claim that something does not exist needs the spec or docs as evidence, with a `seen` date.
- Avoid prices and other figures that change often, unless the spec or public docs state them as a limit.
## What the validator enforces
- Every file matches the schema, and its `id` matches its file name.
- Every `is_a`, `part_of`, relation target, field `ref`, `see:` and step reference resolves.
- Only a `process` has `steps`, and a process has at least two.
- Every `applies_to` names a field, state or action the concept declares.
- Every REST v2 schema, path and field root exists in the vendored spec.
- The `is_a` and `part_of` chains have no loops.
- No file names internal systems, internal source ids, private repositories or internal tools, or
contains an email address or an API key.
- Every evidence entry has the keys its source requires, and every `url`, and every URL in a rule's
statement or `on_violation`, is on a public source host.
- No file uses time-bound wording.
- Every `skill` evidence `path`, on rules and on `states`, points at a file that exists in this repo.
concepts/registry-credential.yaml
id: registry-credential
name: Container registry credential
kind: resource
summary: >
A saved username and password for a private container registry such as Docker
Hub, GHCR, GCR or ECR. Pods, templates and endpoints reference it by id to
pull private images.
product: templates
is_a: null
part_of: runpod-platform
aliases: [registry auth, containerRegistryAuthId, ContainerRegistryAuth, ECR credential, ECR delegation]
surfaces:
rest_v2: { schema: Registry, paths: [/v2/registries, "/v2/registries/{id}", /v2/registries/delegations, "/v2/registries/delegations/{id}"] }
graphql: { type: ContainerRegistryAuth, queries: [getEcrLogin, getDelegations], mutations: [saveRegistryAuth, registerDelegation, revokeDelegation, attemptDelegation] }
mcp: [create-registry, list-registries, get-registry, delete-registry, create-delegation, list-delegations, revoke-delegation]
runpodctl: [registry create, registry list, registry get, registry delete]
console: Settings
fields:
- { name: id, type: string, set: read }
- { name: name, type: string, set: create, required: true, note: "1-191 characters" }
- { name: username, type: string, set: create, required: true, note: write-only }
- { name: password, type: string, set: create, required: true, note: write-only }
relations: []
rules:
- id: registry-credential.write-only
applies_to: ["field:username", "field:password"]
statement: >
`username` and `password` are write-only and never returned by a read.
REST v2 has no update for a credential, so to change one, create a new
credential and point its users at it.
status: documented
evidence:
- { source: rest-v2-spec, ref: "CreateRegistryRequest; GET /v2/registries/{id}", seen: "2026-09-28" }
- id: registry-credential.delete-in-use
statement: >
Deleting a credential fails while a pod uses it to pull its image.
Templates are not checked: a template that references the credential loses
it (`registry` becomes null), and its next deploy of a private image fails
to pull.
on_violation: 400 Registry credential is in use by a pod and cannot be deleted
status: documented
evidence:
- { source: rest-v2-spec, ref: "DELETE /v2/registries/{id}", seen: "2026-09-28" }
- id: registry-credential.ecr-delegation
statement: >
For Amazon ECR, register a delegation with the repository's ECR resource
ARN (`POST /v2/registries/delegations`) instead of storing a password.
List delegations with `GET /v2/registries/delegations` and revoke one with
`DELETE /v2/registries/delegations/{id}`.
status: documented
evidence:
- { source: rest-v2-spec, ref: "CreateDelegationRequest; EcrDelegation", seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/api-reference-v2/registries/register-an-ecr-delegation", seen: "2026-09-28" }
concepts/reservation-pool.yaml
id: reservation-pool
name: Reserved capacity
kind: billing
summary: >
GPU capacity reserved for an account through Runpod's sales team instead of
taken On-Demand when deploying a pod.
product: billing
is_a: null
part_of: runpod-platform
aliases: [reservation, reserved capacity, Reserved pricing]
surfaces:
graphql: { type: Reservation }
console: Pods
fields: []
relations:
- { type: uses, target: gpu-type, cardinality: "1..n" }
rules:
- id: reservation-pool.not-self-serve
statement: >
Reserved capacity is arranged with Runpod's sales team, not created
through the API. The console's default pod deploy flow has no Reserved
option. In the legacy deploy flow, choosing the Reserved pricing option
points to Talk to sales instead of deploying.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/get-started", seen: "2026-10-06", note: The Reserved option appears only under the Legacy flow tab. }
concepts/runpod-platform.yaml
id: runpod-platform
name: Runpod (the platform)
kind: platform
summary: >
The GPU cloud where you rent Pods, deploy Serverless endpoints, call Public
Endpoints and attach storage. You manage it through the console, the REST API,
`runpodctl`, the SDKs and the Runpod MCP server.
product: null
is_a: null
part_of: null
aliases: [Runpod, Runpod console, Runpod API]
surfaces:
console: Home
relations: []
rules:
- id: runpod-platform.rest-base-url
statement: >
The REST v2 base URL is `https://api.runpod.io/v2`, for example
`https://api.runpod.io/v2/pods`, and the OpenAPI spec is served at
`https://api.runpod.io/v2/openapi.json`.
status: documented
evidence:
- { source: rest-v2-spec, ref: servers, seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/api-reference-v2/overview", seen: "2026-09-28" }
- id: runpod-platform.api-generations
statement: >
Build new integrations on REST v2. REST v1 (`https://rest.runpod.io/v1`)
and the GraphQL API (`https://api.runpod.io/graphql`) are deprecated and
have announced retirement dates, so move existing code to REST v2.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/api-reference-v2/migrate-from-v1", seen: "2026-09-28", note: "Gives November 15, 2026 as the REST v1 retirement date." }
- { source: public-docs, url: "https://docs.runpod.io/sdks/graphql/configurations", seen: "2026-09-28", note: Gives early 2027 as the GraphQL retirement date. }
- { source: skill, path: plugins/runpod/skills/runpod-migrate/SKILL.md }
- id: runpod-platform.brand-spelling
statement: >
Write the brand as "Runpod": one word, with only the R capitalized. The
CLI command is `runpodctl`, all lower-case.
status: documented
evidence:
- { source: other, ref: "runpod-plugins-official AGENTS.md, Conventions", url: "https://github.com/runpod/runpod-plugins-official/blob/main/AGENTS.md" }
concepts/runpod-secret.yaml
id: runpod-secret
name: Runpod Secret
kind: resource
summary: >
An encrypted string stored on your account or team, such as an API token or
password. You reference it from pod, endpoint or template environment
variables, and Runpod substitutes the value when the pod or worker boots.
product: account
is_a: null
part_of: runpod-platform
aliases: [secret, RUNPOD_SECRET_, secrets, stored secret, secret environment variable, store a token]
surfaces:
rest_v2: { schema: Secret, paths: [/v2/account/secrets, "/v2/account/secrets/{id}"] }
graphql: { queries: [secret, secrets], mutations: [secretCreate, secretValueUpdate, secretDescriptionUpdate, secretDelete] }
mcp: [create-secret, list-secrets, get-secret, update-secret, delete-secret]
console: Secrets
fields:
- { name: id, type: string, set: read }
- { name: name, type: string, set: create, required: true, note: "unique, immutable, 1-191 characters" }
- { name: value, type: string, set: "create,update", required: true, note: "write-only, under 16 MiB" }
- { name: description, type: string, set: "create,update", note: "up to 65,535 bytes; send an empty string to clear" }
- { name: createdAt, type: string, set: read }
- { name: valueLastUpdatedAt, type: string, set: read }
relations:
- { type: uses, target: template, cardinality: "0..n" }
rules:
- id: runpod-secret.reference-syntax
applies_to: ["field:name"]
statement: >
Set an environment variable's value to `{{ RUNPOD_SECRET_<name> }}` on a
pod, endpoint or template. Runpod substitutes the stored value when the
pod or worker boots, so the plain value never appears in the
configuration.
status: documented
evidence:
- { source: rest-v2-spec, ref: Secret, seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/pods/templates/secrets", seen: "2026-09-28" }
- id: runpod-secret.value-write-only
applies_to: ["field:value"]
statement: >
The value is never returned by the API or shown in the console after it is
set. Keep your own copy if you will need it again.
status: documented
evidence:
- { source: rest-v2-spec, ref: Secret, seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/pods/templates/secrets", seen: "2026-09-28" }
- id: runpod-secret.name-rules
applies_to: ["field:name"]
statement: >
`name` is unique in the account and cannot be changed. It is up to 191
characters, starts with a letter or underscore, and uses letters, digits
and `_.-/`. Names starting with `RUNPOD`, in any case, are rejected.
on_violation: 409 name taken; 400 or 422 invalid name
status: documented
evidence:
- { source: rest-v2-spec, ref: "CreateSecretRequest.name; POST /v2/account/secrets", seen: "2026-09-28" }
- id: runpod-secret.rotation-next-boot
applies_to: ["field:value"]
statement: >
A new value reaches pods and workers at their next boot. Running instances
keep the value they started with, so restart them to pick up a rotation.
status: documented
evidence:
- { source: rest-v2-spec, ref: "PATCH /v2/account/secrets/{id}", seen: "2026-09-28" }
- id: runpod-secret.patch-not-atomic
applies_to: ["field:value", "field:description"]
statement: >
When a `PATCH` sends both `value` and `description`, the value is applied
first. If the description update then fails, the response is an error but
the new value has already taken effect. Send the two fields in separate
requests when that matters.
status: documented
evidence:
- { source: rest-v2-spec, ref: "PATCH /v2/account/secrets/{id}", seen: "2026-09-28" }
- id: runpod-secret.delete-breaks-references
statement: >
Deleting a secret is permanent and is not blocked while the secret is in
use. Environment variables that reference it stop resolving, so update
those references before you delete it.
status: documented
evidence:
- { source: rest-v2-spec, ref: "DELETE /v2/account/secrets/{id}", seen: "2026-09-28" }
- id: runpod-secret.api-routes
statement: >
Manage secrets through REST v2 at `/v2/account/secrets`, through the
Runpod MCP secret tools, or in the console's Secrets page.
status: documented
evidence:
- { source: rest-v2-spec, ref: /v2/account/secrets, seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/pods/templates/secrets", seen: "2026-09-28" }
concepts/runpodctl.yaml
id: runpodctl
name: runpodctl
kind: tool
summary: >
Runpod's command-line tool for managing pods, Serverless endpoints, templates,
network volumes and models from a terminal, CI job or script. It also covers
Hub deploys, SSH keys, `send`/`receive` file transfer and `doctor` setup.
product: null
is_a: null
part_of: runpod-platform
aliases: [Runpod CLI, runpod CLI, runpodctl CLI]
surfaces:
runpodctl: [pod create, serverless create, template create, network-volume create, hub search, send, receive, doctor, update, version]
relations:
- { type: requires, target: api-key, cardinality: "1" }
- { type: uses, target: pod, cardinality: "0..n" }
- { type: uses, target: serverless-endpoint, cardinality: "0..n" }
- { type: uses, target: template, cardinality: "0..n" }
- { type: uses, target: network-volume, cardinality: "0..n" }
- { type: uses, target: hub-repo, cardinality: "0..n" }
rules:
- id: runpodctl.https-install
statement: >
Fetch the install script with `curl -fsSL https://cli.runpod.net` and pipe
it to `sudo bash`. Use an explicit HTTPS URL and keep certificate
verification enabled.
status: documented
evidence:
- { source: skill, path: plugins/runpod/skills/runpodctl/SKILL.md, note: Install command. }
- id: runpodctl.project-sdk-notice
statement: >
Use the Runpod SDK for serverless development workflows on every CLI
version. In builds containing the unreleased runpod/runpodctl#382,
`runpodctl project` invocations (including subcommands, aliases and flags)
and supported help forms such as `project --help`, `help project`, `-h
project` and `--help project` exit 1 with empty stdout and a plain-text
Runpod SDK notice on stderr. Treat this as a fatal error, surface the
notice verbatim and do not retry or require a JSON error code.
status: documented
evidence:
- { source: skill, path: plugins/runpod/skills/runpodctl/reference/output-and-errors.md, note: Project Runpod SDK notice. }
- id: runpodctl.project-released-compatibility
statement: >
In released versions through v3.0.1, `project` runtime errors print to
stdout and exit 0, so inspect the output rather than trusting a zero exit
status. Check `runpodctl version`, but verify the commit for source or
custom builds before assuming the runpod/runpodctl#382 notice contract
because their version string can match a released version. These
compatibility facts do not change the recommendation to use the Runpod
SDK.
status: documented
see: [runpodctl.project-sdk-notice]
evidence:
- { source: skill, path: plugins/runpod/skills/runpodctl/reference/output-and-errors.md, note: Released-version compatibility. }
- id: runpodctl.ssh-host-key-compatibility
statement: >
In v2.14.0 and earlier, the deprecated `project` and `exec python` SSH
paths disable host key checking. Host key pinning applies to `exec python`
in v2.15.0+ and to legacy `project` SSH in v2.15.0 through v3.0.1; in
builds containing runpod/runpodctl#382, `project` only emits the SDK
notice and opens no SSH connection. Check `runpodctl version` before
relying on pinning, and use the Runpod SDK for serverless development
regardless of CLI version.
status: documented
see: [runpodctl.project-sdk-notice]
evidence:
- { source: skill, path: plugins/runpod/skills/runpodctl/reference/command-reference.md, note: Pod host key trust. }
- id: runpodctl.update-first
statement: >
Older `runpodctl` builds lack newer flags and fail with confusing errors;
`--model-reference` needs v2.4.0 or later. Check `runpodctl version`, run
`runpodctl update` before starting, and keep one version for the whole
task.
status: documented
see: [model-store.attach-by-reference]
evidence:
- { source: skill, path: plugins/runpod/skills/runpodctl/SKILL.md, note: Install section. }
- id: runpodctl.help-is-authoritative
statement: >
The command surface comes from the installed binary. Read `runpodctl
<resource> <action> --help` for the flags a command accepts, rather than
relying on a list written for another version.
status: documented
evidence:
- { source: skill, path: plugins/runpod/skills/runpodctl/SKILL.md, note: Install section. }
- { source: skill, path: plugins/runpod/skills/runpod/SKILL.md, note: "These skills are a snapshot; the tools are the source of truth." }
- id: runpodctl.credential-resolution
statement: >
`runpodctl` reads the API key from `RUNPOD_API_KEY`, then a `.env` file,
then `~/.runpod/config.toml`. `rp login` (and the legacy `flash login`)
saves a key to that file, so one key serves `runpodctl` and Flash, and the
same key can be sent to the hosted MCP server as a Bearer token.
status: documented
see: [api-key.bearer-only]
evidence:
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/getting-started.md, note: The Runpod API key section. }
- id: runpodctl.beyond-crud
statement: >
Besides managing the same resources as the REST API, `runpodctl` browses
and deploys Hub repos, manages SSH keys, transfers files with `send` and
`receive`, sets up a machine with `doctor` and manages cached models.
status: documented
see: [file-transfer.send-receive]
evidence:
- { source: skill, path: plugins/runpod/skills/runpodctl/SKILL.md }
- { source: skill, path: plugins/runpod/skills/runpod/SKILL.md, note: The lanes table. }
concepts/savings-plan.yaml
id: savings-plan
name: Savings plan
kind: billing
summary: >
A discounted, prepaid commitment to GPU compute for a pod, bought upfront for
a fixed term of 3 or 6 months.
product: billing
is_a: null
part_of: runpod-platform
aliases: [committed GPU, savingsPlan, savingsPlans]
surfaces:
graphql: { type: SavingsPlan }
console: Savings plans
fields: []
relations:
- { type: requires, target: gpu-type, cardinality: "1" }
- { type: uses, target: pod, cardinality: "0..1" }
rules:
- id: savings-plan.not-in-rest-v2
statement: >
REST v2 does not expose savings plans; GraphQL's `Pod.savingsPlans` and
`adjustedCostPerHr` have no REST v2 equivalent. Manage plans in the
console or through GraphQL.
status: documented
evidence:
- { source: skill, path: plugins/runpod/skills/runpod-migrate/reference/breaking-changes.md }
- { source: rest-v2-spec, ref: "Pod; CreatePodRequest", seen: "2026-09-28" }
- id: savings-plan.discount-not-reservation
statement: >
A savings plan is a prepaid discount on GPU compute, not a hold on
hardware. Plans are non-refundable and expire on a fixed date. When the
pod is stopped, the plan applies to the next pod deployed with the same
GPU type.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/pricing", seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/accounts-billing/billing", seen: "2026-10-08" }
- id: savings-plan.scope
statement: >
Savings plans cover GPU compute only. Storage is billed at standard rates
and keeps accruing while the pod is stopped.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/pricing", seen: "2026-09-28" }
concepts/serverless-billing.yaml
id: serverless-billing
name: Serverless billing
kind: billing
summary: >
What serverless endpoints cost: per-second worker time at the GPU pool's rate,
plus storage. Active workers bill around the clock; flex workers bill from the
moment they start until they fully stop.
product: billing
is_a: billing-record
part_of: null
aliases: [flex pricing, active pricing, GPU-seconds, worker billing]
surfaces:
rest_v2: { schema: ServerlessBillingRecord, paths: [/v2/billing/serverless] }
mcp: [list-serverless-billing]
runpodctl: [billing serverless]
fields:
- { name: serverlessId, type: string, ref: serverless-endpoint, set: read }
- { name: startTime, type: string, set: read }
- { name: endTime, type: string, set: read }
- { name: totalAmount, type: number, unit: USD, set: read }
- { name: gpuAmount, type: number, unit: USD, set: read }
- { name: cpuAmount, type: number, unit: USD, set: read }
- { name: diskAmount, type: number, unit: USD, set: read }
- { name: feeAmount, type: number, unit: USD, set: read, note: Always 0. }
relations:
- { type: requires, target: serverless-endpoint, cardinality: "1" }
rules:
- id: serverless-billing.endpoints-route-is-public-endpoints
statement: >
Serverless spend is `GET /v2/billing/serverless`. `GET
/v2/billing/endpoints` is Public Endpoint spend. In REST v1,
`/billing/endpoints` meant serverless, and both REST v2 routes return 200,
so a migrated call silently reads the wrong product.
status: documented
evidence:
- { source: rest-v2-spec, ref: "GET /v2/billing/serverless; GET /v2/billing/endpoints", seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpod-migrate/reference/breaking-changes.md }
- { source: skill, path: plugins/runpod/skills/runpod-migrate/reference/rest-v1-to-v2.md }
- id: serverless-billing.per-endpoint-per-bucket
applies_to: ["field:serverlessId"]
statement: >
Records are one per endpoint per bucket. Pass `serverlessId` to narrow the
results to one endpoint.
status: documented
evidence:
- { source: rest-v2-spec, ref: ServerlessBillingRecord.serverlessId, seen: "2026-09-28" }
- id: serverless-billing.fee-always-zero
applies_to: ["field:feeAmount"]
statement: >
`feeAmount` is always 0. Platform charges are included in `gpuAmount` and
`cpuAmount`.
status: documented
evidence:
- { source: rest-v2-spec, ref: ServerlessBillingRecord.feeAmount, seen: "2026-09-28" }
- id: serverless-billing.active-workers-bill-idle
statement: >
Active workers, set with a minimum worker count of 1 or more, run and bill
continuously even when idle. Set the minimum to 0 to scale to zero with no
idle cost, and reset it after testing.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/pricing", seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/concepts.md }
- id: serverless-billing.what-is-billed
statement: >
A worker is billed from the moment it starts running until it fully stops,
rounded up to the nearest second. That covers start time (mostly the
handler loading the model), execution time, and the idle timeout after the
last request. Pulling the container image and downloading cached models
happen before the worker starts running and are not billed. Container
disk, billed in 5-minute intervals, and network volume storage are billed
on top.
status: documented
see: [endpoint-autoscaling.idle-timeout, container-disk.billing, worker.billing-by-state]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/pricing", seen: "2026-09-28" }
- id: serverless-billing.cut-start-time
statement: >
Start time is billed, so shortening it lowers cost. Use FlashBoot or model
caching to cut start time, and set an execution timeout to stop runaway
jobs.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/pricing", seen: "2026-09-28" }
concepts/serverless-endpoint.yaml
id: serverless-endpoint
name: Serverless endpoint
kind: resource
summary: >
A URL backed by autoscaling workers that run your container only while there
is work. Requests go either through a managed job queue or straight to the
workers' own HTTP servers, and the worker count scales between a minimum and a
maximum.
product: serverless
is_a: null
part_of: runpod-platform
aliases: [endpoint, Serverless, queue-based endpoint, load-balanced endpoint, load balancing endpoint]
surfaces:
rest_v2: { schema: Endpoint, paths: [/v2/serverless, "/v2/serverless/{id}"] }
graphql: { type: Endpoint, mutations: [saveEndpoint, deleteEndpoint] }
mcp: [create-endpoint, list-endpoints, get-endpoint, update-endpoint, delete-endpoint, endpoint-health, set-endpoint-gpus, purge-endpoint-queue]
runpodctl: [serverless create, serverless list, serverless get, serverless update, serverless delete, serverless health, serverless env, serverless model-status]
console: Serverless
fields:
- { name: name, type: string, set: "create,update", required: true }
- { name: type, type: enum, values: [QUEUE, LOAD_BALANCER], set: create, required: true, note: how requests reach workers }
- { name: gpu.pools, type: "string[]", ref: gpu-pool, set: "create,update", note: serverless GPU pool ids }
- { name: gpu.excludedTypes, type: "string[]", ref: gpu-type, set: "create,update", note: GPU type ids removed from the selected pools }
- { name: gpu.count, type: integer, default: 1, set: "create,update", note: GPUs per worker }
- { name: gpu.allowedCudaVersions, type: "string[]", set: "create,update" }
- { name: gpu.minCudaVersion, type: string, set: "create,update" }
- { name: cpu, type: "CreateCpuConfig[]", ref: cpu-flavor, set: "create,update", note: present only on CPU endpoints }
- { name: templateId, type: string, ref: template, set: "create,update" }
- { name: image, type: string, set: "create,update" }
- { name: env, type: object, set: "create,update" }
- { name: registry, type: string, ref: registry-credential, set: "create,update" }
- { name: mounts.network, type: "EndpointMount[]", ref: network-volume, set: "create,update", note: "list of {volumeId}, at most one per data center; no path, workers mount at /runpod-volume" }
- { name: networkVolumes, type: "string[]", ref: network-volume, set: "create,update", note: "deprecated; plain volume ids, superseded by mounts.network" }
- { name: dataCenterIds, type: "string[]", ref: data-center, set: "create,update", note: placement preference }
- { name: workers.min, type: integer, default: 0, set: "create,update" }
- { name: workers.max, type: integer, default: 3, set: "create,update" }
- { name: workers.idleTimeout, type: integer, unit: seconds, default: 10, set: "create,update", note: "1 to 3600" }
- { name: scaling.type, type: enum, values: [QUEUE_DELAY, REQUEST_COUNT], set: "create,update", required: true }
- { name: scaling.queueDelay, type: number, unit: seconds, set: "create,update", note: minimum 0.5 }
- { name: scaling.requestCount, type: integer, set: "create,update", note: minimum 1 }
- { name: timeout, type: integer, unit: ms, default: 300000, set: "create,update", note: "per-request execution timeout; the spec default, but a create that omits it reads back 0" }
- { name: flashboot, type: enum, values: ["OFF", FLASHBOOT, PRIORITY_FLASHBOOT], default: "OFF", set: "create,update" }
- { name: requestUrls, type: EndpointRequestUrls, set: read, note: "invoke URLs for the endpoint's type" }
relations:
- { type: uses, target: template, cardinality: "0..1" }
- { type: runs_on, target: gpu-pool, cardinality: "0..n" }
- { type: runs_on, target: cpu-flavor, cardinality: "0..n" }
- { type: uses, target: network-volume, cardinality: "0..n" }
- { type: located_in, target: data-center, cardinality: "0..n" }
- { type: uses, target: registry-credential, cardinality: "0..1" }
- { type: requires, target: container-image, cardinality: "1" }
- { type: billed_by, target: serverless-billing, cardinality: "1" }
- { type: constrained_by, target: gpu-availability, cardinality: "1" }
- { type: constrained_by, target: credit-balance, cardinality: "1" }
rules:
- id: serverless-endpoint.gpu-xor-cpu
applies_to: ["field:gpu.pools", "field:cpu"]
statement: >
Set exactly one of `gpu` or `cpu` on create; sending neither or both
returns 400. The compute family is fixed after that, so a PATCH that sends
`gpu` to a CPU endpoint or `cpu` to a GPU endpoint also returns 400.
on_violation: 400
status: documented
evidence:
- { source: rest-v2-spec, ref: "POST /v2/serverless; PATCH /v2/serverless/{id}", seen: "2026-09-28" }
- id: serverless-endpoint.image-or-template
applies_to: ["field:image", "field:templateId", "field:env"]
statement: >
`image` is required unless `templateId` is set. The template is resolved
into container settings at create time: fields in the request body
override it, and `env` merges key by key with the body's values winning.
`templateId` on a PATCH applies the template once and leaves no lasting
link.
status: documented
evidence:
- { source: rest-v2-spec, ref: "CreateEndpointRequest.templateId; UpdateEndpointRequest.templateId", seen: "2026-09-28" }
- id: serverless-endpoint.type-fixed
applies_to: ["field:type", "field:requestUrls"]
statement: >
`type` is required on create and cannot be changed later. A `QUEUE`
endpoint gets the job URLs (`run`, `runSync`, `status`, `stream`,
`cancel`, `retry`, `purgeQueue` and `health`). A `LOAD_BALANCER` endpoint
gets only its base URL and `health`, and its workers serve their own HTTP
paths.
status: documented
evidence:
- { source: rest-v2-spec, ref: "EndpointType; CreateEndpointRequest.type; EndpointRequestUrls", seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpod-mcp/SKILL.md, note: The routing type is fixed at creation. }
- id: serverless-endpoint.lb-worker-ports
applies_to: ["field:type", "field:env"]
statement: >
A worker on a `LOAD_BALANCER` endpoint listens on the port in `PORT`
(default 80) and answers health checks on `PORT_HEALTH` (defaults to
`PORT`) at `HEALTH_CHECK_PATH` (default `/ping`).
status: documented
evidence:
- { source: rest-v2-spec, ref: EndpointType, seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/serverless/load-balancing/build-a-worker", note: The example worker serves its health check at /ping. }
- { source: public-docs, url: "https://docs.runpod.io/serverless/load-balancing/overview", seen: "2026-10-14", note: Environment variable defaults. }
- id: serverless-endpoint.lb-readiness
applies_to: ["field:type"]
statement: >
A load-balanced worker's health route returns HTTP 204 while initializing
and HTTP 200 when ready to serve requests; other response codes mean
unhealthy. Keep model loading behind this readiness gate so traffic only
reaches a worker that can perform the requested work.
status: documented
see: [serverless-endpoint.lb-worker-ports]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/load-balancing/overview", seen: "2026-10-14", note: Health check response codes. }
- id: serverless-endpoint.lb-no-queue
applies_to: ["field:type"]
statement: >
A `LOAD_BALANCER` endpoint has no job queue. When every worker is busy,
new requests are dropped instead of waiting, so use a `QUEUE` endpoint
when requests must be buffered.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/load-balancing/overview", note: "The comparison table lists backpressure as dropping requests when overloaded, against queue buffering." }
- id: serverless-endpoint.pools-not-types
applies_to: ["field:gpu.pools", "field:gpu.excludedTypes"]
statement: >
`gpu.pools` takes serverless pool ids (the `pool` field of `GET
/v2/catalog/gpus`), not the GPU type ids that pods use.
`gpu.excludedTypes` takes GPU type ids and removes them from those pools.
A PATCH that sends `pools` without `excludedTypes` clears the exclusions.
on_violation: 400 when an excluded type is not in any selected pool
status: documented
see: [gpu-pool.pool-vs-type-ids]
evidence:
- { source: rest-v2-spec, ref: "BaseEndpointGpuConfig.pools; BaseEndpointGpuConfig.excludedTypes", seen: "2026-09-28" }
- id: serverless-endpoint.cuda-constraints
applies_to: ["field:gpu.allowedCudaVersions", "field:gpu.minCudaVersion"]
statement: >
A non-empty `allowedCudaVersions` and a `minCudaVersion` cannot be sent
together. A change to either applies as workers are replaced, not to
workers that are already running.
on_violation: 400
status: documented
evidence:
- { source: rest-v2-spec, ref: "CreateEndpointGpuConfig; UpdateEndpointGpuConfig", seen: "2026-09-28" }
- id: serverless-endpoint.network-volume-pins-dc
applies_to: ["field:mounts.network", "field:networkVolumes", "field:dataCenterIds"]
statement: >
Attaching network volumes limits workers to those volumes' data centers.
To spread workers across data centers, attach one volume per data center.
Volumes in different data centers do not sync with each other.
status: documented
evidence:
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/gotchas.md, note: Attach one volume per data center to spread workers. }
- id: serverless-endpoint.volumes-via-mounts-network
applies_to: ["field:mounts.network", "field:networkVolumes"]
statement: >
A v2 create or update attaches network volumes as `mounts.network`, a list
of `{volumeId}` entries with at most one volume per data center. There is
no `path`: every worker mounts the volume at `/runpod-volume`. A list you
send replaces the endpoint's current one, and `[]` detaches it. The flat
`networkVolumes` list of ids is deprecated, and if both are sent they must
agree exactly.
status: documented
evidence:
- { source: rest-v2-spec, ref: "CreateEndpointRequest; EndpointMountsInput; EndpointMount", seen: "2026-11-02" }
- id: serverless-endpoint.leave-placement-broad
applies_to: ["field:gpu.pools", "field:dataCenterIds"]
statement: >
List every GPU pool your workload can run on, and leave `dataCenterIds`
unset unless a network volume requires it. Pinning a scarce GPU to one
data center leaves workers throttled and jobs waiting in `IN_QUEUE`.
status: documented
evidence:
- { source: rest-v2-spec, ref: POST /v2/serverless (Checking what you can deploy), seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/endpoint-workflows.md }
- id: serverless-endpoint.accepts-jobs-before-workers
statement: >
A new endpoint accepts jobs right away, but it has no running workers
unless `workers.min` is above 0, so the first request pays for a cold
start. Create does not check that the image runs; a broken image shows up
later as jobs stuck in `IN_QUEUE`.
status: documented
evidence:
- { source: rest-v2-spec, ref: POST /v2/serverless, seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpodctl/SKILL.md, note: Describes how a broken image looks from the queue. }
- id: serverless-endpoint.diagnose-with-health
statement: >
Read the endpoint's health before waiting longer. Jobs queued with zero
workers for more than about two minutes point to a placement failure, not
a cold start. A `throttled` count above 0 means there is no capacity in
the selected pools and data centers. Ready workers with jobs stuck in
`IN_QUEUE` and nothing in progress point to a broken image.
status: documented
evidence:
- { source: skill, path: plugins/runpod/skills/runpodctl/SKILL.md, note: Serverless facts section. }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/gotchas.md }
- id: serverless-endpoint.update-effect-timing
applies_to: ["field:workers.min", "field:workers.max", "field:scaling.type", "field:timeout", "field:image", "field:env"]
statement: >
Changes to `workers`, `scaling` and `timeout` reach the autoscaler
promptly. Container changes (`image`, `env`, `args`, `disk`, `ports` and
`registry`) create a new release that rolls out as workers are replaced.
status: documented
see: [endpoint-release.rollout]
evidence:
- { source: rest-v2-spec, ref: "PATCH /v2/serverless/{id}", seen: "2026-09-28" }
- id: serverless-endpoint.defaults-by-layer
applies_to: ["field:workers.idleTimeout", "field:timeout", "field:flashboot", "field:workers.max"]
statement: >
Defaults depend on the tool that creates the endpoint. REST v2 create uses
`workers.max` 3, `idleTimeout` 10 seconds and `flashboot` `OFF`. The spec
documents a `timeout` default of 300 seconds, but a create that omits
`timeout` reads back 0, so always send it. The console uses a 5-second
idle timeout, a 600-second execution timeout and FlashBoot on. runpodctl
and the Flash SDK set their own defaults, so set these values explicitly.
status: documented
evidence:
- { source: rest-v2-spec, ref: "CreateEndpointRequest (workers, timeout and flashboot defaults)", seen: "2026-09-28" }
- { source: live-probe, ref: "POST /v2/serverless without timeout read back \"timeout\": 0", seen: "2026-10-29" }
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/endpoint-configurations", note: "Console defaults for idle timeout, execution timeout and FlashBoot." }
- { source: skill, path: plugins/runpod/skills/runpod-flash/reference/api.md, note: "The Flash SDK's defaults differ from the platform's." }
- id: serverless-endpoint.max-zero-stops-workers
applies_to: ["field:workers.max"]
statement: >
Setting `workers.max` to 0 stops every worker on the endpoint. Use it to
drain an endpoint before an update that must not mix versions, then
restore the previous value. Runpod also sets `workers.max` to 0 on an
endpoint that receives no requests for 7 days.
status: documented
see: [endpoint-autoscaling.idle-endpoint-scale-down]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/rolling-releases", note: Scale to zero before updating for strict version consistency. }
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/endpoint-configurations", note: Idle endpoint scale-down. }
- id: serverless-endpoint.delete-is-total
statement: >
Deleting an endpoint also deletes its bound template, terminates all of
its workers, and cancels queued and in-progress jobs. It cannot be undone.
status: documented
evidence:
- { source: rest-v2-spec, ref: "DELETE /v2/serverless/{id}", seen: "2026-09-28" }
concepts/serverless-handler.yaml
id: serverless-handler
name: Handler function
kind: capability
summary: >
The code inside a worker image that receives each job's input and returns its
output. A queue-based worker runs the Runpod SDK's handler loop, and a
load-balanced worker runs its own HTTP server.
product: serverless
is_a: null
part_of: worker
aliases: [handler, handler.py, runpod.serverless.start, concurrent handler, worker contract]
relations:
- { type: requires, target: container-image, cardinality: "1" }
rules:
- id: serverless-handler.errors-become-failed
statement: >
When the handler raises an exception, the Runpod SDK marks the job
`FAILED` and returns the exception details in the job's results; the
caller does not get an HTTP 5xx. Return `{"error": "..."}` for a graceful
failure, and avoid `try`/`except` blocks that hide errors.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/workers/handler-functions", note: Error handling. }
- id: serverless-handler.lb-vs-queue
statement: >
A load-balanced worker runs its own HTTP server on the port in `PORT`
(default 80), with a health check on `PORT_HEALTH` at `HEALTH_CHECK_PATH`
(default `/ping`). A queue-based worker calls `runpod.serverless.start`
and receives jobs from the queue.
status: documented
see: [serverless-endpoint.lb-worker-ports]
evidence:
- { source: rest-v2-spec, ref: EndpointType, seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/serverless/load-balancing/overview" }
- id: serverless-handler.generator-for-streaming
statement: >
Streaming needs a generator handler that yields output. Pass
`return_aggregate_stream=True` to `runpod.serverless.start` to also return
the full list from `/run`, `/runsync` and `/status`. A single chunk is
capped at 1 MB.
status: documented
evidence:
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/endpoint-workflows.md, note: Streaming section. }
- id: serverless-handler.load-model-once
statement: >
Load the model at module level or once per worker, not inside the handler,
so warm workers reuse it between jobs.
status: documented
evidence:
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/gotchas.md, note: Cold starts and timeouts. }
- { source: skill, path: plugins/runpod/skills/runpod-flash/reference/patterns.md, note: Loading ML models. }
- id: serverless-handler.input-shape
statement: >
The handler receives the job's `input` object. The worker SDK rejects a
job with an empty `input` ("Job has missing field(s): id or input").
status: documented
see: [job.input-envelope]
evidence:
- { source: skill, path: plugins/runpod/skills/runpod-flash/reference/runpod-flash-legacy.md, note: Dev server loop. }
- id: serverless-handler.local-serve-flag
statement: >
Run `python handler.py --rp_serve_api` to serve a handler locally as an
HTTP API for testing. Add `--rp_api_port` to change the port.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/development/local-testing" }
concepts/spot-pod.yaml
id: spot-pod
name: Spot (interruptible) pod
kind: resource
summary: >
A discounted, interruptible pod rented through the GraphQL API with a per-GPU
bid. It can be stopped when the GPU is needed for another rental.
product: pods
is_a: pod
part_of: null
aliases: [spot, interruptible pod, bid pod]
surfaces:
graphql: { type: Pod, mutations: [podRentInterruptable, podBidResume] }
relations:
- { type: runs_on, target: gpu-type, cardinality: "1" }
- { type: located_in, target: data-center, cardinality: "1" }
- { type: constrained_by, target: gpu-availability, cardinality: "1" }
- { type: billed_by, target: pod-billing, cardinality: "1" }
rules:
- id: spot-pod.not-in-rest-v2
statement: >
REST v2 has no spot, interruptible or bid field, so REST v2, the Runpod
MCP and `runpodctl` create on-demand pods only. Spot pods are created with
the GraphQL mutation `podRentInterruptable` (with `bidPerGpu`) and resumed
with `podBidResume`. When migrating to REST v2, keep these calls on
GraphQL or move them to on-demand pods.
status: documented
evidence:
- { source: rest-v2-spec, ref: CreatePodRequest, seen: "2026-09-28", note: "No spot, interruptible or bid field." }
- { source: public-docs, url: "https://graphql-spec.runpod.io/#mutation-podRentInterruptable" }
- { source: skill, path: plugins/runpod/skills/runpod-migrate/reference/breaking-changes.md }
- id: spot-pod.checkpoint-work
statement: >
A spot pod can be stopped at any time, and a stop clears the container
disk. Checkpoint work to the volume disk or a network volume.
status: documented
see: [container-disk.what-survives]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/maintenance-and-outages", note: "Data only on container storage is lost if the pod is interrupted, restarted, stopped or terminated." }
- { source: public-docs, url: "https://docs.runpod.io/pods/storage/types" }
concepts/ssh-key.yaml
id: ssh-key
name: SSH key
kind: resource
summary: >
An SSH public key registered on the account. Registered keys are injected into
pods created with `startSsh` and authenticate SSH connections to them.
product: account
is_a: null
part_of: runpod-platform
aliases: [SSH public key, authorized key, PUBLIC_KEY]
surfaces:
rest_v2: { schema: SshKeys, paths: [/v2/account/ssh-keys] }
mcp: [get-ssh-keys, update-ssh-keys]
runpodctl: [ssh list-keys, ssh add-key, ssh remove-key]
console: Settings (SSH public keys)
fields:
- { name: keys, type: "string[]", set: update, required: true, note: "One `authorized_keys` line per entry: `<type> <base64-key> [comment]`." }
relations: []
rules:
- id: ssh-key.put-replaces-all
applies_to: ["field:keys"]
statement: >
`PUT /v2/account/ssh-keys` replaces the full set. Keys missing from the
request are removed, and `[]` removes them all. To add one key, read the
current set and send it back with the new key appended.
status: documented
evidence:
- { source: rest-v2-spec, ref: "PUT /v2/account/ssh-keys; UpdateSshKeysRequest.keys", seen: "2026-09-28" }
- id: ssh-key.new-pods-only
applies_to: ["field:keys"]
statement: >
Key changes apply to pods created afterwards with `startSsh`. Running pods
keep the keys they were created with.
status: documented
see: [pod-ssh-access.start-ssh-needs-keys]
evidence:
- { source: rest-v2-spec, ref: PUT /v2/account/ssh-keys, seen: "2026-09-28" }
concepts/template.yaml
id: template
name: Template
kind: resource
summary: >
A saved container configuration (image, command, disk, ports, environment
variables, registry credential and startup options) that pods and Serverless
endpoints copy when they are created.
product: templates
is_a: null
part_of: runpod-platform
aliases: [Pod template, PodTemplate, Serverless template, official template, community template]
surfaces:
rest_v2: { schema: Template, paths: [/v2/templates, "/v2/templates/{id}", /v2/catalog/templates] }
graphql: { type: PodTemplate, mutations: [saveTemplate, deleteTemplate] }
mcp: [create-template, list-templates, get-template, update-template, delete-template, list-public-templates]
runpodctl: [template create, template list, template get, template search, template update, template delete]
console: Templates
fields:
- { name: id, type: string, set: read }
- { name: name, type: string, set: "create,update", required: true }
- { name: image, type: string, ref: container-image, set: "create,update", required: true }
- { name: args, type: string, set: "create,update", note: container command in stored form }
- { name: disk, type: integer, unit: GB, set: "create,update", note: "container disk, minimum 1" }
- { name: env, type: object, set: "create,update" }
- { name: ports, type: "string[]", set: "create,update", note: "port/protocol, e.g. 8888/http" }
- { name: registry, type: string, ref: registry-credential, set: "create,update" }
- { name: mounts.persistent.size, type: integer, unit: GB, set: "create,update" }
- { name: mounts.persistent.path, type: string, set: "create,update" }
- { name: serverless, type: boolean, default: false, set: "create,update" }
- { name: public, type: boolean, default: false, set: "create,update" }
- { name: category, type: enum, values: [CPU, NVIDIA, AMD], default: NVIDIA, set: "create,update" }
- { name: startSsh, type: boolean, default: true, set: "create,update" }
- { name: startJupyter, type: boolean, default: true, set: "create,update" }
- { name: allowedCudaVersions, type: "string[]", set: "create,update" }
relations:
- { type: requires, target: container-image, cardinality: "1" }
- { type: uses, target: registry-credential, cardinality: "0..1" }
- { type: uses, target: runpod-secret, cardinality: "0..n" }
rules:
- id: template.snapshot-at-create
applies_to: ["field:image", "field:env"]
statement: >
A pod or endpoint copies the settings of the template you create it from,
and later edits to that template do not change it. A pod created through
REST v2 with `templateId` keeps no link to the template, so its `template`
field stays null. An endpoint also gets its own bound template, deleted
with the endpoint; changing the endpoint's container settings updates that
one and starts a rolling release.
status: documented
see: [endpoint-release.rollout]
evidence:
- { source: rest-v2-spec, ref: "PATCH /v2/templates/{id}; CreatePodRequest.templateId", seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpod-migrate/reference/graphql-to-v2.md }
- id: template.body-overrides-template
applies_to: ["field:env", "field:allowedCudaVersions"]
statement: >
When you create a pod from a template, fields in the create body override
the template's. `env` is merged per key, with body values winning. Sending
either CUDA field replaces the template's CUDA constraint entirely.
status: documented
evidence:
- { source: rest-v2-spec, ref: CreatePodRequest.templateId, seen: "2026-09-28" }
- id: template.delete-blocked-in-use
statement: >
Deleting a template fails while a pod references it in its `template`
field or a Serverless endpoint is bound to it. Editing a template that is
in use is allowed.
on_violation: 400 Template is in use by a pod or serverless endpoint and cannot be deleted
status: documented
evidence:
- { source: rest-v2-spec, ref: "DELETE /v2/templates/{id}", seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpod-migrate/reference/rest-v1-to-v2.md }
- id: template.owner-only-writes
statement: >
`GET /v2/templates/{id}` reads your own templates and public catalog
templates, but only the owner can update or delete a template. `PATCH` or
`DELETE` on a catalog template returns 404.
on_violation: 404
status: documented
evidence:
- { source: rest-v2-spec, ref: "GET, PATCH and DELETE /v2/templates/{id}", seen: "2026-09-28" }
- id: template.catalog-sources
statement: >
`GET /v2/catalog/templates` returns at most 100 templates, with no
pagination. The `source` parameter selects `official` (the default,
curated by Runpod), `verified` or `community`. Pod and Serverless
templates are mixed in the results; the `serverless` flag tells them
apart.
status: documented
evidence:
- { source: rest-v2-spec, ref: GET /v2/catalog/templates, seen: "2026-09-28" }
- id: template.catalog-registry-hidden
applies_to: ["field:registry"]
statement: >
`registry` is always null on templates you do not own, so a catalog
template never exposes its registry credential.
status: documented
evidence:
- { source: rest-v2-spec, ref: GET /v2/catalog/templates, seen: "2026-09-28" }
- id: template.types-and-support
applies_to: ["field:public"]
statement: >
Official templates are curated, tested and supported by Runpod. Community
templates are made by users and are supported only by their creators and
the community Discord. A template you create is private unless you set
`public`, which lists it for all Runpod users in Explore.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/pods/templates/overview", seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/pods/templates/manage-templates", seen: "2026-09-28" }
- { source: rest-v2-spec, ref: Template.public, seen: "2026-09-28" }
- id: template.no-network-mount
applies_to: ["field:mounts.persistent.size"]
statement: >
A template can carry only a single persistent mount. A network volume
cannot be set on a template and is rejected, so attach network volumes
when you create the pod or endpoint.
on_violation: 422
status: documented
evidence:
- { source: rest-v2-spec, ref: TemplateMounts, seen: "2026-09-28" }
- id: template.serverless-flag
applies_to: ["field:serverless"]
statement: >
A template with `serverless: true` is for Serverless workers and cannot be
used to create a pod.
on_violation: 422
status: documented
evidence:
- { source: rest-v2-spec, ref: "CreatePodRequest.templateId; Template.serverless", seen: "2026-09-28" }
- id: template.category-console-only
applies_to: ["field:category"]
statement: >
`category` (`CPU`, `NVIDIA` or `AMD`; default `NVIDIA`) only groups the
template in the console. It does not affect hardware selection, scheduling
or billing.
status: documented
evidence:
- { source: rest-v2-spec, ref: TemplateCategory, seen: "2026-09-28" }
- id: template.startup-injection
applies_to: ["field:startSsh", "field:startJupyter"]
statement: >
`startSsh` and `startJupyter` default to true. They inject `PUBLIC_KEY`
(your registered SSH keys) and a generated `JUPYTER_PASSWORD` unless `env`
already sets them. `startSsh` does nothing when the account has no SSH
keys registered, and only images that follow this convention act on either
variable.
status: documented
evidence:
- { source: rest-v2-spec, ref: "CreateTemplateRequest.startSsh; CreateTemplateRequest.startJupyter", seen: "2026-09-28" }
- { source: live-probe, ref: "POST /v2/pods with startSsh and startJupyter on runpod/base:1.0.2-ubuntu2204; both variables injected and the registered keys reached authorized_keys", seen: "2026-09-28" }
- id: template.cuda-ignored-on-cpu
applies_to: ["field:allowedCudaVersions"]
statement: >
`allowedCudaVersions` applies to GPU pods and Serverless endpoints created
from the template. An empty list means any version, and CPU pods ignore
the field.
status: documented
evidence:
- { source: rest-v2-spec, ref: Template.allowedCudaVersions, seen: "2026-09-28" }
concepts/volume-cache.yaml
id: volume-cache
name: VolumeCache
kind: capability
summary: >
A helper in the Runpod Python SDK that mirrors local directories, such as a
model cache, onto the endpoint's network volume. On cold start it copies the
mirror onto the worker's container disk, so the model loads from local disk
instead of the network mount.
product: serverless
is_a: null
part_of: serverless-handler
aliases: [volume cache, network-volume warm cache, runpod.serverless.VolumeCache, hydrate, sync]
relations:
- { type: requires, target: network-volume, cardinality: "1" }
- { type: uses, target: container-disk, cardinality: "1" }
rules:
- id: volume-cache.copy-to-local-disk
statement: >
`from runpod.serverless import VolumeCache` needs the Runpod Python SDK
`runpod>=1.12.0` in the worker. Wrap the model load in `with
VolumeCache(dirs=[<local cache dir>]):`. On entry it copies cached files
from the volume into the local directories (`hydrate()`), and on exit it
copies new or changed files back to the volume (`sync()`).
status: documented
see: [network-volume.mount-path]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/development/volume-cache", seen: "2026-10-14" }
- { source: other, url: "https://github.com/runpod/runpod-python/blob/main/CHANGELOG.md", seen: "2026-10-14", note: "1.12.0 adds the network-volume warm cache (VolumeCache)." }
- id: volume-cache.local-dirs-not-the-mount
statement: >
List directories on the container disk in `dirs`, not paths under
`/runpod-volume`. The volume holds only the mirror, so the model loads
from local disk, and the files are re-hydrated after the container disk is
wiped at scale-down.
status: documented
see: [container-disk.what-survives]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/development/volume-cache", seen: "2026-10-14", note: VolumeCache warms local directories using an attached network volume. }
- id: volume-cache.opt-in-handler-code
statement: >
`VolumeCache` runs only where your handler code wraps it, so it applies to
your own handler or Flash code. A prebuilt worker image that does not call
it reads its model path directly, and a network volume attached to that
image serves the model from the mount.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/development/volume-cache", seen: "2026-10-14", note: "You add it explicitly by wrapping the code that populates your cache, so nothing runs until you opt in." }
- id: volume-cache.best-effort-hides-failures
statement: >
With the default `best_effort=True`, any cache error is logged and
swallowed, and the worker falls back to a normal cold start. A failing
cache therefore shows up as a slow cold start, not an error. Pass
`best_effort=False` while debugging so failures raise.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/development/volume-cache", seen: "2026-10-14" }
- id: volume-cache.symlinks-not-copied
statement: >
`VolumeCache` never copies symbolic links, and it skips `refs/` and
`.no_exist/` directories and `*.lock` files. The `snapshots/` entries of a
Hugging Face hub cache are symlinks, so after a hydrate let
`huggingface_hub` re-link the snapshot with a Hub metadata call. Do not
set `HF_HUB_OFFLINE=1` on a cache that `VolumeCache` restores.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/development/volume-cache", seen: "2026-10-14", note: Symbolic links are never followed or copied. }
- { source: other, url: "https://huggingface.co/docs/huggingface_hub/guides/manage-cache", seen: "2026-10-14", note: Snapshot files are symlinks to blobs. }
- { source: other, url: "https://github.com/runpod/runpod-python/blob/v1.12.0/runpod/serverless/utils/rp_volume_cache.py#L74", seen: "2026-10-15", note: Paths containing /refs/ or /.no_exist/ are excluded from sync. }
- id: volume-cache.container-disk-holds-cache
statement: >
The worker's container disk holds the whole local cache, because both the
hydrate and a fallback Hub download write there. Size the container disk
for the full model plus runtime headroom. Flash SDK resources default to
30 GB on GPU, so raise it with `container_disk_gb=` (legacy `runpod-flash`
endpoints default to 64 GB and use
`template=PodTemplate(containerDiskInGb=...)`), and on a template use
`runpodctl template create --container-disk-in-gb`.
status: documented
see: [container-disk.what-survives]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/development/volume-cache", seen: "2026-10-14", note: VolumeCache restores cached files into the local directories. }
- { source: public-docs, url: "https://docs.runpod.io/sdk/configuration/parameters", seen: "2026-10-16", note: container_disk_gb defaults to 30 on GPU and 10 on CPU. }
- { source: skill, path: plugins/runpod/skills/runpod-flash/reference/runpod-flash-legacy.md, note: PodTemplate containerDiskInGb defaults to 64. }
- { source: skill, path: plugins/runpod/skills/runpodctl/SKILL.md }
- id: volume-cache.silent-without-namespace
statement: >
`VolumeCache` needs a mounted volume and a `namespace`, which defaults to
`RUNPOD_ENDPOINT_ID`. Without either, `available` is False and `hydrate()`
and `sync()` do nothing and log nothing. Serverless workers set
`RUNPOD_ENDPOINT_ID`, but a Pod or a local run does not, so pass
`namespace=` there or check `available` before relying on the cache.
status: documented
evidence:
- { source: other, url: "https://github.com/runpod/runpod-python/blob/v1.12.0/runpod/serverless/utils/rp_volume_cache.py#L87", seen: "2026-10-15", note: "namespace falls back to RUNPOD_ENDPOINT_ID, then to an empty string." }
- { source: other, url: "https://github.com/runpod/runpod-python/blob/v1.12.0/runpod/serverless/utils/rp_volume_cache.py#L374", seen: "2026-10-15", note: "hydrate() and sync() return early when not available, with no log line." }
- id: volume-cache.mirror-sizing
statement: >
The mirror lives at `/runpod-volume/.cache/<namespace>` and the
`namespace` defaults to `RUNPOD_ENDPOINT_ID`. Size the network volume for
the mirror plus anything else on it. Large files that you delete or rename
locally stay on the volume, so usage grows as you swap model versions.
status: documented
see: [network-volume.size-grow-only]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/development/volume-cache", seen: "2026-10-14" }
- id: volume-cache.first-population
statement: >
Workers that cold-start together against an empty mirror each download the
model and sync a full copy back, with no coordination. Populate the mirror
with one worker first. The default `sync()` runs on a background thread,
so call `sync(background=False)` when the process may be killed before a
normal interpreter shutdown.
status: documented
see: [network-volume.concurrent-writes]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/development/volume-cache", seen: "2026-10-14" }
- id: volume-cache.restore-beats-mount-load
statement: >
With Qwen2.5-1.5B (3.1 GB), a cold start that restored the model with
`VolumeCache` and loaded it from local disk measured faster than loading
it straight from the network volume or downloading it: restore 5.2 s plus
load 3.9 s, against 11 s loading from the mount and 16-21 s downloading
from Hugging Face and loading. Smaller models or a faster mount can narrow
or flip the gap, so time your own model.
status: verified
evidence:
- { source: live-probe, ref: "Runpod SDK app, Qwen2.5-1.5B-Instruct on L4/RTX 4090 workers: fresh-worker VolumeCache hydrate 5.2 s + load 3.9 s (3.10 GB restored); load from /runpod-volume 10.7-11.6 s; Hugging Face download + load 16.3-21.4 s", seen: "2026-10-17" }
- id: volume-cache.global-volume
statement: >
`VolumeCache` mirrors onto a global volume mounted at its `volume_path`
(`/runpod-volume` on a GPU endpoint). Use `runpod>=2.0.0`. Before
2.0.0rc2, files of 256 KiB or more failed to copy onto a global volume.
Before 2.0.0, a restore from a global volume could drop large files. Both
failures are swallowed, so the worker downloads the model again. Restores
that copied whole files took 26-51 s for 3.1 GB, against 5.2 s from a
network volume. 2.0.0 restores files in parallel byte ranges and retries
failed reads.
status: verified
see: [volume-cache.best-effort-hides-failures]
evidence:
- { source: live-probe, ref: "Runpod SDK app, Qwen2.5-1.5B-Instruct, GlobalVolume mounted at /runpod-volume: fresh-worker hydrate restored all 3.10 GB in 26.2 s, 46.6 s and 51.4 s; hydrate plus model load 30-55 s", seen: "2026-10-17" }
- { source: other, url: "https://github.com/runpod/runpod-python/blob/fc4a6c73c3915625d65df63416db835d0b927b8f/runpod/serverless/utils/rp_volume_cache.py#L292", seen: "2026-10-22", note: "copy2 copies mtime and mode after the data; the global volume rejects utime/chmod, so big files failed to copy. The fix copies the data and treats the metadata step as optional." }
- { source: other, url: "https://github.com/runpod/runpod-python/blob/fc4a6c73c3915625d65df63416db835d0b927b8f/runpod/serverless/utils/rp_volume_cache.py#L389", seen: "2026-10-22", note: "Hydrate copies big files as parallel byte ranges and retries ENOMEM, EIO and EAGAIN reads, which a global volume returns under whole-file parallel copies." }
- id: volume-cache.namespace-per-endpoint
statement: >
The mirror is keyed by `namespace`, which defaults to the endpoint ID.
Each Runpod SDK dev session creates new endpoints, so it starts from an
empty mirror and leaves its copy on the volume. Pass a fixed `namespace=`
to share one mirror across dev sessions and deploys.
status: documented
see: [volume-cache.mirror-sizing, flash.dev-session]
evidence:
- { source: other, url: "https://github.com/runpod/runpod-python/blob/v1.12.0/runpod/serverless/utils/rp_volume_cache.py#L87", seen: "2026-10-15", note: namespace falls back to RUNPOD_ENDPOINT_ID. }
- { source: public-docs, url: "https://docs.runpod.io/sdk/apps/dev-sessions", seen: "2026-10-16", note: Dev sessions create temporary endpoints and delete them when the session ends. }
concepts/worker.yaml
id: worker
name: Serverless worker
kind: component
summary: >
One running copy of an endpoint's container. Workers start on demand, process
jobs or HTTP requests, and scale down when idle.
product: serverless
is_a: null
part_of: serverless-endpoint
aliases: [flex worker, active worker, serverless worker]
surfaces:
rest_v2: { schema: Worker, paths: ["/v2/serverless/{id}/workers", "/v2/serverless/{id}/workers/{workerId}/logs"] }
graphql: { type: WorkerState }
mcp: [list-endpoint-workers, stream-worker-logs]
runpodctl: [serverless logs, serverless health]
console: Serverless > Workers
fields:
- { name: id, type: string, set: read }
- { name: status, type: WorkerStatus, set: read }
- { name: isStale, type: boolean, set: read, note: running an older endpoint configuration }
- { name: version, type: integer, set: read, note: endpoint configuration version }
- { name: gpuCount, type: integer, set: read }
- { name: gpuTypeId, type: string, ref: gpu-type, set: read, note: null until placed }
- { name: dataCenterId, type: string, ref: data-center, set: read, note: null until placed }
- { name: image, type: string, set: read }
- { name: uptimeSeconds, type: integer, set: read }
- { name: startedAt, type: string, set: read }
states:
field: status
values:
INITIALIZING: Starting up; not ready for jobs.
IDLE: Ready and waiting for jobs.
RUNNING: Processing a job.
THROTTLED: Waiting for compute capacity on the host.
UNHEALTHY: Failing health checks.
evidence:
- { source: rest-v2-spec, ref: WorkerStatus, seen: "2026-09-28" }
relations:
- { type: runs_on, target: gpu-type, cardinality: "0..1" }
- { type: runs_on, target: cpu-flavor, cardinality: "0..1" }
- { type: located_in, target: data-center, cardinality: "1" }
- { type: requires, target: container-image, cardinality: "1" }
- { type: billed_by, target: serverless-billing, cardinality: "1" }
rules:
- id: worker.status-vocabulary
applies_to: ["field:status", "field:isStale"]
statement: >
REST v2 reports a worker as `INITIALIZING`, `IDLE`, `RUNNING`, `THROTTLED`
or `UNHEALTHY`. The console's "Outdated" state is not a REST v2 status: a
worker on an older configuration has `isStale` set to true, derived by
comparing its `version` with the response's `endpointVersion`.
status: documented
evidence:
- { source: rest-v2-spec, ref: "WorkerStatus; Worker.isStale", seen: "2026-09-28" }
- { source: public-docs, url: "https://docs.runpod.io/serverless/workers/overview", note: "The console's worker states table lists Outdated." }
- id: worker.status-derived
applies_to: ["field:status"]
statement: >
`status` combines the worker's lifecycle with the live job queue. When the
queue view is unavailable, `status` and the summary counts fall back to
the lifecycle alone. Workers that have scaled down are not listed.
status: documented
evidence:
- { source: rest-v2-spec, ref: "GET /v2/serverless/{id}/workers", seen: "2026-09-28" }
- id: worker.billing-by-state
applies_to: ["state:RUNNING", "state:IDLE", "state:INITIALIZING", "state:THROTTLED", "state:UNHEALTHY"]
statement: >
A worker is billed from when its handler starts loading the model until it
scales down: while it loads the model, while it processes requests, and
through the idle timeout after each request. Pulling the image and
downloading cached models happen before that and are not billed. A worker
kept up by `workers.min` never scales down, so it bills continuously. The
words differ between surfaces: REST v2 `IDLE` is a running worker polling
for jobs, which is billed, while the console's "Idle" is a scaled-down
worker, which is not.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/workers/overview", seen: "2026-10-06", note: Worker states table and billing paragraph. }
- { source: public-docs, url: "https://docs.runpod.io/serverless/pricing", seen: "2026-10-06", note: "Compute cost covers start time, execution time and the idle timeout." }
- { source: rest-v2-spec, ref: WorkerStatus, seen: "2026-09-28" }
- { source: skill, path: plugins/runpod/skills/runpodctl/SKILL.md, note: A worker kept up by a minimum of 1 bills continuously. }
- id: worker.crash-loop-looks-queued
applies_to: ["state:INITIALIZING", "state:RUNNING"]
statement: >
A worker that crash-loops on an image pull or a CUDA mismatch can show as
`INITIALIZING` or `RUNNING` while its job stays in `IN_QUEUE`. Read the
worker's logs with `source=system`. Repeated "start container" lines with
no container output mean the container exits before the handler runs.
status: documented
evidence:
- { source: skill, path: plugins/runpod/skills/runpodctl/reference/command-reference.md }
- { source: skill, path: plugins/runpod/skills/runpod-usage/reference/endpoint-workflows.md }
- { source: rest-v2-spec, ref: LogSource, seen: "2026-09-28" }
- id: worker.unhealthy-scale-down
applies_to: ["state:UNHEALTHY"]
statement: >
Runpod retries an unhealthy worker for up to 7 days. An endpoint that
keeps producing unhealthy workers is scaled down automatically.
status: documented
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/workers/overview", note: Worker states table and the note below it. }
- id: worker.cold-start
statement: >
A cold start happens when no warm worker is available: the container
starts and the model loads into GPU memory before the job runs. Time spent
waiting in the queue plus the cold start is reported as delay time,
separately from execution time.
status: documented
see: [flashboot.modes]
evidence:
- { source: public-docs, url: "https://docs.runpod.io/serverless/endpoints/job-states", note: Delay time and cold start time metrics. }
evals/concept-lookup-restart.eval.md
# Answer an exact-fact question from the concept files
## Prompt
I stopped my pod last night. The console shows my GPU type available in that data
center, but starting the pod again fails with no GPUs available. Why, and what should
I do?
## Expected behavior
Per `runpod-usage/SKILL.md` ("Exact facts: the concept files"):
1. The agent should find the matching concept files (`pod-deployment.yaml`, `machine.yaml`)
rather than answer from memory.
2. It should explain that a pod is bound to the machine it was placed on. Stopping releases
that machine's GPUs, so others can rent them, and data-center stock does not mean the pod's
own machine has a free GPU.
3. It should give the fix the rules support: wait and retry, start with zero GPUs to
reach the volume disk, or keep data on a network volume and deploy a new pod anywhere in that
data center.
## Assertions
- Explains that a stopped pod restarts only on its own machine, not anywhere in the data center.
- Distinguishes data-center stock from the pod's machine having a free GPU.
- Recommends a network volume, which any pod in the data center can attach, for data that must survive this.
- Does NOT tell the user to simply retry in another data center without moving their data.
evals/cpu-threads-vs-vcpus.eval.md
# CPU threads on a GPU pod
## Prompt
My video preprocessing on a GPU pod is way slower than on my laptop. `nproc` says 96 cores, so
I set my thread pool and data loader to 96, but it got even slower. What's going on?
## Expected behavior
The agent should:
1. Explain that the pod can see every core on the machine but can only use the vCPUs it rents,
so `nproc` and `os.cpu_count()` overstate what the pod gets
2. Explain that starting a thread per visible core gets the extra threads throttled, which slows
CPU-bound work down
3. Show how to read the real limit from the cgroup (`/sys/fs/cgroup/cpu.max`, or
`cpu.cfs_quota_us` / `cpu.cfs_period_us` under `/sys/fs/cgroup/cpu/`)
4. Recommend capping threads to that limit: `OMP_NUM_THREADS`, `MKL_NUM_THREADS`,
`torch.set_num_threads()` and data loader `num_workers`
## Assertions
- Says `nproc` / `os.cpu_count()` can report more cores than the pod's vCPUs
- Names CPU throttling from too many threads as the cause
- Gives a way to read the cgroup CPU limit
- Recommends setting thread counts to the vCPU limit, not the visible core count
- Does NOT recommend raising thread or worker counts further
evals/cuda-floor-on-create.eval.md
# Set a CUDA floor when creating a GPU pod
## Prompt
Create a Runpod GPU pod running the latest official PyTorch image on an RTX 4090 so
I can iterate on a training script over SSH.
## Expected behavior
Per `runpod-usage/reference/gpu-selection.md` ("Step 3: pin the CUDA floor") and the
`runpodctl` skill's decision rules:
1. **Pins the host CUDA floor to the image's CUDA line** — the current official
PyTorch image is a `cu128` build, so the create carries `--min-cuda-version 12.8`
(or `gpu.minCudaVersion: "12.8"` on REST v2). A create with no floor accepts any
host, and the container then dies at startup or silently drops to CPU.
2. **Does not jump to `13.0`** just because it's the newest — 13.0 is for a CUDA-13
image (`runpod/comfyui:cuda13.0`, the `cu1300` cluster PyTorch image), not for a
newer GPU.
3. **Does not reach for the old default** — `runpod-torch-v21` is torch 2.1 on CUDA
11.8; the current default is `runpod-torch-v280`.
4. **Sets the floor in whichever lane it uses** — `--min-cuda-version` on
`runpodctl pod create`, or `gpu.minCudaVersion` on a v2 `POST /v2/pods` or the MCP
`create-pod` tool.
5. **Cost guard + SSH** as usual — `--terminate-after`, and the SSH key registered
before creation.
## Assertions
- The pod create includes a CUDA floor (`--min-cuda-version` / `gpu.minCudaVersion`), not just a GPU id.
- The floor matches the image's CUDA line rather than defaulting to the highest number available.
- Does not use `runpod-torch-v21` / a CUDA 11.8 image as the "latest PyTorch" default.
- If MCP is the chosen lane, the `create-pod` call carries `gpu.minCudaVersion`.
evals/fast-disk-for-installs.eval.md
# Fast disk for installs on a pod
## Prompt
`pip install` and unpacking a dataset into `/workspace` on my pod are slow. Someone told me
regions with network volumes put everything on network storage, so I should move to a data
center without network volumes to get a local disk. Is that right?
## Expected behavior
The agent should:
1. Say the container disk (`/`, including `/root`, `/opt`, `/tmp`) is local, so installs
outside `/workspace` are fast wherever the pod runs
2. Say `/workspace` (the volume disk) is network-backed in data centers that offer network
volumes, and that a data center without network volumes does not guarantee a local
`/workspace`
3. Suggest `findmnt /workspace` to check (`fuse` = network-backed, `xfs` on a device = local)
4. Recommend installing packages, building and unpacking scratch data on the container disk,
sized with `--container-disk-in-gb`, and keeping only what must survive a restart in
`/workspace`
5. Note that the container disk is wiped when the pod stops or restarts, so anything that must
survive goes in `/workspace` or a network volume
## Assertions
- States the container disk is local and faster than a network-backed `/workspace`
- Does NOT treat "no network volumes" as a guarantee of a local `/workspace`
- Does NOT recommend switching data centers just for disk speed
- Mentions checking with `findmnt /workspace`
- Recommends the container disk for installs and scratch, and mentions sizing it
- Notes the container disk is wiped on stop or restart
evals/image-guidance.eval.md
# Image construction, startup and model cache integration
These are manual review scenarios, not an automated test suite.
## Prompt
"I have a Runpod PyTorch image and want a Pod with SSH plus my HTTP app. I currently
replace CMD with a Python script that prints CUDA information and exits. Can you fix
the startup and speed up rebuilds? I use Windows and Docker may already be in WSL."
## Expected behavior
- Preserve the selected base's `/start.sh` and explain its SSH/key/service startup.
- Give a concrete `/post_start.sh` hook or a long-running app startup pattern, and
distinguish a successful one-shot check from a usable running pod.
- Check actual SSH and HTTP readiness rather than treating a fixed sleep as proof.
- Keep the dependency install before changing code, retain a published base tag and
digest, and distinguish build cache reuse from deployment-host layer residency.
- Inspect existing native/WSL/remote Docker and its daemon before installation.
If selecting a missing WSL2 setup, retain the `wsl --install` route.
- For a new Windows installation, install and launch Docker Desktop before enabling
distribution integration and checking `docker info` in that distribution.
- For local Linux Docker Engine permission errors, explain the optional `docker`
group command, its root-level privileges, and logging out and back in before
verifying access. Do not apply host installation steps to a Runpod pod.
## Additional prompts
1. "My queue worker uses Transformers. Does attaching the cached model make it load
automatically, and is HF_HOME the same as my downloaded model folder?"
2. "For my load-balanced API, should I test handler.py through the queue job API?"
3. "Can I copy all of PyTorch to my network volume each startup to improve startup?"
## Assertions
- A custom handler configures its cache or resolves a matching snapshot. Explain
`HF_HOME` versus `HF_HUB_CACHE` and a plain local model directory. Cite the public
cache integration docs; do not assert an unverified relationship between `MODEL_NAME`
and the endpoint's Model field.
- Queue testing uses a local handler input; load-balanced testing hits HTTP routes and
follows GP14. Do not add SSH, exposed ports or a pod proxy to a queue endpoint.
- Preserve immutable runtime layers outside mounts and persist mutable application
data deliberately. Explain local-image versus network-volume I/O and the training
storage tradeoff, linking GP25/GP21 without reclassifying their live-verified results.
evals/pod-service-loop.eval.md
# Stand up a service on a pod — the loop's load-bearing details
## Prompt
Stand up an Ollama server on a Runpod pod over SSH and give me the URL. Walk
through the commands.
## Expected behavior
Per `runpod-usage/reference/{pod-workflows.md,on-pod-setup.md}`:
1. **Ports + env at creation** — expose `11434/http` and set `OLLAMA_HOST=0.0.0.0`
at `pod create` time (they can't be added to a running pod without a reset).
2. **Env isn't in the SSH shell** — when starting the service over SSH, pass the
env explicitly (`env OLLAMA_HOST=0.0.0.0 … ollama serve`), because creation
`--env` vars only reach PID 1.
3. **Detach the server** — start it with `setsid … < /dev/null &` so it survives
the SSH channel closing, and return immediately.
4. **Verify from outside** — poll `https://<pod-id>-11434.proxy.runpod.net/api/tags`
until 200 (expect warm-up 502s) before reporting the URL.
5. **Cost guard** — remove the pod when done (`runpodctl pod remove`). Never `--terminate-after` / `--stop-after`: they were never enforced and v2.12.0 removed them.
## Assertions
- Sets the port and env at `pod create` (not after).
- Passes env explicitly on the SSH-launched service command (doesn't rely on `--env` reaching the shell).
- Starts the server detached (`setsid`/`nohup` + `</dev/null`), not a bare `&`.
- Verifies by polling the proxy URL, and tells the user to remove the pod when done. Does NOT use `--terminate-after` / `--stop-after`.
evals/verify-with-real-request.eval.md
# Verify readiness with a real request — "up" ≠ "ready"
## Prompt
I started a web server on my Runpod pod and `runpodctl pod get` says it's RUNNING,
but requests to the proxy URL are failing. Is it broken? How should I confirm it's
actually ready?
## Expected behavior
Per `runpod-usage/reference/development-loop.md` (step 6) + `pod-workflows.md`:
1. The agent should explain that **"RUNNING" only means the container exists**, not
that the service serves — and that the proxy commonly returns **502 for ~30–60s**
during warm-up, so failing requests right after boot are expected, not broken.
2. It should confirm readiness by **polling a real request** to the proxy URL
(e.g. `until curl -sf https://<pod-id>-<port>.proxy.runpod.net/<health>; do sleep 5; done`)
with a timeout, not by trusting the status field.
3. It should also check the likely real causes if it stays down: the service isn't
bound to `0.0.0.0`, or it died because it wasn't started detached (`setsid`).
## Assertions
- States that RUNNING/ready is not the same as serving; expects a warm-up 502 window.
- Recommends polling a real request against the proxy URL to confirm readiness.
- Mentions the `0.0.0.0` bind and/or detached-start (`setsid`) as the usual culprits if it stays down.
- Does NOT conclude "it's broken" from the RUNNING status alone.
reference/building-images.md
---
concepts: [container-image, network-volume, container-disk, model-store]
---
# Building container images for Runpod
How to think about building an image for Runpod: pick the right base, layer for speed,
decide what to **bake in** vs **mount at runtime**, and match the **image contract** to your
target (pod vs serverless queue vs serverless load-balanced). CLI mechanics (login, tag,
push) live in [runpod-companion-clis docker](../../runpod-companion-clis/reference/docker.md) and
[docker.md](docker.md); this is the strategy layer.
## Start from an official Runpod base image
**For a PyTorch GPU workload, prefer a compatible official `runpod/pytorch:<tag>` image.**
Reuse its framework/CUDA stack rather than installing a second copy. Check the application's
dependency requirements and run a representative operation on the intended GPU before
claiming compatibility. The [custom Pod guide](https://docs.runpod.io/pods/templates/create-custom-template)
shows how to extend these bases.
Existing host-cached layers can avoid transfer, but do not promise that every selected
host already has the base. Verify layer reuse and cold-start time separately.
**Exceptions (both shown in the golden paths):** a trivial CPU-only workload may use a slim
base (e.g. `python:3.11-slim`) — see [GP23](../../runpod/golden-paths/23-minimal-queue-image/README.md);
and if you build from a **non-Runpod base you must reproduce SSH yourself** when the pod
promises SSH access (see the section below and [GP22](../../runpod/golden-paths/22-minimal-pod-image/README.md)).
For a different runtime, a pinned Ubuntu base with explicit dependencies can be suitable;
do not add PyTorch to an application that does not use it.
Then, whatever the base:
- **Pin an exact published tag**, e.g. `runpod/pytorch:1.0.2-cu1281-torch280-ubuntu2404`,
and record its resolved digest, source revision, dependency locks and model revision.
A versioned base alone does not pin later downloads. See
[image versioning](https://docs.runpod.io/serverless/workers/deploy#image-versioning).
- **Build for x86_64:** `docker build --platform=linux/amd64 …` — Runpod hosts are x86_64
(see [docker.md](docker.md)).
## Layer for fast, cacheable pulls
Order layers **least- to most-frequently-changing** so a code edit doesn't invalidate the
heavy dependency layers, and independent layers pull in parallel:
1. base image (`FROM runpod/pytorch:…`)
2. system deps (`apt-get …`)
3. Python deps (`pip install …`)
4. **your code last**
Unchanged build steps can reuse the builder's cache. A deployment host separately reuses
image layers it already has. Extending the **same published dependency image** preserves
its heavy layers across your templates; independently reinstalling the same PyTorch
version does not establish identical layers. Compare layer digests after a code-only edit.
## Dockerfile best practices
- **`.dockerignore`** — exclude `.git`, virtualenvs, datasets, local caches so the build context stays small and pushes fast.
- **Cache the dependency layer** — `COPY requirements.txt` and `pip install` *before* you `COPY` your code, so a code edit doesn't reinstall everything:
```dockerfile
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
```
- **Shrink the image** (smaller = faster pull + cold start) with concrete steps: `apt-get install --no-install-recommends …` then `rm -rf /var/lib/apt/lists/*`; `pip install --no-cache-dir`; use a **multi-stage** build when heavy build tools aren't needed at runtime.
- **BuildKit cache mounts** for fast rebuilds: `RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt`.
- **Never bake secrets** into layers (API keys, tokens) — layers are extractable; pass secrets as runtime env.
- **`ENV PYTHONUNBUFFERED=1`** — so logs stream unbuffered. (Pin the base image tag too — see above.)
Keep changing build arguments near their first use so a source-version change does not
invalidate an earlier dependency install. See Docker's
[cache examples](https://docs.docker.com/build/cache/optimize/).
## Don't clobber the base image's startup (SSH / web terminal) — **pods**
Runpod PyTorch bases provide `/start.sh` for their interactive services: it reads
`$PUBLIC_KEY` into `~/.ssh/authorized_keys`, runs `ssh-keygen -A`,
starts `sshd`, and brings up the web terminal / Jupyter. It also runs `/pre_start.sh` before
and `/post_start.sh` after, if those exist.
**For a pod that promises these services:** inherit `/start.sh`, or invoke it from your
custom startup. Inspect the selected tag's command and hooks before overriding them.
Application-only pods may intentionally omit SSH/Jupyter; queue workers use their handler.
Why: if a custom `CMD`/`ENTRYPOINT` doesn't chain `/start.sh`, none of that startup runs — you
get **no SSH, no web terminal**, and can be locked out of the pod. For a pod this is the #1
footgun.
Three safe patterns, in order of preference:
1. **Don't override `CMD` at all** (default — use this unless you need your own foreground
process). Add your layers, leave `CMD ["/start.sh"]`. Do per-pod work via the env-driven
hooks the base already runs:
```dockerfile
FROM runpod/pytorch:<tag>
COPY post_start.sh /post_start.sh # base runs this AFTER sshd is up
RUN chmod +x /post_start.sh
# no CMD — inherit the base's /start.sh
```
2. **Override only if you need your own foreground process** (a long-running service as PID 1):
call the base start first, then `exec` your workload:
```dockerfile
COPY run.sh /run.sh
RUN chmod +x /run.sh
CMD ["/run.sh"]
```
```bash
#!/usr/bin/env bash
/start.sh & # SSH + web terminal (base startup), backgrounded
exec python -u my_service.py # your long-running workload in the foreground
```
This starts both independently; it does not prove base services are ready. If the app
needs completed base initialization, use the supported `/post_start.sh` hook instead.
Verify SSH and the app separately, and use explicit supervision if both must fail
together. A fixed sleep is not a readiness check.
3. **From a non-Runpod base, reproduce SSH yourself** (only if you can't start from
`runpod/pytorch`). Minimum to not get locked out of a pod:
```dockerfile
RUN apt-get update && apt-get install -y --no-install-recommends openssh-server python3 \
&& rm -rf /var/lib/apt/lists/*
COPY start.sh /start.sh
RUN chmod +x /start.sh
CMD ["/start.sh"]
```
```bash
#!/usr/bin/env bash
mkdir -p ~/.ssh && chmod 700 ~/.ssh
[ -n "$PUBLIC_KEY" ] && echo "$PUBLIC_KEY" >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys
ssh-keygen -A # host keys
service ssh start # or: /usr/sbin/sshd -D
exec python3 -u -m http.server "${PORT:-8000}" # concrete long-running workload
```
Reference implementation: `justinwlin/Runpod-GPU-And-Serverless-Base` (a dual pod+serverless
base) and the vendored `start.sh` in
[golden path 09](../../runpod/golden-paths/09-custom-serverless-dev-loop/README.md). Worked
end-to-end in [golden path 22 — minimal pod image](../../runpod/golden-paths/22-minimal-pod-image/README.md).
## Bake in vs mount at runtime (this drives startup speed)
The **image** and its baked files live on the host's **local disk** — fast.
When a **network volume** is attached it takes over the working directory (`/workspace` on a
pod, `/runpod-volume` on serverless); anything written there lives on **networked storage**,
which is slower — **especially for many small files**.
Keep runtimes outside volume mount points, for example `/opt/venv`: a mount can hide
baked files. Avoid copying the entire Python/PyTorch environment into `/workspace` at
every startup. Persist the mutable outputs and files the application actually needs.
Live proof that this is a real filesystem boundary (baked = `overlay`/local, volume =
`fuse`/MooseFS network mount): [golden path 25 — bake vs mount](../../runpod/golden-paths/25-bake-vs-mount/README.md).
- **Bake into the image:** packages, libraries, and lots of small static files → local, fast.
- **Mount a volume:** large/few files (model weights, datasets), anything that must persist
across pods, or data you stream/live-load.
- **High-throughput / I/O-bound training:**
- **temporary / one-off run** → a **pod using local (non-network) storage** is fastest;
- **persistent, many small files, or I/O-bound** → a **high-performance network volume**.
See [golden path 21 — storage tiers](../../runpod/golden-paths/21-storage-tiers.md).
## Match the image contract to the target
| Target | Needs a handler? | Entry point |
| --- | --- | --- |
| **Pod** | No | your `CMD`/entrypoint — a long-running service; bind `0.0.0.0`, expose ports |
| **Serverless — queue-based** | **Yes** | `runpod.serverless.start({"handler": handler})` |
| **Serverless — load-balanced** | No (different contract) | your own **HTTP server** exposing routes (no queue handler) |
Minimal runnable image per contract (each built + deployed live): pod →
[golden path 22](../../runpod/golden-paths/22-minimal-pod-image/README.md), queue →
[golden path 23](../../runpod/golden-paths/23-minimal-queue-image/README.md), load-balanced →
[golden path 14](../../runpod/golden-paths/14-load-balancing-endpoint.md).
Queue vs load-balanced request/response shapes and when to pick each are covered in
[endpoint-workflows.md](endpoint-workflows.md) and golden paths
[12 (streaming)](../../runpod/golden-paths/12-serverless-streaming.md),
[14 (load-balancing)](../../runpod/golden-paths/14-load-balancing-endpoint.md), and
[17 (WebSocket)](../../runpod/golden-paths/17-serverless-websocket.md). One image can be
**dual-mode** (pod dev + serverless) — see
[golden path 09](../../runpod/golden-paths/09-custom-serverless-dev-loop/README.md).
## Test locally before deploying
- Build for the right platform: `docker build --platform=linux/amd64 …`.
- **Queue handler:** run the container and invoke `handler.py` locally (the dual-mode
`python handler.py` loop in [golden path 09](../../runpod/golden-paths/09-custom-serverless-dev-loop/README.md))
before pushing.
- **Load-balanced:** run the container and hit the HTTP routes locally.
- Then push ([runpod-companion-clis docker](../../runpod-companion-clis/reference/docker.md)) and deploy
(runpodctl or runpod-mcp).
Run the actual container entrypoint as well as unit checks. For an interactive pod, a
successful one-shot verification script is not the long-running process that keeps the
pod usable. Preserve base startup and run such checks through a hook or a separate command.
Report GPU execution and cloud behavior as unverified when only local checks ran.
## Exact rules
See [container images](../concepts/container-image.yaml),
[container disk](../concepts/container-disk.yaml),
[network volumes](../concepts/network-volume.yaml) and
[model delivery](../concepts/model-store.yaml).
reference/concepts.md
# Pods vs Serverless
Two ways to run compute on Runpod. Pick based on the shape of the work.
## Pods — interactive / long-lived
A Pod is a GPU or CPU container you rent by the minute and keep running. You get
full control: SSH, a web terminal, JupyterLab, VS Code/Cursor, exposed ports. It
stays up until you stop or delete it, and you pay for every minute it exists
(running or stopped-with-disk).
Use a Pod when:
- You are developing, experimenting, or debugging interactively.
- Work is long-running or stateful (training, fine-tuning, rendering, a notebook).
- You want a persistent environment you shell into and iterate in.
- You need a service up continuously with a stable address.
Not a good fit when traffic is bursty or idle much of the day — you pay for idle time.
Two clouds — **default to Secure Cloud** (data-center-grade T3/T4 hosts; production and
sensitive data) and use **Community Cloud** (approved vendors, cheaper, no new hosts being
onboarded) only when the user explicitly asks. The comparison and the reason for the
default: [`gpu-selection.md`](gpu-selection.md#step-5-secure-vs-community-cloud).
Limits: no Docker Compose (Runpod runs Docker for you), no UDP (TCP/HTTP only), no Windows.
## Serverless — request/response + autoscale
Serverless runs your container only while it is processing requests. You deploy a
worker image behind an **endpoint** (a URL). Workers spin up on demand, process
jobs, and spin down when idle. You pay for compute time used, with no idle cost
when nothing is running.
Use Serverless when:
- Work is request-shaped: inference, image generation, transcription, batch jobs.
- Traffic is bursty or unpredictable and you want it to scale to zero.
- You want a managed URL, not a machine to babysit.
## Serverless building blocks
- **Endpoint** — the access point (URL) clients send requests to. Holds the scaling
and GPU config.
- **Worker** — a container instance running your image + code. Runpod starts and
stops workers automatically based on load.
- **Handler function** (queue-based) — `def handler(event)` reads `event["input"]`,
processes it, returns a result. Started with `runpod.serverless.start({"handler": handler})`.
- **Job** — one unit of work: the input payload, queued until a worker is free.
## Cold starts and FlashBoot
A **cold start** is the gap between a request arriving at an endpoint with no ready
worker and that worker being warmed up — container start + model load into VRAM +
runtime init. Bigger models = longer cold starts.
Reduce cold starts by:
- **FlashBoot** (on by default) — retains worker state after spin-down so a worker
"revives" faster than a fresh boot. Most effective with steady traffic where
workers cycle between active and idle.
- **Cached models** — schedule workers onto machines with your model files
pre-loaded, cutting model-load time.
- **Active workers** ≥ 1 — keep workers always warm (see below).
## Active vs flex workers, scale-to-zero
- **Active (min) workers** — always-on, kept warm at all times. Setting this to 1+
eliminates cold starts for those slots but bills continuously, even when idle.
Default is **0**.
- **Flex workers** — the elastic pool between active count and **max workers**.
Spun up under load, spun down when idle. **Scale-to-zero** = active workers 0, so
the endpoint drops to zero running workers (and zero cost) when idle, at the price
of a cold start on the next request.
- **Idle timeout** — how long a flex worker stays warm after finishing before it
shuts down (default 5s). Longer = fewer cold starts, more cost.
- **Max workers** — concurrency cap and cost safety limit (default 3). Set ~20%
above expected peak concurrency to absorb spikes.
> **These are the platform defaults** (Console / `runpodctl` / API). The **flash SDK**
> applies its *own* defaults for the same settings — `idle_timeout` **60s** (not 5s),
> `workers` **(0, 1)** i.e. max 1 (not 3), `execution_timeout` unlimited (not 600s). So a
> default value depends on which layer you configured through; don't assume the flash
> number holds for a platform-created endpoint or vice-versa. See
> [`../../runpod-flash/reference/api.md`](../../runpod-flash/reference/api.md).
Auto-scaling type decides *when* to add workers:
- **Queue delay** — add workers when requests wait longer than a threshold
(default 4s). Good when small delays are acceptable.
- **Request count** — scale on pending + in-progress work
(`ceil((inQueue + inProgress) / scalerValue)`). More aggressive; good for LLMs
and frequent short requests.
## Queue-based vs load-balanced endpoints
Two endpoint types, chosen at creation:
**Queue-based** (traditional)
- Requests go into a queue and are processed in order; execution is guaranteed with
automatic retries.
- Uses a handler function; fixed operations: `/run`, `/runsync`, `/status`, `/stream`,
`/cancel`, `/health`, etc.
- Best for async tasks, batch, long-running jobs. Higher latency (queue + worker).
**Load-balanced**
- Requests route directly to a worker's HTTP server — no queue, no backlog buffering
(overloaded workers drop requests).
- You run any HTTP server (FastAPI, Flask) and define your own URL paths; workers
expose a `/ping` health check.
- Lower latency (single hop). Best for real-time inference, streaming, custom REST APIs.
- No built-in retry.
Analogy from the docs: queue-based is like TCP (guaranteed delivery), load-balanced
is like UDP (fast, no guarantees).
## Templates
A **template** is a saved, pre-configured setup: a Docker image plus its default
config (exposed ports, environment variables, container/volume disk, start command).
Runpod ships official templates (e.g. PyTorch) so you can launch a working
environment without wiring dependencies yourself, and you can save your own custom
templates for repeatable deployments of both Pods and endpoints.
## Where to act
This file is a mental model. To actually create or manage resources use
**runpodctl** / **runpod-mcp** (infra), **runpod-flash** (deploy your own code), or the
Runpod console.
reference/development-loop.md
---
concepts: [pod, serverless-endpoint, container-image, job]
---
# The Runpod development loop (golden loop)
Every Runpod task an agent runs follows the same spine — proven across the golden
paths (Ollama pod, ComfyUI pod, Whisper endpoint). Learn this loop; it has two
specializations depending on the workload shape.
```
decide shape → prefer prebuilt → plan resources → provision → (set up if scratch)
→ run/deploy → VERIFY with a real request → deliver → cost-guard + teardown
```
## 1. Decide the workload shape
- **A server you open / interactive / long-lived** (Ollama, ComfyUI, Jupyter,
training) → a **pod**. Reached at a proxy URL. Detailed loop: `pod-workflows.md`.
- **A request/response API that should scale to zero** (transcription, inference
endpoint) → a **serverless endpoint**. Invoked via `/run`/`/runsync`. Detailed
loop: `endpoint-workflows.md`.
See `concepts.md` if unsure.
## 2. Prefer a prebuilt / known option before building from scratch
This is the biggest lever for speed and reliability:
- Pod service → look for an **official Runpod template / prebuilt image**
(`runpodctl template search <app>`) — it auto-starts and skips the install
gotchas. Official pod templates are indexed in
[`runpod-templates`](../../runpod-templates/SKILL.md).
- Serverless → look for a **Hub worker** (`runpodctl hub search <app>`).
- Build **from scratch** (install on a pod, or `flash`, or a custom image) only
when no good prebuilt exists, or you need something **custom or lighter** than
what's shipped.
- **Exception:** when the user explicitly requests a custom template project or
Dockerfile, that is the deliverable (route to `runpod-build-template`); an existing
prebuilt does not replace it.
## 3. Plan resources
GPU/VRAM and the **CUDA floor** (`gpu-selection.md` — default
`--min-cuda-version 12.8` on any GPU create; a create with no floor accepts any host
CUDA version and a modern image can land on one too old to run it), storage
(**default a network volume** — `storage.md`), and the execution lane (`../../runpod/SKILL.md` router:
runpod-mcp / runpodctl / flash).
- **Default to Secure Cloud.** Only deploy Community Cloud when the user explicitly asks
(`runpodctl pod create` already defaults to `--cloud-type SECURE`; don't override it).
Why: [`gpu-selection.md`](gpu-selection.md#step-5-secure-vs-community-cloud).
- **Cache vs network volume is a latency call, not a cost call.** Download time is never
billed, but on a cache miss Runpod holds the worker start until the model lands: the
job sits in the queue and the endpoint looks idle. Latency-sensitive → pre-load a
network volume; otherwise the cache is the cheaper default. Cache capacity is per
machine, so time both on the user's own endpoint rather than quoting a threshold. See
[`runpodctl/reference/model-caching.md`](../../runpodctl/reference/model-caching.md).
## 4. Provision & 5. Set up
Provision through the chosen lane. If from-scratch, do the setup step of the
matching sub-loop (pods: SSH-exec install; serverless: write handler / build image).
Prebuilt options usually skip setup entirely.
## 6. Verify with a real request — "up" ≠ "ready"
The load-bearing step. A pod showing **Running**, or a serverless worker showing
**ready**, does **not** mean it serves. Always confirm from **outside** with a real
call, and expect a warm-up window:
- **Pod:** poll the proxy URL until it answers — expect ~30–60s of **502s** during
boot.
- **Serverless:** send a real input; the **first call cold-starts** (may exceed
`runsync`'s 60s → use `/run` + poll `/status/<id>`). A worker that is `ready` but
leaves jobs `IN_QUEUE` with `inProgress: 0` is a **broken image** — switch, don't
wait.
Only report success once a real request returns the right result.
## 7. Deliver
Return the access URL (pod) or endpoint id + a **working sample call** (serverless),
and note the security posture (proxy URLs and endpoints are public unless you add
auth).
## 8. Escalate on manual steps
If something needs a human — OAuth, a quota/capacity increase, a gated-model
license, a missing credential, a payment issue — **stop and say exactly what's
blocked**. Don't spin or fake progress.
## 9. Cost-guard + teardown
- Pod → `runpodctl pod remove <pod-id>` when done. Not `--terminate-after` /
`--stop-after`: they were never enforced and v2.12.0 removed them.
- Serverless → `--workers-min 0` (scale-to-zero, ~$0 idle).
- Delete test resources when done (`runpodctl pod remove` / `serverless delete` /
`rp flash app delete`; then any network volume).
## Which sub-loop?
| Workload | Sub-loop |
| --- | --- |
| A service you open at a URL (Ollama, ComfyUI, dev box, training) | `pod-workflows.md` |
| A request/response API that scales to zero (Whisper, inference) | `endpoint-workflows.md` |
reference/docker.md
# Building a Docker image Runpod can run
Runpod pulls a container image and runs it on x86_64 Linux GPU/CPU hosts. For
serverless, the image runs a **handler**; for pods, the image runs whatever your
`CMD`/start script does. This file covers the serverless handler contract and the
build/push mechanics that apply to both.
## The serverless handler contract
A queue-based serverless worker is a Python script that starts the Runpod SDK
with a handler function:
```python
# handler.py
import runpod
def handler(job):
job_input = job["input"] # your request payload lives under "input"
prompt = job_input.get("prompt")
# ... do the work ...
return {"result": prompt} # returned value becomes the job output
runpod.serverless.start({"handler": handler}) # required — blocks and serves jobs
```
Request/result shape:
- The platform hands your handler a job dict: `{"id": "<uuid>", "input": { ... }}`.
`id` is Runpod's job id; `input` is exactly what the client sent.
- Whatever the handler **returns** is the job result (must be JSON-serializable).
- Raising an exception marks the job `FAILED` and returns the error details.
Handler variants: return a value (standard), `yield` values (streaming — add
`return_aggregate_stream: True` to expose them via `/run`), or `async def` +
`yield` (async). Concurrent handlers serve multiple requests per worker.
**Load-balanced endpoints do not use a handler.** You expose your own HTTP server
(FastAPI, Flask, vLLM, etc.) and Runpod routes to it. The `runpod.serverless.start`
contract is only for queue-based endpoints.
Best practice: load models and other heavy state **at module level, outside the
handler**, so it initializes once per worker instead of once per request.
## A minimal Dockerfile
```dockerfile
FROM python:3.11.1-slim
WORKDIR /
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY handler.py .
CMD ["python", "-u", "/handler.py"]
```
`requirements.txt` must include the `runpod` SDK plus your libraries:
```
runpod~=1.7.6
torch==2.0.1
transformers==4.30.2
```
> **Version pin gotcha (numpy 2.x):** if you `pip install torch==2.2.x`, also pin
> `numpy<2`. That torch wheel is built against NumPy 1.x; pip otherwise pulls NumPy
> 2.x and the image **builds and starts fine but crashes at inference** with
> `RuntimeError: Numpy is not available`. Testing the container locally (below) catches
> it before deploy (`gotchas.md`).
> **`cryptography` uninstall gotcha (on a `runpod/pytorch` base):** installing the
> `runpod` SDK on an official `runpod/pytorch` image (e.g. behind `runpod-torch-v280`)
> fails with `Cannot uninstall cryptography 41.0.7 … no RECORD file` — the base's copy is
> **Debian-managed**, so pip can't replace it with the newer one `runpod` wants. Fix:
> `pip install --ignore-installed cryptography runpod` (add `--break-system-packages` for
> the base's PEP-668 "externally managed" pip). Install conflict-free deps like
> `faster-whisper` first, on their own. Verified live 2026-07-10 (golden path 09).
For GPU/CUDA workloads, start from a CUDA base and install Python yourself, or
build on a framework image:
```dockerfile
FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04
RUN apt-get update && apt-get install -y python3.11 python3-pip
# or: FROM pytorch/pytorch:2.0.1-cuda11.7-cudnn8-runtime
```
Keep images small (slim bases, multi-stage builds, `.dockerignore`, clean up apt
caches) — smaller images pull and cold-start faster.
## Build for x86_64 — always `--platform=linux/amd64`
Runpod hosts are x86_64. If you build on an ARM machine (Apple Silicon Mac) the
default build produces an arm64 image that Runpod cannot run. **Always pass the
platform flag:**
```bash
docker build --platform=linux/amd64 -t DOCKER_USER/worker-name:v1.0.0 .
```
This is the single most common deployment failure. It applies to serverless
workers, pod images, and template images alike.
## Pin tags — never rely on `latest`
`latest` is mutable: it points to whatever was last pushed without an explicit
tag, and Runpod caches images per host, so a worker can silently keep running a
stale `latest`. Use explicit semantic versions (`v1.0.0`, `v1.0.1`), and for
critical deploys pin the immutable digest (`name:tag@sha256:...`).
```bash
docker build --platform=linux/amd64 -t DOCKER_USER/worker-name:v1.0.0 . # good
docker build -t DOCKER_USER/worker-name:latest . # avoid
```
## Test the container locally before pushing
Provide a `test_input.json` next to the handler and run the image — the SDK
detects it, runs one job, prints the output, and exits:
```json
{ "input": { "prompt": "Hey there!" } }
```
```bash
python handler.py # bare, on host
python handler.py --test_input '{"input":{"prompt":"hi"}}' # inline input
docker run -it DOCKER_USER/worker-name:v1.0.0 # inside the image
docker run --rm --gpus all DOCKER_USER/worker-name:v1.0.0 # with GPU
```
Fix any import/dependency/handler errors here — it is far faster than debugging
after deploy.
## Push, and handle private images
```bash
docker login
docker push DOCKER_USER/worker-name:v1.0.0
```
Runpod pulls public images with no extra config. For **private** registry images
you must register container-registry credentials with Runpod once (Console →
Settings → Container Registry, then select the credential on the template/endpoint).
Runpod supports `docker login`-style credentials. Without this, workers fail to
pull the image.
## Deploy the pushed image as a serverless endpoint
`runpodctl serverless create` takes **`--template-id` or `--hub-id`, not `--image`** —
so deploying your own image is a **two-step**: register a serverless *template* pointing
at the tag, then create the endpoint from that template. Use `--compute-type CPU` for a
CPU-only image to sidestep GPU scarcity, and `--workers-min 0` for scale-to-zero:
```bash
runpodctl template create --name my-tpl --serverless \
--image DOCKER_USER/worker-name:v1.0.0 --container-disk-in-gb 10 # → template id
runpodctl serverless create --template-id <template-id> --name my-ep \
--compute-type CPU --workers-min 0 --workers-max 1 # → endpoint id
```
For a **private** image, attach the registry credential to the template (Console →
Container Registry, or `runpodctl registry create`) so workers can pull it. Full worked
example: golden path 05 (`golden-paths/05-model-to-endpoint-pipeline.md`).
## Payload limits
Handler input/output flows through the job API: `/run` caps at ~10 MB, `/runsync`
at ~20 MB. For anything larger, pass URLs or use a network volume / S3 and return
references instead of inlining bytes.
reference/endpoint-builds-and-releases.md
---
concepts: [endpoint-build, endpoint-release, serverless-endpoint, worker, container-image]
---
# Endpoint builds and releases
Every change to a Serverless endpoint's container becomes a **release**, and workers move
to it gradually. When the endpoint is connected to a GitHub repository, each commit can
also produce a **build**, and a completed build becomes a release.
## Releases and rolling releases
- **What creates a release:** a change to `image`, `env`, `args`, `disk`, `ports` or
`registry`. These settings live in the endpoint's own bound template, created and deleted
with the endpoint. Editing a template you created the endpoint *from* does not affect it.
- **How workers move:** idle workers on the old release are replaced right away. Busy
workers finish their current job first, so the endpoint keeps serving while it rolls.
- **Tracking a rollout:** the release list's rollout summary describes the endpoint's current
state, and each worker's `isStale` shows whether it still runs an older release. REST:
`GET /v2/serverless/{id}/releases`. MCP: `list-endpoint-releases` and `list-endpoint-workers`.
- **Where a release came from:** `source` is `MANUAL` for a configuration change, or
`GIT_BUILD` for a release produced by a GitHub build, which also sets `buildId`.
## Old and new versions overlap
During a rollout some workers run the old release and some the new one. When that is not
acceptable:
1. Set `workers.max` to `0` and wait for the workers to stop.
2. Update the endpoint.
3. Restore `workers.max`.
Use versioned image tags rather than `:latest`. Existing workers can keep serving a cached
`:latest` image, so pushing a new `:latest` does not guarantee they run it.
## GitHub builds
- A build runs from a commit to the connected repository and moves through `PENDING`,
`BUILDING`, `UPLOADING` and `TESTING`.
- It ends in `COMPLETED`, `FAILED`, `CANCELLED` or `TEST_FAILED`. For `FAILED` and
`TEST_FAILED`, the build's `error` field explains why.
- A completed build produces a release, and workers move to it the same way as any release.
- List builds newest first with `GET /v2/serverless/{id}/builds` (MCP: `list-endpoint-builds`),
and fetch an older one by id with `GET /v2/serverless/{id}/builds/{buildId}`
(MCP: `get-endpoint-build`).
## Exact rules
These facts come from the concept files, which cite the REST v2 spec and the public docs:
[`endpoint-release`](../concepts/endpoint-release.yaml) and
[`endpoint-build`](../concepts/endpoint-build.yaml).
reference/endpoint-workflows.md
# The serverless endpoint loop
The serverless specialization of the development loop (`development-loop.md`) — for
a **queue-based request/response API that scales to zero** (transcription, inference).
Unlike a pod there's no SSH, no exposed ports, no proxy: you deploy a worker and invoke it
over the Runpod job API. For direct HTTP applications, use the separate
[load-balancing contract in GP14](../../runpod/golden-paths/14-load-balancing-endpoint.md).
Proven on the Whisper golden path (both variants).
## 1. Pick the source (in order of preference)
1. **Hub worker (fastest, least fragile)** — a maintained prebuilt worker.
Select the execution lane using the
[router capability matrix](../../runpod/SKILL.md#runpod-mcp-vs-runpodctl-the-overlap).
Best when a good worker ships what you need.
2. **flash (from scratch, custom/light)** — write an `@app.queue` function with the
Runpod SDK and `rp flash deploy`. Best when you need your own model size / I/O schema / a lighter
image, or no good Hub worker exists.
3. **Custom image + endpoint** — write a handler, `docker build
--platform=linux/amd64`, push (private image → registry auth), create the
endpoint (runpodctl/MCP). Choose this when explicitly requested or when neither
of the above fits the application.
## 2. Deploy (scale-to-zero)
```bash
# Hub
runpodctl serverless create --hub-id <id> --name <name> --workers-min 0 --workers-max 3
# flash
rp flash deploy # @app.queue(workers=(0, 3)) in the code
# Custom image — TWO steps: serverless create takes --template-id/--hub-id, NOT an image.
runpodctl template create --name <tpl> --serverless \
--image <you>/<img>:<tag> --container-disk-in-gb 15 \
--env '{"KEY":"VALUE"}' # --env is a JSON object here; → template id
runpodctl serverless create --template-id <template-id> --name <name> \
--gpu-id "NVIDIA GeForce RTX 4090" --workers-min 0 --workers-max 2 # --gpu-id, not --gpu-type
```
`--workers-min 0` = no GPU billing while idle (pay only per request-second). Don't pin
`--data-center-ids` for a custom-image endpoint **unless a network volume forces it** — a
single-DC pin on a scarce GPU leaves workers `throttled` and jobs stuck `IN_QUEUE`
(observed live, 2026-07-10: EU-RO-1 4090 pin → `throttled:1`; unpinned → scheduled at once).
### Picking a Hub worker (this decides success)
Prefer an **actively-maintained** worker on a **broad, high-availability GPU pool**
— don't pin a scarce large tier a small model doesn't need. If deployed workers go
`ready` but jobs sit `IN_QUEUE` with `inProgress: 0`, the image is broken /
mis-dispatching — **switch workers, don't wait it out.** (Confirm it before switching:
`runpodctl serverless health <id>` for the counts (v2.9.0+) and `runpodctl serverless logs
<id> --source system` for the cause (v2.10.0+) — repeated `start container` with no
`container` output is a crash loop, not a capacity problem. MCP `stream-worker-logs` reads
the same logs.)
## 3. Invoke
The raw protocol is below because it is what you hand a user for copy-paste, and what a
non-Runpod client speaks. If you are driving it yourself, the tool lanes wrap it:
`runpodctl serverless run <id> --input '{...}'` (v2.9.0+) does submit-and-poll with auth,
local payload validation and bounded waiting; the MCP lane has typed job tools.
```bash
# warm / small payloads (sync, 60s window):
curl -s https://api.runpod.ai/v2/<endpoint-id>/runsync \
-H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" \
-d '{"input": { ... }}'
# first / cold call — async, then poll:
curl -s https://api.runpod.ai/v2/<endpoint-id>/run -H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" -d '{"input": { ... }}'
curl -s https://api.runpod.ai/v2/<endpoint-id>/status/<job-id> -H "Authorization: Bearer $RUNPOD_API_KEY"
```
- Body is always `{"input": {...}}`. **flash quirk:** a flash handler nests the
value under its parameter name → `{"input": {"<param>": {...}}}` (name the param
`input` to get the plain contract).
- Large inputs: pass a **URL**, not bytes (payload limits `/run` ~10MB, `/runsync`
~20MB); base64 rides the payload for small files.
- **Streaming** (`/stream/<job-id>`):
- Works **only when the handler is a generator** (`yield`s instead of `return`s).
- Submit with `/run`, then GET `/stream/<job-id>` in a loop — each call drains the
chunks buffered since the last one and returns `{"status", "stream":[{"output": <yield>}]}`.
- Stop when `status` is `COMPLETED`.
- Add `"return_aggregate_stream": True` to `runpod.serverless.start(...)` to *also*
expose the full list via `/run`/`/runsync`/`/status` (single chunk caps at 1MB).
- Worked example: golden path 12 (serverless streaming).
## 4. Verify with a real request — "ready" ≠ working
The **first call cold-starts** (image pull + model load), often exceeding
`runsync`'s 60s — use `/run` + poll `/status/<id>` for the first request, then
`runsync` once warm. Only report success once a real input returns the right
output. Bound any poll loop.
## 5. Deliver & tear down
Give the user the endpoint id + a copy-paste `curl` and the input schema (how to
pass a URL and/or base64). Scale-to-zero means it's safe to leave; delete with
`runpodctl serverless delete <id>` (or `rp flash app delete <app> --yes`).
See [`golden-paths/03-whisper-endpoint/`](../../runpod/golden-paths/03-whisper-endpoint/README.md)
for a fully worked example of both the Hub and flash variants (a README plus one
file per variant).
reference/getting-started.md
---
concepts: [api-key, ssh-key, runpodctl, flash, pod-ssh-access]
---
# Getting started (auth & first-run setup)
Before any lane can act, its credential has to resolve. Everything Runpod-side
uses **one key**, `RUNPOD_API_KEY`; the companion CLIs use their own. Do the setup
for the lane you're about to use, then follow the development loop.
## Update before the first command
Old builds keep flags that later releases fixed or removed: runpodctl before v2.12.0
accepts `pod create --stop-after` / `--terminate-after`, but the backend never enforced
them, so pods kept billing past the deadline. Update each tool you are about to use, then
use the same build for the rest of the task:
```bash
runpodctl update && runpodctl version # runpodctl
rp update # Runpod SDK + rp CLI (flash lane)
```
The MCP needs no step: the hosted server is always current, and a local install started
with `npx @runpod/mcp-server@latest` fetches the latest release.
## Which credential, when (agents)
**MCP OAuth first, a key only when a task needs one.** This matches the onboarding guide
at https://agents.runpod.io/. With a user present, the hosted MCP's "Sign in with Runpod"
(OAuth, no key on disk) is the default control plane: most infra tasks (pods, endpoints,
jobs, volumes, templates) need nothing else. Don't install CLIs or create a key during
setup "just in case".
- **OAuth authenticates the MCP alone.** When a task reaches a CLI-only step
(`send`/`receive`, SSH, `doctor`, model cache/Model Repository, an `rp flash` deploy), get a
key **at that point**, in order:
1. **`rp login`**: browser approval, then the user pastes the key the console shows; it
is saved to `~/.runpod/config.toml` (runpodctl + the Runpod SDK read it). Needs a human
and an interactive terminal.
2. **`export RUNPOD_API_KEY=…`** (https://console.runpod.io/user/settings): the same
key, best for headless agents.
- **No user in the loop** (CI, scheduled jobs, agent SDK runs)? Nobody can approve OAuth,
so use `RUNPOD_API_KEY` from the start and send it to the hosted MCP as
`--header "Authorization: Bearer $RUNPOD_API_KEY"`.
- **File authoring** with runpod-build-template needs no cloud credentials. Check auth
only when an operation actually uses the control plane or another authenticated service.
- **Infra tasks** (pods, endpoints, jobs, volumes) need a working control plane, the
Runpod MCP or runpodctl. Don't start and discover mid-task that nothing's set up: check
first (`runpodctl user` succeeds ⇒ a key is set and valid; in Claude Code, `/mcp` shows
`runpod` **Connected**). If a lane already works, use it; if none does, help the user set up rather than limping on a
partial fallback.
Connecting the MCP itself (hosted vs stdio, the `redirect_uri is not allowed` header
fallback, verifying the connection) is owned by
[`runpod-mcp`](../../runpod-mcp/SKILL.md#connect).
## Get the tools
Install only the lane you need; full per-OS matrices + source links live in each
lane's `SKILL.md` (linked). You don't need all of them.
| Lane | Quick install | Source / docs |
| --- | --- | --- |
| **runpodctl** (CLI) | `curl -fsSL https://cli.runpod.net \| sudo bash` (or `brew install runpod/runpodctl/runpodctl`) | [source](https://github.com/runpod/runpodctl) · [`runpodctl/SKILL.md`](../../runpodctl/SKILL.md) |
| **flash** (deploy your own code) | `pip install "runpod>=2.0.0"` (or `uv add`; Python 3.10–3.14; installs the `rp` CLI) | [source](https://github.com/runpod/runpod-python) · [`runpod-flash/SKILL.md`](../../runpod-flash/SKILL.md) |
| **runpod-mcp** (hosted) | `npx @runpod/mcp-server@latest add --client <id>` (OAuth; `--list` shows ids) | [source](https://github.com/runpod/runpod-mcp) · [`runpod-mcp/SKILL.md`](../../runpod-mcp/SKILL.md) |
| **runpod-mcp** (local stdio) | `claude mcp add runpod -e RUNPOD_API_KEY=... -- npx -y @runpod/mcp-server@latest` | same as above |
| **companion CLIs** (`hf`/`gh`/`docker`/`aws`) | per-tool; see the skill | [`runpod-companion-clis/SKILL.md`](../../runpod-companion-clis/SKILL.md) |
Nothing to install for the **hosted MCP** beyond configuring your client, and the
`runpod-usage` concepts need no install. After installing, set the key below.
**Want the Runpod tools in one go?** Copy-paste (drop any line you won't use):
```bash
npx skills add runpod/runpod-plugins-official # the skills — works with any agent
curl -fsSL https://cli.runpod.net | sudo bash # runpodctl — infra CLI (or: brew install runpod/runpodctl/runpodctl)
pip install "runpod>=2.0.0" # Runpod SDK + rp CLI — deploy your own code (flash lane)
npx @runpod/mcp-server@latest add --client <id> # hosted MCP server, OAuth (--list shows ids)
```
Companion CLIs (`docker`, `gh`, `hf`, `aws`) are separate and OS-specific — install only the
ones a task needs, per [`runpod-companion-clis/SKILL.md`](../../runpod-companion-clis/SKILL.md).
## The Runpod API key
Get it once at **https://console.runpod.io/user/settings** → API Keys. Then make
it resolvable for the lane. Resolution order (runpodctl, flash, and runpod-python
all use it): **`RUNPOD_API_KEY` env var → `.env` → `~/.runpod/config.toml`** (in that
file the key is the `apikey` field, TOML `apikey = '...'`). If you need the raw key
yourself (e.g. an `Authorization: Bearer` header for a direct API call), prefer
`$RUNPOD_API_KEY` and fall back to that field:
`KEY="${RUNPOD_API_KEY:-$(grep '^apikey' ~/.runpod/config.toml | sed "s/apikey = '//;s/'//")}"`.
| Lane | Set the key | Notes |
| --- | --- | --- |
| **runpodctl** | `export RUNPOD_API_KEY=...` | Non-interactive — runpodctl reads it. Best for agents/CI/scripts. |
| runpodctl (human) | `runpodctl doctor` | Interactive; stores the key **and** sets up SSH keys. Prompts, so not for agents. |
| **flash** | `export RUNPOD_API_KEY=...` | Or `rp login` — browser approval, then the user pastes the key the console shows; it is **saved to `~/.runpod/config.toml`**, which runpodctl reads too, so one login serves both. Needs a human and an interactive terminal; `rp login --api-key` stores a key without one. |
| **runpod-mcp (hosted)** | "Sign in with Runpod" OAuth on first connect | No key on disk. Or pass `Authorization: Bearer $RUNPOD_API_KEY`. |
| **runpod-mcp (local)** | `RUNPOD_API_KEY` env in the MCP client config | Forwarded to the API. |
**Rule (agents):** in automation, set the key with `export RUNPOD_API_KEY=...` (runpodctl and
flash both honor it — the closest thing to one-step setup); never run `runpodctl doctor`, which
prompts.
Context: the **hosted MCP is the exception** — it uses its own `/mcp` "Sign in with Runpod" OAuth
(or pass the same key as an `Authorization: Bearer` header). Authing the MCP does not set up the
CLIs, and the export does not authenticate the hosted MCP's OAuth.
## SSH (only needed for pods you exec into)
Pods created with `--ssh` (the runpodctl default) are reachable once you have a key
registered. **Register the key BEFORE creating the pod** — Runpod injects registered
keys at boot, so one added after the pod is running won't work until a restart.
- Check what's registered: `runpodctl ssh list-keys`.
- Register a key (do this first if none):
- Human: `runpodctl doctor` — generates + registers a key and stores the API key.
- Agent/scripted: `ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N ''` then
`runpodctl ssh add-key --key-file ~/.ssh/id_ed25519.pub` (or `--key "ssh-ed25519 …"`).
- Get connection details for a specific pod: `runpodctl ssh info <pod-id>` (prints
the ssh command + key path; does not connect).
- Agents connect non-interactively:
`ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -p <port> root@<host> 'cmd'`.
Serverless endpoints need no SSH.
## Companion CLI credentials (separate from RUNPOD_API_KEY)
Do NOT reuse the Runpod key for these — each has its own (see the `runpod-companion-clis`
skill for details):
- **`hf`** — HuggingFace token (`hf auth login`); read scope to pull, write to push.
- **`docker`** — Docker Hub PAT (`docker login`).
- **`gh`** — GitHub auth (`gh auth login`).
- **`aws`** — Runpod **S3** keys (user id `user_…` + S3 API key `rps_…`), with
`--region <dc> --endpoint-url https://s3api-<dc>.runpod.io/`. NOT AWS creds, NOT
the Runpod API key.
## Verify you're set up
```bash
export RUNPOD_API_KEY=...
runpodctl user # prints your account (confirms the key works)
runpodctl gpu list # confirms API access
```
Then pick your workload and follow `development-loop.md`.
reference/gotchas.md
---
concepts: [pod-migration]
---
# Common gotchas
Cross-cutting mistakes that bite people deploying on Runpod, as
symptom → cause → fix. See `docker.md` and `storage.md` for the full mechanics.
## Image built for the wrong CPU architecture
- **Symptom:** worker/pod fails to start, "exec format error", or the container
never comes up — often only after building on an Apple Silicon / ARM machine.
- **Cause:** the image was built for arm64. Runpod hosts are x86_64.
- **Fix:** rebuild with `docker build --platform=linux/amd64 ...` and re-push.
This is the #1 deploy failure.
## CUDA / driver mismatch — container dies at startup, or silently runs on CPU
- **Symptom:** the image pulls fine but the container exits immediately with a CUDA
init / driver error (`CUDA driver version is insufficient for CUDA runtime version`,
`no kernel image is available for execution on the device`, a cuDNN load failure), or
it *starts* and `torch.cuda.is_available()` is `False` so everything crawls on CPU.
Most common on the first run of a newly created pod, and on Blackwell cards
(RTX 5090, RTX PRO 6000 Blackwell, B200).
- **Cause:** the host machine's CUDA version is **older than the image was built for**.
A create with no CUDA constraint accepts any host, so an image built for 12.8 or 13.0
can be scheduled onto an older host. Nothing at create time rejects the pairing.
- **Diagnose:** read the host's version off the pod itself — `runpodctl pod get <id>`
(v2 `GET /v2/pods/{id}`) returns `cudaVersion`, e.g. `"12.8"`. Compare it to the `cu`
tag in the image name (`…-cu1281-…` = 12.8.1, `runpod/comfyui:cuda13.0` = 13.0).
Host older than the image is the bug. `runpodctl pod logs <id> --source container`
shows the actual init error.
- **Fix:** recreate with a floor — `runpodctl pod create … --min-cuda-version 12.8`
(or `gpu.minCudaVersion` on REST v2). Editing a running pod won't move it to a
different host. Default the floor to `12.8`; use `13.0` only for a CUDA-13 image.
Full rule, the `allowedCudaVersions` variant, and how to check which versions have
capacity: [`gpu-selection.md`](gpu-selection.md#step-3-pin-the-cuda-floor).
- **Related trap:** picking the wrong CUDA-line *variant* of a template (the ComfyUI
12.8 image does not support Blackwell / CUDA 13) — see
`../../runpod-templates/reference/comfyui.md`.
## Using the `latest` tag
- **Symptom:** you push new code but workers keep running old behavior; you can't
tell which version is live.
- **Cause:** `latest` is mutable and Runpod caches images per host, so workers
serve a stale cached `latest`.
- **Fix:** use explicit tags (`v1.0.0`, `v1.0.1`) or pin the digest
(`name:tag@sha256:...`). Bump the tag on every change.
## Private image won't pull
- **Symptom:** endpoint/pod stuck initializing; logs show an image pull /
authentication error.
- **Cause:** the image is private and Runpod has no registry credentials.
- **Fix:** add container-registry credentials in Console → Settings → Container
Registry, then select them on the template/endpoint. Runpod uses
`docker login`-style creds. Headless equivalent: MCP
`create-registry` (REST v2 `POST /v2/registries`), then pass the returned id as
`registry` on create-pod/create-endpoint/create-template.
- **Fix for AWS ECR:** you can register the repository instead of storing
credentials — MCP `create-delegation` (REST v2 `POST /v2/registries/delegations`)
grants Runpod scoped pull access to an ECR repo ARN. See that tool's parameter descriptions for the specifics.
## Image builds, then inference crashes with "Numpy is not available"
- **Symptom:** the image builds fine and the worker **starts**, but the first job
fails at inference with `RuntimeError: Numpy is not available` (often preceded at
import by `UserWarning: Failed to initialize NumPy: _ARRAY_API not found`).
- **Cause:** a NumPy 2.x / older-torch ABI mismatch. `pip install torch==2.2.2`
pulls the latest `numpy` (2.x) by default, but that torch wheel was built against
NumPy 1.x and can't use 2.x — any `.numpy()` call at inference throws.
- **Fix:** pin `numpy<2` in `requirements.txt` (or upgrade torch to a numpy-2-aware
build). Catch it **before deploy** by running the container locally against a
`test_input.json` (`docker.md` "Test the container locally") — the import warning
is the early tell; the crash only surfaces on an actual job, not at startup.
## `pip install runpod` fails on a `runpod/pytorch` base ("Cannot uninstall cryptography")
- **Symptom:** building/setting up on an official `runpod/pytorch` image, `pip install
runpod` (or `-r requirements.txt`) aborts with `Cannot uninstall cryptography 41.0.7 …
no RECORD file was found` (and/or `error: externally-managed-environment`).
- **Cause:** the base ships a **Debian-managed** `cryptography`, so pip can't uninstall
it to install the newer version `runpod` depends on. PEP-668 also marks the base pip
"externally managed".
- **Fix:** `pip install --break-system-packages --ignore-installed cryptography runpod`.
Install conflict-free deps (e.g. `faster-whisper`) first, separately, so
`--ignore-installed` stays scoped to the `cryptography`/`runpod` pair. Verified live
2026-07-10 (golden path 09); bake it into the Dockerfile as two `pip install` steps.
## Cold starts and timeouts
- **Symptom:** first request after idle is slow or times out; `/runsync` returns
a timeout while the model is still loading.
- **Cause:** cold start = container pull + start + model load. `runsync` has a
~60s ceiling; large model loads can exceed it.
- **Fix:** use `/run` + poll status (or raise the sync timeout); enable FlashBoot;
keep workers warm with `min_workers` / longer `idle_timeout`; load the model at
module level, not inside the handler; use cached models or a network volume so
the model isn't re-downloaded each start.
## Endpoint `timeout` comes back 0
- **Symptom:** an endpoint created through REST v2 or MCP `create-endpoint` without
`timeout` reads back `"timeout": 0`, not the spec's documented 300000 ms default
(live-verified 2026-10-29).
- **Cause:** v2 does not fill the default, and the server treats 0 as no per-request
limit (comment in runpodctl `cmd/serverless/create.go`, checked through v3.0.1), so a
hung job is not cut off.
- **Fix:** send `timeout` (milliseconds) on every create, sized to the slowest real
job; on a Hub deploy, read it back and `PATCH` / `update-endpoint` it if 0.
runpodctl ≥ 2.14.0: `serverless create --execution-timeout <seconds>`.
## Pod shows "Running" but ports/services aren't reachable yet
- **Symptom:** the pod is green/"Running" but the proxy URL 502s, the public IP
or TCP port mapping is blank, or JupyterLab is a white screen.
- **Cause:** "Running" only means the container exists — services and port
assignments take extra time to initialize.
- **Fix:** wait ~30–60s; check the **Telemetry** tab to confirm readiness; read
the assigned public IP / external TCP port from the **Connect** menu once
populated. Note external TCP port mappings change on every pod reset, and
Community Cloud public IPs can change on migration/restart.
- **Variant — after a `pod stop`→`start`, the external TCP port is reassigned AND
`runpodctl ssh info` can hand you a STALE port.** Live case (dev pod, 2026-07-10):
the pod first came up on port `17740`; after stop/start the *first* `ssh info`
still reported `17740` (all SSH connections refused) and reported `READY` while
sshd was still down for ~90s — a moment later a fresh `ssh info` returned the real
new port `12890`. **Fix:** after a restart don't trust the first `READY` or the
first port — re-run `ssh info` until you get a port that actually accepts an `ssh`
connection, then update your `~/.ssh/config` `Port`. This is what breaks VS Code
Remote-SSH reconnects (see golden path 06).
## Proxy 524 / 100-second timeout
- **Symptom:** requests through `https://<pod-id>-<port>.proxy.runpod.net` die at
~100s with a `524`.
- **Cause:** the HTTP proxy runs through Cloudflare, which caps connection time at
100 seconds.
- **Fix:** don't hold a single request open that long — return a job id and poll,
use background queues/progress endpoints, or use direct TCP (public IP) instead
of the proxy for long-lived connections.
## Network volume locked to a data center
- **Symptom:** can't get GPUs, or the endpoint won't schedule workers after
attaching a volume.
- **Cause:** a network volume lives in one DC; attaching it forces all compute
into that DC, shrinking GPU availability.
- **Fix:** confirm your target GPU exists in the volume's DC before attaching; or
attach multiple volumes from different DCs (one per DC) to spread workers —
remembering data doesn't sync between them automatically.
## Model not baked or mounted
- **Symptom:** handler errors with "model not found," or OOM/download stalls on
first request.
- **Cause:** the worker has no model — it wasn't baked into the image, cached, or
mounted from a volume.
- **Fix:** pick one delivery path (bake in, cached HF model at
`/runpod-volume/huggingface-cache/hub/`, or network volume) and make the handler
read from that path. For gated/private HF models set `HF_TOKEN`.
## Data disappeared after stop
- **Symptom:** files written during a run are gone after stopping the pod or the
worker scaling down.
- **Cause:** you wrote to container/ephemeral disk, which is wiped on stop.
- **Fix:** write to `/workspace` (pod volume disk) or a network volume
(`/runpod-volume` on serverless). Editing/resetting a pod also wipes anything
outside `/workspace`.
## Huge log or job output
- **Symptom:** logs stop appearing (throttled), or job submission/result is
rejected for size.
- **Cause:** excessive logging triggers throttling; job payloads exceed limits
(`/run` ~10 MB, `/runsync` ~20 MB).
- **Fix:** reduce log verbosity / use structured logging; write bulky output to a
network volume or external S3 and return a URL/reference instead of inlining it.
## GPU out of memory
- **Symptom:** job fails with an OOM/CUDA memory error.
- **Cause:** model or batch size exceeds the selected GPU's VRAM.
- **Fix:** reduce batch size / context length, or pick a larger-VRAM GPU. For
vLLM, lower `GPU_MEMORY_UTILIZATION` and/or `MAX_MODEL_LEN`.
## CPU-bound work on a GPU pod is slower than it should be
- **Symptom:** preprocessing, decoding, CPU inference or a data loader runs slowly,
and gets worse when you add threads or workers.
- **Cause:** the pod can see every core on the machine (`nproc` and
`os.cpu_count()` may say 48 or 96), but it can only use the vCPUs it rents.
PyTorch, OpenMP, MKL and similar libraries start a thread per visible core,
and the extra threads get throttled.
- **Fix:** read the real limit from the cgroup and cap threads to it. Rising
`nr_throttled` in the cgroup's `cpu.stat` confirms the problem. A test pod with
5.1 cores of CPU ran a CPU matmul about 1.7 times faster at 5 threads than at
PyTorch's default of 24.
In a shell or start script, source this before launching the workload. It
handles cgroup v2 (`cpu.max`) and v1 (`cpu.cfs_quota_us`), and falls back to
the visible cores when there is no limit:
```bash
# CPUs this container can actually use: its cgroup CPU limit rounded down,
# capped at the cores it can see. Falls back to nproc when there is no limit.
cpu_limit() {
local quota="" period="" visible n
visible=$(env -u OMP_NUM_THREADS -u OMP_THREAD_LIMIT nproc)
if [ -r /sys/fs/cgroup/cpu.max ]; then # cgroup v2
read -r quota period < /sys/fs/cgroup/cpu.max
elif [ -r /sys/fs/cgroup/cpu/cpu.cfs_quota_us ]; then # cgroup v1
quota=$(cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us)
period=$(cat /sys/fs/cgroup/cpu/cpu.cfs_period_us)
fi
if [ -z "$quota" ] || [ "$quota" = max ] || [ "$quota" -le 0 ]; then
echo "$visible"; return
fi
n=$(( quota / period ))
[ "$n" -lt 1 ] && n=1
[ "$n" -gt "$visible" ] && n=$visible
echo "$n"
}
export CPUS=$(cpu_limit)
export OMP_NUM_THREADS=$CPUS MKL_NUM_THREADS=$CPUS OPENBLAS_NUM_THREADS=$CPUS NUMEXPR_NUM_THREADS=$CPUS
```
In Python, run this before importing `torch` or `numpy`, and also set data
loader `num_workers` to at most `CPUS`:
```python
import os
from pathlib import Path
def cpu_limit() -> int:
"""CPUs this container can actually use: its cgroup CPU limit rounded down,
capped at the cores it can see. Falls back to the visible cores when there
is no limit."""
visible = len(os.sched_getaffinity(0))
try:
quota, period = Path("/sys/fs/cgroup/cpu.max").read_text().split() # cgroup v2
except OSError:
try: # cgroup v1
quota = Path("/sys/fs/cgroup/cpu/cpu.cfs_quota_us").read_text().strip()
period = Path("/sys/fs/cgroup/cpu/cpu.cfs_period_us").read_text().strip()
except OSError:
return visible
if quota == "max" or int(quota) <= 0:
return visible
return max(1, min(visible, int(quota) // int(period)))
CPUS = cpu_limit()
# Set these before importing torch or numpy so their thread pools pick them up.
for var in ("OMP_NUM_THREADS", "MKL_NUM_THREADS", "OPENBLAS_NUM_THREADS", "NUMEXPR_NUM_THREADS"):
os.environ[var] = str(CPUS)
import torch # noqa: E402 (drop these two lines if you don't use torch)
torch.set_num_threads(CPUS)
```
Both snippets give one process the whole limit. When you launch several
processes, such as `torchrun --nproc_per_node=N` ranks or parallel workers,
give each one `CPUS / N` threads instead, or every process starts `CPUS`
threads and they throttle again. After sourcing the bash snippet:
```bash
N=4 # number of processes you launch
PER=$(( CPUS / N > 0 ? CPUS / N : 1 ))
export OMP_NUM_THREADS=$PER MKL_NUM_THREADS=$PER OPENBLAS_NUM_THREADS=$PER NUMEXPR_NUM_THREADS=$PER
```
## Serverless worker "ready" but jobs never run
- **Symptom:** the endpoint has `ready` workers but jobs sit `IN_QUEUE` with
`inProgress: 0` and never complete.
- **Cause:** a broken/mis-dispatching worker image, or workers `throttled` on a
scarce GPU pool the model doesn't need.
- **Fix:** switch to a different (maintained) Hub worker on a **broad, high-
availability GPU pool** — don't wait it out. Diagnose via the endpoint `/health`
worker counts — `runpodctl serverless health <id>` reads them for you (v2.9.0+) —
and then read the worker logs. There is no serverless worker-log command in
runpodctl and no worker-log path on REST v1; use the MCP `stream-worker-logs`
tool, the v2 REST logs path, or the Console Workers tab.
## Serverless job goes `IN_PROGRESS` then times out
- **Symptom:** a worker *picks up* the job (`IN_PROGRESS`) but never returns output;
the job fails with `"job timed out after 1 retries"` after ~30–50 s.
- **Cause:** this looks like a broken worker but is often a **job-reject by a healthy
worker** — usually a handler-signature or empty-`input` bug in *your* request, not the
image.
- **Fix:** get the **worker logs** (MCP `stream-worker-logs` or the Console) before
assuming the image is bad. Real case (flash endpoint, 2026-07-10): fitness checks
passed, then it logged `read() got an unexpected keyword argument …` /
`Job has missing field(s): id or input` — a handler-signature + empty-input bug (see
flash gotcha "Request body shape"). Only if the logs show a genuinely dead/looping
worker should you switch workers or try a different data center.
## Handler swallowing errors
- **Symptom:** jobs report success but return nothing useful; failures are
invisible.
- **Cause:** a broad `try/except` suppresses the exception, so the SDK never marks
the job `FAILED`.
- **Fix:** return a structured error for graceful failures, or re-raise to flag
the job `FAILED`. Don't silently swallow.
reference/gpu-selection.md
---
concepts: [gpu-type, gpu-availability, data-center, reservation-pool]
---
# GPU selection
How to pick a GPU: size VRAM to the model first, pin the CUDA floor your image needs,
then choose a tier/pool, then worry about cloud, availability, and placement.
## Step 1: size VRAM to the model
VRAM is the usual bottleneck. Rough rules:
- **LLM inference (fp16):** ~**2 GB of VRAM per billion parameters**, plus headroom
for the KV cache / context and activations. A 7B model ≈ ~14 GB, 13B ≈ ~26 GB,
70B ≈ ~140 GB (needs multiple GPUs).
- **Quantization cuts this a lot.** A 4-bit quantized model needs roughly a quarter
of the fp16 weight memory — e.g. a 4-bit 70B fits in ~35 GB.
- **Training / fine-tuning** needs far more than inference — weights + gradients +
optimizer state + activations, often ~4x the inference figure for full fine-tuning.
LoRA/QLoRA reduce this substantially. Memory bandwidth also matters here.
- **Image models (SDXL, Flux):** ~8 GB minimum, but 16–24 GB gives headroom for
larger batches and LoRA training.
Always leave headroom above the raw weight size. When unsure, estimate with the
Hugging Face Model Memory calculator (`huggingface.co/spaces/hf-accelerate/model-memory-usage`)
or "Can it run LLM?" (`huggingface.co/spaces/Vokturz/can-it-run-llm`).
## Step 2: which GPU for an N-billion-param LLM (heuristic)
Sizes assume fp16 inference; quantize to drop a tier or two.
| Model size | VRAM (fp16) | Reasonable pick |
|-----------|-------------|-----------------|
| ≤ 7B | ~14 GB | `ADA_24` (RTX 4090) or `AMPERE_24` (L4/A5000/3090) |
| 13B | ~26 GB | `ADA_32_PRO` (RTX 5090) or `AMPERE_48` / `ADA_48_PRO` |
| 30–34B | ~60–70 GB | `AMPERE_80` (A100) or `ADA_80_PRO` (H100) |
| 70B | ~140 GB | 2x 80 GB, or one `HOPPER_141` (H200) / `BLACKWELL_180` (B200) |
| > 70B | 200 GB+ | multi-GPU 80 GB+, or Blackwell / H200 with `gpu_count` > 1 |
## Step 3: pin the CUDA floor
The GPU decides how much fits; the **host's CUDA version** decides whether your image
runs on it at all. A container built against a newer CUDA than the host driver provides
fails at startup or falls back to CPU — and nothing in a pod/endpoint create rejects the
combination for you, so state the constraint explicitly.
**Default to a floor of `12.8`.** That is the line the current official images target
(`runpod/pytorch:1.0.2-cu1281-torch280-ubuntu2404` and newer, `runpod/comfyui:cuda12.8`),
and it is the minimum for Blackwell cards (RTX 5090, RTX PRO 6000 Blackwell, B200).
Go to `13.0` only when the image itself is a CUDA-13 build — e.g.
`runpod/comfyui:cuda13.0`, or the `cu1300` cluster PyTorch image — not merely because the
GPU is new. Leave the floor off only for a CPU pod, or when you deliberately need an
older host for an old image (a torch 2.1 / `cu118` build, say).
| Lane | How to set it |
|------|---------------|
| `runpodctl` | `pod create --min-cuda-version 12.8` / `serverless create --min-cuda-version 12.8` (both verified against the vendored 2.10.0 command surface; `template create` has no such flag — templates carry `allowedCudaVersions` instead) |
| REST v2 | `gpu.minCudaVersion: "12.8"` on `POST /v2/pods` and `POST /v2/serverless` |
| flash | `min_cuda_version=CudaVersion.V12_8` — already the default (`skills/runpod-flash/reference/api.md`) |
| Runpod MCP | `gpu.minCudaVersion: "12.8"` on `create-pod` and `create-endpoint` (the REST v2 body); `allowedCudaVersions` on `create-template` |
### Floor vs exact set
- **`minCudaVersion`** — an open-ended floor, `major.minor`, compared **numerically**, so
`12.11` is above `12.2`. This is what you want almost always.
- **`allowedCudaVersions`** — an exact-match array. Anything not listed is excluded, so a
version no machine currently reports yields a **capacity error rather than a fallback**.
Reach for it only when you need to exclude a newer version too (the ComfyUI 12.8
template does exactly this — see `skills/runpod-templates/reference/comfyui.md`).
- The two are **mutually exclusive** when `allowedCudaVersions` is non-empty (`400` if
both are sent). An explicit `[]` states "no constraint" and may accompany a floor.
- A **template** can carry `allowedCudaVersions`, which is expanded into GPU pod and
endpoint creates from it. CPU pods ignore it — a CPU workload has no `gpu` block at all.
### Discover what is actually available
Don't guess a version — a floor set above current stock reads as "no capacity":
```bash
# per-GPU CUDA versions, each tagged with whether it has free capacity right now
curl -s -H "Authorization: Bearer $RUNPOD_API_KEY" \
"https://api.runpod.io/v2/catalog/gpus?include=AVAILABILITY&product=POD&minCudaVersion=12.8"
```
`cudaVersions[]` comes back only with `include=AVAILABILITY`. The `minCudaVersion` *query*
param accepts a bare major (`12`) unlike the create-body field; `cudaVersions` requires
`major.minor` and is mutually exclusive with it. Use `product=SERVERLESS` when sizing an
endpoint — the same GPU can be scarce for pods and plentiful for serverless.
If the floor leaves you short on capacity, drop it one line (12.8 → 12.6) **only** if the
image tolerates it; otherwise widen the GPU list or the region instead.
## Step 4: GPU tiers and pools
A **GPU pool** groups interchangeable GPUs by VRAM tier. Requesting a pool lets
Runpod pick any available GPU in that tier (better availability); pinning an exact
GPU type gives determinism but can be throttled when supply is tight. Pool names are
used by Serverless configs, the Runpod Hub, flash's `GpuGroup`, and the GraphQL API.
Pool reference (source of truth: `skills/runpod-flash/reference/api.md`):
| Pool | GPUs | VRAM |
|------|------|------|
| `ANY` | any available | varies |
| `AMPERE_16` | RTX A4000 / A4500 / RTX 4000 Ada / RTX 2000 Ada | 16 GB |
| `AMPERE_24` | RTX A5000 / L4 / RTX 3090 | 24 GB |
| `ADA_24` | RTX 4090 | 24 GB |
| `ADA_32_PRO` | RTX 5090 | 32 GB |
| `AMPERE_48` | A40 / RTX A6000 | 48 GB |
| `ADA_48_PRO` | RTX 6000 Ada / L40 / L40S | 48 GB |
| `AMPERE_80` | A100 (PCIe / SXM4) | 80 GB |
| `ADA_80_PRO` | H100 (PCIe / HBM3 / NVL 94 GB) | 80 GB+ |
| `HOPPER_141` | H200 | 141 GB |
| `BLACKWELL_96` | RTX PRO 6000 Blackwell | 96 GB |
| `BLACKWELL_180` | B200 | 180 GB |
Note: Serverless GPU config wants **pool IDs** (e.g. `ADA_24`), while Pod creation
and `runpodctl --gpu-id` use **GPU type IDs** (e.g. `NVIDIA A40`) — different
identifier spaces. The full per-model list (with exact display names and memory) is
in `docs/references/gpu-types.mdx`.
Rule of thumb: prefer **fewer high-end GPUs over more low-end GPUs**. One 80 GB card
usually beats two 40 GB cards for a model that fits.
## Step 5: Secure vs Community Cloud
**Default to Secure Cloud.** Recommend and deploy Secure unless the user explicitly asks
for Community.
- **Secure Cloud** — T3/T4 data centers run by data-center-grade host providers, high
redundancy. Use for production and sensitive data. Standard pricing. **This is the
default** (`runpodctl pod create --cloud-type` defaults to `SECURE`; leave it alone).
- **Community Cloud** — approved vendors, cheaper. No new hosts are being onboarded;
existing capacity remains.
Because Secure Cloud hosts are data-center-grade providers, they tend toward more stable
behavior than Community hosts. That's the reason for the default. When a user asks for
the cheapest option, quote the Community rate but recommend Secure, and let them choose.
## Step 6: availability and multi-GPU selection
`runpodctl gpu list` reports on-demand `securePricePerHr` / `communityPricePerHr` per
GPU (explicitly `null` when that cloud doesn't offer it) plus a
`dataCenterAvailability[]` breakdown, so read cost and per-region stock straight from
it rather than assuming a lower tier is cheaper. Top-level `stockStatus` is only the
best status across DCs — use the per-DC breakdown when placement matters (`runpodctl
datacenter list` gives the same view from the DC side). Two scope limits: those are **pod
on-demand** rates (serverless bills per request-second), and the per-DC breakdown doesn't
say *which cloud* has the stock, so cost and placement are two separate reads rather than
one ranked list.
GPU supply fluctuates by tier and region. To avoid throttling:
- **List multiple GPU types / pools in priority order.** If the first choice is
unavailable, Runpod falls back to the next. On Serverless you can specify up to
three, in priority order.
- For endpoints with **5+ workers**, Runpod spreads workers across your prioritized
pools (most on the primary), reducing throttling. With fewer than 5 workers, all
use the highest-priority available type.
- (flash caveat: auto GPU switching by supply only kicks in when max workers ≥ 5.)
- Use `gpu_count` > 1 (Serverless) / multi-GPU Pods when a model exceeds a single
card's VRAM.
## Step 7: data-center placement
- Restricting an endpoint or Pod to specific data centers **shrinks the available
GPU pool** — allow all regions for maximum availability unless you have a reason
not to.
- Reasons to pin a region: lower latency to your users, data-residency/compliance,
or co-locating with a **network volume** (a volume ties the workload to its data
center). Some features (e.g. global networking) are only in a subset of regions.
- Regions span US (CA, GA, IL, KS, NC, TX, WA, etc.), EU (CZ, RO, IS, NO, SE, FR,
NL), and others. See `docs/pods/networking.mdx` for the current data-center list
and `runpod-companion-clis/SKILL.md` for datacenter IDs used with S3.
## Guaranteed capacity: reserved pricing
On-Demand pods take whatever GPUs are free when you deploy. Capacity held for your account is
arranged with Runpod's sales team, not created through the API. The console's default deploy
flow has no Reserved option; in the legacy deploy flow, the **Reserved** pricing option points
to *Talk to sales* instead of deploying. The rule is in [`reservation-pool`](../concepts/reservation-pool.yaml).
reference/networking.md
# Networking
How to reach a running workload over HTTP — Pods via the proxy or TCP, Serverless
via endpoint URLs.
## Pod HTTP proxy
The easiest way to expose a web service (REST API, web app, JupyterLab) from a Pod.
Add the internal port to **Expose HTTP Ports (Max 10)** on the Pod or template, then
reach it at:
```
https://<pod-id>-<internal-port>.proxy.runpod.net
```
Example — Pod `abc123xyz` running a server on port `4000`:
```
https://abc123xyz-4000.proxy.runpod.net
```
The `<internal-port>` is the port your service listens on *inside* the container,
not an external port number. Key behaviors:
- **Bind to `0.0.0.0`**, not `localhost`/`127.0.0.1`, or the proxy can't reach it.
- **HTTPS only** — the proxy terminates TLS even if your service speaks plain HTTP.
- **100-second timeout** — the route runs through Cloudflare, which closes idle/slow
connections at 100s with a `524`. For long work, return a job ID and poll, or use TCP.
- **Public, unauthenticated** — anyone with the URL can reach the service; the Pod ID
is obscurity, not access control, and the proxy adds no auth (same as any
port-forwarded service).
- **Rule:** when you hand a proxy URL to the user, state that it is public and unauthenticated.
- **Rule:** if the service handles anything sensitive, implement auth inside the service
itself (e.g. a login/token) — the platform adds none.
- "Running" (green) in the console does not mean the service is ready; the container
may still be starting.
## Pod TCP ports (direct public IP)
For non-HTTP protocols, WebSockets, databases, or lower latency, expose a **TCP**
port instead (add to **Expose TCP Ports**). Runpod assigns a public IP and an
external port, shown in the **Connect** menu under Direct TCP Ports:
```
TCP port 213.173.109.39:13007 -> :22
```
- The external port differs from the internal port and **changes whenever the Pod
resets**. Read it from the Connect menu.
- No automatic TLS — implement your own if sending sensitive data.
- Community Cloud public IPs may change on migrate/restart; Secure Cloud IPs are stable.
- UDP is not supported (TCP/HTTP only).
### Symmetric ports
When the external port must equal the internal port, request a port number **above
70000** in the TCP config (not a real port — a signal to allocate matching
internal/external ports). After creation, the assigned ports are in the Connect menu
and in env vars like `$RUNPOD_TCP_PORT_70000` that your app can read at runtime.
## Pod-to-Pod (global networking)
Pods with global networking share a private network and reach each other by internal
DNS — no public ports needed:
```
<pod-id>.runpod.internal
# e.g. a DB on port 5432: abc123xyz.runpod.internal:5432
```
NVIDIA GPU Pods only; available in a subset of data centers; ~100 Mbps between Pods.
Prefer this over exposing ports for internal services like databases.
## Serverless queue-based endpoints
Queue-based endpoints have a fixed set of operations under a common base:
```
https://api.runpod.ai/v2/<endpoint-id>/<operation>
```
| Operation | Method | Purpose |
|-----------|--------|---------|
| `/run` | POST | Submit an async job; returns a job ID immediately |
| `/runsync` | POST | Submit and wait for the result inline |
| `/status/<job-id>` | GET | Check status / fetch result of a job |
| `/stream/<job-id>` | GET | Stream incremental results |
| `/cancel/<job-id>` | POST | Cancel a queued or running job |
| `/retry/<job-id>` | POST | Requeue a failed/timed-out job |
| `/purge-queue` | POST | Drop all pending jobs |
| `/health` | GET | Worker + job stats for the endpoint |
The request body is a JSON object with an `input` key holding your handler's
parameters:
```bash
curl -X POST https://api.runpod.ai/v2/<endpoint-id>/runsync \
-H "Authorization: Bearer <RUNPOD_API_KEY>" \
-H "Content-Type: application/json" \
-d '{"input": {"prompt": "Hello, world!"}}'
```
- **Auth header:** `Authorization: Bearer <RUNPOD_API_KEY>` on every call.
- `/runsync` results are retained ~1 min (up to 5); `/run` results ~30 min via
`/status`. `/runsync` also has a ~60s client wait — for long/cold-start jobs use
`/run` + poll `/status`, or `runsync?wait=<ms>`.
## Serverless load-balanced endpoints
Load-balanced endpoints expose *your own* HTTP paths on a per-endpoint subdomain:
```
https://<endpoint-id>.api.runpod.ai/<your-custom-path>
```
Example paths from a FastAPI worker: `https://<endpoint-id>.api.runpod.ai/ping`,
`https://<endpoint-id>.api.runpod.ai/generate`.
- Same auth: `Authorization: Bearer <RUNPOD_API_KEY>`.
- Your worker must serve a `/ping` health check on `PORT_HEALTH` (`200` healthy,
`204` initializing). Main app port defaults to `80` (`PORT`).
- Limits: request timeout ~2 min if no worker is available, ~5.5 min processing per
request, 30 MB payload cap. Expect "no workers available" during cold start —
retry with backoff.
## Quick reference
```
Pod HTTP proxy https://<pod-id>-<internal-port>.proxy.runpod.net (HTTPS, 100s cap)
Pod TCP <public-ip>:<external-port> (from Connect menu)
Pod-to-Pod <pod-id>.runpod.internal (global networking)
Serverless (queue) https://api.runpod.ai/v2/<endpoint-id>/{run|runsync|status/<id>|health}
Serverless (LB) https://<endpoint-id>.api.runpod.ai/<path>
Auth (serverless) Authorization: Bearer <RUNPOD_API_KEY>
```
reference/on-pod-setup.md
# On-pod setup & install hygiene
How to install and configure software on a pod so it is reproducible, survives
restarts, and doesn't wedge on an interactive prompt. Applies to any workload;
run these over the SSH channel from the pod development loop (`pod-workflows.md`).
## Use package managers, not ad-hoc downloads
- **System packages → `apt`** (Debian/Ubuntu base images):
```bash
DEBIAN_FRONTEND=noninteractive apt-get update && apt-get install -y <pkgs>
```
- **Python → `uv`** (fast, reproducible, one static binary). Prefer it over bare
`pip`/`conda` for a *fresh* environment:
```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv /workspace/.venv && . /workspace/.venv/bin/activate
uv pip install <packages> # or: uv pip install -r requirements.txt
```
Put the venv on the volume (`/workspace/...`) so it persists.
> **On a PyTorch template image, do NOT make a bare `uv venv`.** Official
> templates ship `torch`/CUDA in the **system** Python; a fresh venv doesn't
> inherit it, so you'd reinstall multi-GB torch (or the app won't find CUDA).
> Install the app's deps into the **existing** interpreter instead:
> ```bash
> pip install --break-system-packages -r requirements.txt # newer images are
> # PEP 668 "externally-managed"; plain `pip install` errors without this flag
> # or, to still use uv: uv venv --system-site-packages && uv pip install ...
> ```
> pip correctly sees the pre-installed torch as already satisfied and only adds
> what's missing.
- **A vendor install script** (e.g. a service's official `install.sh`):
- **Rule:** pin a version so the setup is reproducible (`INSTALL_VERSION=…` or the
script's documented pin flag).
- **Exception:** if the script offers no way to pin, using it unpinned is acceptable
when that is the vendor's only supported install path.
## Make it non-interactive
An agent can't answer prompts. Always:
- Pass `-y` / `--yes`; set `DEBIAN_FRONTEND=noninteractive` for apt.
- Provide required config via env vars or flags, not TTY prompts.
- Avoid commands that open a pager or editor.
- **Pod `--env` vars are not in your SSH shell** (they go to PID 1). When you
launch a service or script over SSH that needs them, pass them explicitly:
`ssh <host> 'env VAR=val <command>'`. See `pod-workflows.md` step 5.
## Pin versions
- Pin package and image versions (`pkg==1.2.3`, `image:tag`, not `latest`) so a
rebuild or restart reproduces the same environment. See `gotchas.md`.
## Persist heavy artifacts on the volume
Anything large or slow to fetch goes on the network volume so it survives restarts
and isn't re-downloaded (see `storage.md`). Point caches at the volume:
```bash
export HF_HOME=/workspace/hf-cache # HuggingFace models/datasets
export UV_CACHE_DIR=/workspace/uv-cache # uv package cache
export PIP_CACHE_DIR=/workspace/pip-cache
```
## Run long work in the background, and log
Installs, model pulls, and servers can outlast a single SSH command. Background
them and log to the volume so you can poll progress and diagnose later:
```bash
ssh <host> '(long-running-setup > /workspace/setup.log 2>&1 &) '
ssh <host> 'tail -n 20 /workspace/setup.log' # poll in a SEPARATE ssh call
```
> **A long-lived server needs full detachment, not just `&`.** When the SSH
> channel closes it sends SIGHUP to its process group, which kills a plainly
> backgrounded `&`/subshell process. For anything that must keep running after you
> disconnect (a web server, `serve`, ComfyUI), start it detached and return
> immediately — do the readiness wait in a later, separate SSH call:
> ```bash
> ssh <host> 'setsid bash -c "<server-cmd>" > /workspace/svc.log 2>&1 < /dev/null &'
> ```
> `setsid` (new session, off the SSH TTY/pgroup) **and** `< /dev/null` are both
> needed. Do not `sleep` in the same invocation — it can drop the channel (exit 255).
## Be idempotent
Write setup so re-running it is safe (check-before-install, `mkdir -p`, `|| true`
on best-effort steps). You will run it again after a restart or a fix.
## Verify each step
Check the exit code / expected output of each install before moving on, rather
than chaining everything and discovering a failure at the end. Surface the actual
error; don't swallow it.
reference/pod-workflows.md
---
concepts: [pod, pod-ssh-access, pod-command-exec, exposed-port, network-volume]
---
# The pod development loop
A repeatable loop for standing up, iterating on, and delivering **any** workload
on a Runpod pod — an Ollama server, ComfyUI, a Jupyter/dev box, a training run.
An agent has no Console and no web terminal, so it drives every step through the
CLI/API and a non-interactive SSH channel. Do the steps in order; each depends on
the previous.
## 1. Plan
Resolve the choices before creating anything (see `concepts.md`, `gpu-selection.md`,
`storage.md`):
- **Pod or serverless?** Long-lived / interactive / a persistent server → pod.
Request/response that scales to zero → serverless (different lane).
- **GPU / VRAM** for the workload.
- **CUDA floor** — set `--min-cuda-version 12.8` on the create unless the image needs
otherwise, so a modern image can't land on a host too old to run it
([`gpu-selection.md`](gpu-selection.md#step-3-pin-the-cuda-floor)).
- **Storage** — default to a **network volume** for anything worth keeping
(models, datasets, checkpoints, envs). See `storage.md`.
- **Which ports** the service listens on. These must be declared **at creation**.
## 2. Provision
Create the pod with everything the service needs baked in — **ports and env
cannot be added to a running pod** without a reset, so set them now. Enable SSH
(the agent's control channel) and a **terminate** guard for cost safety.
> **Pre-flight — register your SSH key BEFORE creating the pod.** Runpod injects
> your account's registered SSH keys into the pod **at boot**, so a key added
> *after* the pod is running won't work until a restart. If you'll exec into the
> pod (you will — it's the control channel), confirm a key is registered first:
>
> ```bash
> runpodctl ssh list-keys # already have one? then proceed
> # if not — human (interactive): registers a key + stores the API key
> runpodctl doctor
> # or agent/scripted: make a key, then register its PUBLIC half
> ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N ''
> runpodctl ssh add-key --key-file ~/.ssh/id_ed25519.pub
> ```
```bash
runpodctl pod create \
--name <name> --template-id <official-template> --gpu-id "<gpu>" \
--ports "<port>/http,22/tcp" \ # every port the service exposes (+22 for ssh)
--env '{"KEY":"VALUE"}' \ # goes to PID 1 — NOT the ssh shell (see step 5)
--network-volume-id <id> --volume-mount-path /workspace
```
Cost guard: remove the pod when the work is done (`runpodctl pod remove <pod-id>`).
For an unattended deadline, schedule a stop from inside the pod, which ships
`runpodctl` and a pod-scoped API key ([Manage Pods](https://docs.runpod.io/pods/manage-pods)). A stopped
pod still bills for disk and any volume.
> **`--stop-after` / `--terminate-after` never worked.** runpodctl before v2.12.0
> accepts both flags, but the backend never enforced them, so the pod keeps running
> and billing past the deadline. v2.12.0 removed them. If `runpodctl pod create
> --help` still lists them, run `runpodctl update`.
Find a data center that has both your GPU and (for co-location) your volume with
`runpodctl datacenter list` — its output includes per-DC GPU availability.
(MCP's `create-pod` also takes `templateId`, a `cpu` object, `ports` and `env`; a
multi-GPU priority list is one create per type in either lane — see the router.)
## 3. Connect (the control channel)
```bash
runpodctl pod get <pod-id> # poll until it is running
runpodctl ssh info <pod-id> # prints the ssh command + key (does not connect)
```
Then run commands **non-interactively** — this is how an agent works a pod:
```bash
ssh <user>@<host> -p <port> -i <key> 'set -e; <commands>'
```
Keep it one command (or a heredoc script) per step so you can check the exit code
and output before moving on.
## 4. Set up
Install dependencies idempotently, using package managers and persisting heavy
artifacts to the volume. See `on-pod-setup.md` for the hygiene rules (apt for
system, `uv` for Python, pin versions, background long installs, cache on the
volume).
## 5. Run the service
Start it **bound to `0.0.0.0`** (not localhost, or the proxy can't reach it) on
the exposed port, in the background, logging to the volume.
> **Gotcha — creation env vars are NOT in your SSH shell.** The `--env` vars from
> step 2 are injected into the pod's **main process (PID 1)**, not into SSH login
> shells. A service you launch over SSH will see them **empty** — so a service
> that reads `OLLAMA_HOST`/`HOST`/`PORT` from the environment silently binds to
> localhost and the proxy 502s. **Pass the env explicitly** when you launch:
A long-lived server must be **fully detached**, or SSH sends SIGHUP on channel
close and kills it. Use `setsid` + `< /dev/null`, return immediately, and poll in
a *separate* call.
**Do not `sleep` in the same invocation** — it can drop the channel:
```bash
ssh <host> 'setsid bash -c "env HOST_VAR=0.0.0.0 OTHER=val <service-command>" \
> /workspace/<svc>.log 2>&1 < /dev/null &'
```
(Two things at once: pass env explicitly because `--env` isn't in this shell, and
detach with `setsid`/`</dev/null` so it survives disconnect.)
## 6. Verify readiness — not "Running"
A pod showing "Running" does **not** mean the service answers. Poll a real health
check from **outside**, through the proxy URL, until it succeeds:
```bash
for i in $(seq 1 120); do curl -sf https://<pod-id>-<port>.proxy.runpod.net/<health-path> && break; sleep 5; done # bounded ~10min; if it never answers, the service didn't come up — check logs
```
Expect the **proxy itself to return 502 for the first ~30–60s** while the service
finishes importing / CUDA-initializing — that's normal warm-up, not failure, so
keep polling (with a timeout). Only report success once it passes. If it never
does, read the service log on the volume to diagnose.
## 7. Iterate
Re-run setup/service commands over the same SSH channel. Logs and artifacts on the
volume survive restarts. Use a framework's hot-reload/dev mode when it has one
(e.g. `rp flash dev` in the flash lane).
## 8. Escalate on manual steps
If a step genuinely needs a human — OAuth sign-in, a quota/capacity increase, a
license/EULA click, a missing credential, a payment issue — **stop and tell the
user exactly what is blocked and what you need**. Do not silently retry or fake
progress.
## 9. Deliver & tear down
- Return the **access URL** (`https://<pod-id>-<port>.proxy.runpod.net`) and a
sample request.
- Note any security caveat (proxy URLs are public; most dev servers have no auth).
- Tell the user how to stop/terminate; data on the network volume persists. The
pod bills until someone stops or removes it.
## The loop in one line
**plan → provision (ports+env+volume+ssh) → connect (ssh-exec) → set up → run
(bind 0.0.0.0) → poll readiness → iterate → escalate if blocked → deliver + tear
down.**
reference/pods-and-machines.md
---
concepts: [machine, pod, pod-deployment, pod-migration, pod-volume-disk, network-volume, gpu-type]
---
# Pods and machines
A pod runs on one physical **machine** in a data center. Knowing that explains the most
confusing pod behavior: a stopped pod that cannot restart with a GPU even though the
data center shows plenty of stock.
## A pod is bound to its machine
- Runpod assigns the machine when the pod is **first deployed**. The pod stays on it while
it runs, while it is stopped, and when it starts again.
- **Stopping releases the pod's GPUs** on that machine, and another customer can rent them
while the pod is stopped.
- **Starting again only looks at that one machine.** Data-center availability describes free
GPUs across all of its machines, so high stock for your GPU type does not mean your pod's
machine has a free GPU.
- **Moving to another machine means a new pod** with a new pod id, either through migration
(offered in the console when a stopped pod cannot get its GPU back) or a fresh deploy. The
REST v2 pod actions are `start`, `stop`, `restart` and `terminate`; there is no migrate action.
## When a stopped pod will not start with a GPU
1. **Wait and retry** if the pod's machine may free up.
2. **Start it with zero GPUs** to reach the data on its volume disk, copy what you need off,
then deploy a new pod.
3. **Migrate it** from the console. Runpod creates a new pod with the same configuration on
a machine with free GPUs and moves the pod's data to it. The new pod has a new id, IP
address and proxy URL.
4. **Deploy a new pod** anywhere in the data center. This is only painless when the data you
need is on a **network volume**, which any pod in that data center can attach.
The habit that avoids the problem: keep data you need on a network volume, not only on the
pod's volume disk. See [`storage.md`](storage.md) for the storage layers.
## What else is per machine
| Property | Why it matters |
| --- | --- |
| **Volume disk** | Stored on the pod's machine. It survives a stop, is lost if the machine fails, and only that pod can use it. A network volume lives at the data-center level instead. |
| **CUDA version** | Set by the machine's driver, not the GPU type, so two machines with the same GPU can report different versions. A pod's `cudaVersion` shows what its machine reports. |
| **GPU count** | One pod runs on one machine, so the most GPUs a single pod can use is the GPU count of the largest machine of that type. `maxCount` on the GPU type reports it for each cloud. |
## Exact rules
The facts above come from the concept files, which cite their public sources:
[`machine`](../concepts/machine.yaml), [`pod-deployment`](../concepts/pod-deployment.yaml)
(deploy, stop, restart and recover as ordered steps), [`pod-migration`](../concepts/pod-migration.yaml)
and [`pod-volume-disk`](../concepts/pod-volume-disk.yaml).
reference/secrets.md
---
concepts: [runpod-secret, pod, serverless-endpoint, template]
---
# Secrets
A Runpod **secret** stores a sensitive value, such as an API token or S3 key, in your
account. Pods, endpoints and templates reference it by name, so the plain value never
appears in their configuration.
## Use a secret
1. **Create it** with a `name` and `value`. REST: `POST /v2/account/secrets`.
MCP: `create-secret`.
2. **Reference it** from an environment variable on a pod, endpoint or template by setting
the variable's value to `{{ RUNPOD_SECRET_<name> }}`.
3. Runpod substitutes the stored value when the pod or worker boots.
## Rules to know
- **The value is write-only.** No API response or console page shows it after it is set,
so keep your own copy if you will need it again.
- **Names are permanent and unique** in the account. A name is up to 191 characters,
starts with a letter or underscore, and uses letters, digits and `_.-/`. Names that start
with `runpod` in any letter case, such as `RUNPOD_TOKEN` or `RunpodKey`, are rejected.
- **Rotation takes effect at the next boot.** Running pods and workers keep the value they
started with, so restart them after you change it.
- **Update the value and the description separately** when it matters. A `PATCH` that sends
both applies the value first, so if the description update fails the new value is already
live.
- **Deleting is permanent and is not blocked while the secret is in use.** Variables that
reference it stop resolving, so update them first.
List secrets with `GET /v2/account/secrets` (MCP: `list-secrets`); each entry has an `id`,
`name` and timestamps but never the value. Change one with `PATCH /v2/account/secrets/{id}`
and delete it with `DELETE /v2/account/secrets/{id}`.
## Exact rules
These facts come from [`runpod-secret`](../concepts/runpod-secret.yaml), which cites the
REST v2 spec and the public docs.
reference/storage.md
---
concepts: [network-volume, global-volume, container-disk, pod-volume-disk, model-store, volume-cache, pod-deployment]
---
# Where data lives on Runpod
Pick storage based on whether data must survive a stop, be shared across
machines, and where your compute runs. There are four layers.
## Default: prefer a network volume
**Unless the user says otherwise, put anything worth keeping on a network
volume** — models, datasets, checkpoints, environments, caches. It survives pod
stop/terminate and serverless scale-to-zero, is reusable across pods and
endpoints, and means expensive downloads happen once. Read GPU stock per data center
first, create the volume once in a data center that has the GPU, then place the
compute in that same DC. If a Pod create is refused for stock, re-read stock:
when the volume's DC shows stock, retry there once with the same volume; otherwise the
user chooses between waiting, another GPU type there, the console's Deploy when
available, and a move to another DC. Never delete the volume and recreate it in the next DC.
Use the faster, non-persistent layers deliberately, not by default: container /
ephemeral disk for throwaway scratch, and pod volume disk when a single pod's data
doesn't need to outlive it. When in doubt, choose the network volume.
**One exception: model weights for GPU serverless.** If the model is on HuggingFace,
the host-side cache needs no volume, no download billing, and no
data-center pin narrowing your GPU availability. See
[Getting a model to the worker](#getting-a-model-to-the-worker) below; the default
above is about datasets, checkpoints, and everything else worth keeping.
## The four layers
### Container / ephemeral disk
- Exists only while the container runs; **wiped when the pod stops or the worker
scales down**. Local to the machine, so it's several times faster than a
network-backed `/workspace` and the place for heavy I/O.
- Everything a serverless handler writes goes here by default.
- On pods, editing/resetting a running pod also erases it — only `/workspace`
survives (see below).
- Use for: OS, package installs, builds, temp files, scratch, caches you don't need
to keep. Size `--container-disk-in-gb` for them.
### Volume disk (pods only)
- Persistent storage mounted at `/workspace`. Retained across stop/restart, but
**deleted when the pod is terminated**. Not shareable.
- Whether it's local depends on the data center. Where the data center offers
network volumes, expect `/workspace` to be network-backed and several times
slower than the container disk. Where it doesn't, `/workspace` may be local or
network-backed. Check from inside the pod with `findmnt /workspace`: `fuse`
means network-backed, `xfs` on a device means local. Either way, install
packages outside `/workspace` (the container disk is always local) rather
than picking a data center for disk speed.
- Roughly $0.10/GB/month running, $0.20/GB/month while stopped. Can be increased
(never decreased); optionally encrypted at rest.
- Use for: models, datasets, and checkpoints you reuse across sessions on one pod.
### Network volume (shared, portable)
- Persistent storage that lives **independently of any compute**. Attachable to
multiple pods and to serverless endpoints; survives termination/scale-to-zero.
- NVMe-backed (roughly 200–400 MB/s, higher peak). Standard and High-Performance
tiers. Pricing ~$0.07/GB/month for the first 1 TB, ~$0.05/GB/month beyond.
- **Picking the tier:** the runpod-mcp `create-network-volume` tool takes
`volumeType` (`STANDARD` | `HIGH_PERFORMANCE`); omit it for the data center's default.
`runpodctl` has no tier flag, so from the CLI request **High-Performance** via the
console (a ⚡ data center → toggle) or a raw v2 REST call (`POST
https://api.runpod.io/v2/network-volumes` with `"type":"HIGH_PERFORMANCE"`); the tier
is immutable after creation. High-Performance (~3× throughput / 4× IOPS) is worth it when
I/O is the bottleneck — training, checkpointing, many small files. Launch: golden path
[21 — storage tiers](../../runpod/golden-paths/21-storage-tiers.md); more:
[high-performance storage docs](https://docs.runpod.io/storage/high-performance-storage).
- **Data-center-scoped** — a volume lives in one DC. See the constraint below.
- Mount paths:
- Pods: mounts at `/workspace`, **replacing** the volume disk. Must be attached
at pod creation and cannot be detached later.
- Serverless: mounts at `/runpod-volume`.
- Use for: sharing models/datasets across workers or pods, and anything that must
outlive a single machine.
### Global volume (any data center)
A **global volume** is storage that mounts into workloads in **any** data center, so it has
no data-center constraint. It is usage-billed, so it has no size: create it with just a
`name` (REST v2 `POST https://api.runpod.io/v2/global-volumes`, or MCP
`create-global-volume`). Accounts hold a capped number; a create past the cap is `400`.
- **Pods:** at most one global volume per pod, on GPU pods and on CPU pods (CPU pods
through the API only), under `mounts.global`. It combines with a network volume but not
with a pod volume disk (`400`). A CPU pod created through REST v2 or MCP without a
`path` or a network volume mounts it at `/workspace-global` (for other cases the spec
names no default); the console's deploy flow mounts it at `/workspace` when no network
volume is attached.
Read `mounts.global[0].path` back rather than assuming.
The mount is fixed at create; only its `path` can change later.
- **Serverless:** at most one per endpoint, **GPU endpoints only** (a CPU endpoint gets
`400`), under `mounts.global`, seen by workers at `/runpod-volume`. An endpoint carries
network volumes or a global volume, not both.
- **Delete** removes the stored data and is refused with `409` while any pod (stopped pods
count) or endpoint mounts it.
- **Limits (beta):** object-backed, so no file locking, atomic rename, hard links or
permission bits. A template that installs a Python environment into `/workspace` (the
official ComfyUI template, for one) cannot use it as its primary volume; mount it at a
second path for model files. Changes made after a pod or worker starts may not be
visible inside it, and concurrent writes to one file are last-write-wins.
- **Billing:** stored data plus requests: Class A (writes, creates, lists) $0.005 and
Class B (reads, metadata lookups) $0.0005 per 1,000 requests. It is permanently deleted
15 days after the account balance reaches $0.
- Use for: model weights and other read-mostly files shared across data centers; keep
write-heavy work (training, checkpoints) on a network volume. For checkpoints a global
volume is the fallback when no data center that offers network storage has the GPU in
stock, not the default. Without tools that create or attach one, the console path is
Storage, create or open the global volume, then **Configure Pod with volume**
([global volumes for Pods](https://docs.runpod.io/storage/globalvolume/globalvolume-pods)).
Source: the REST v2 spec (`GlobalVolume`, `Mounts`, `EndpointMountsInput`,
`/v2/global-volumes`) and the public docs
([global volumes](https://docs.runpod.io/storage/globalvolume/overview)); live-verified 2026-10-29 (name-only create, CPU pod mount, `409`
delete while mounted, `400` on a CPU endpoint, `204` delete after the pod was gone).
The GPU-endpoint mount and `/runpod-volume` path were not live-run; the REST default
`/workspace-global` on a CPU pod was live-verified 2026-11-02.
### Data-center constraint (important)
A network volume is pinned to its data center, so any compute that mounts it must
be placed in that **same DC**. That narrows GPU availability. To improve
availability, attach multiple volumes from different DCs (one per DC) — but data
does **not** sync between them automatically; copy it yourself (S3 API /
runpodctl). **Do not write to one volume from multiple workers concurrently** — it can corrupt data.
For read-mostly files needed in several data centers, a
[global volume](#global-volume-any-data-center) avoids the pin, within its limits.
## Getting a model to the worker
Four ways to make a model available; choose by size, privacy, and how often it
changes. On HuggingFace and running GPU serverless → **cached model**; your own
artifact → **Model Repository** (or a volume if you want to manage the files
yourself); need a reproducible image → **bake**.
| Option | How | Best when |
| --- | --- | --- |
| Bake into image | `COPY` model files, or download during `docker build` | Small/private models not on Hugging Face; fully reproducible images |
| Cached model (HF) | Attach a Hugging Face model to the endpoint (`--model-reference`); Runpod caches it host-side | Public/gated/private HF models; fastest load once the worker runs, smaller images. Worker start waits for a host that holds the model, so time the whole cold request |
| Network volume | Pre-load the model onto a volume, mount it; in your own handler, load through `VolumeCache` so it reads local disk | Large shared models reused across many workers/pods |
| Model Repository | Upload your own artifacts with `runpodctl model add`; Runpod versions + distributes them | Your own/custom models not on HF, without image bloat or a DC-pinned volume |
Full comparison + commands: [`runpodctl/reference/model-caching.md`](../../runpodctl/reference/model-caching.md).
Trade-off: baking bloats the image and slows pulls; cached models and volumes
decouple the model from the image so it loads without re-downloading, cutting cold
starts. With cached models you are **not billed** for download time, and it works
for GPU serverless endpoints.
### HuggingFace cache directory
Runpod's cached-model feature stores models in the standard HF cache layout at:
```
/runpod-volume/huggingface-cache/hub/
```
Structure follows HF conventions — `models--{org}--{name}/snapshots/{hash}/`
(slashes in the model name become `--`), e.g.
`/runpod-volume/huggingface-cache/hub/models--Qwen--Qwen2.5-0.5B-Instruct/snapshots/<hash>/`.
Official workers can already integrate this location. Custom workers must configure
the loader's cache directory or pass the matching local snapshot; an HF-compatible
library alone does not establish discovery. See the public
[cached-model guide](https://docs.runpod.io/serverless/endpoints/model-caching) and
[custom-handler example](https://docs.runpod.io/serverless/development/huggingface-models).
For a standard HF cache, `HF_HOME` names the root and `HF_HUB_CACHE` names its `hub`
directory. A baked cache might use `HF_HOME=/opt/models` and
`HF_HUB_CACHE=/opt/models/hub`. A plain directory of model files is different: pass that
directory directly to a loader that accepts it. Match the layout instead of setting
`HF_HOME` to an arbitrary model directory. See
[HF environment variables](https://huggingface.co/docs/huggingface_hub/en/package_reference/environment_variables).
## Accessing a network volume over S3
Runpod exposes network volumes through its **own** S3-compatible API (not AWS).
The bucket name is the network volume ID. Every request needs both a region and
an endpoint URL derived from the volume's data center:
```
--region <DC> --endpoint-url https://s3api-<DC>.runpod.io/
```
Credentials (from Console → Settings → S3 API Keys):
- `AWS_ACCESS_KEY_ID` = your Runpod **user ID** (format `user_...`)
- `AWS_SECRET_ACCESS_KEY` = an **S3 API key** you generate (format `rps_...`, shown once)
```bash
aws s3 ls \
--region CA-2 \
--endpoint-url https://s3api-CA-2.runpod.io/ \
s3://NETWORK_VOLUME_ID/
```
Path mapping: `/workspace/my-folder/file.txt` on a pod ==
`s3://NETWORK_VOLUME_ID/my-folder/file.txt` over S3. (See the `runpod-companion-clis`
skill for full AWS CLI usage; each volume on the Storage page shows a pre-filled
`aws s3` command with the right region/endpoint.)
## Quick decision guide
- Throwaway scratch, package installs and builds → container disk (local; size it
with `--container-disk-in-gb`).
- Keep data between sessions on one pod → volume disk (`/workspace`).
- Share models/data across workers or pods, or persist past termination →
network volume — and put your compute in the volume's DC.
- Read-mostly models/files shared across data centers → global volume (no file locking
or permission bits).
- Public/gated HF model on serverless → cached model, not a bake.
- Big files that blow past API payload limits → network volume or external S3;
return references, not bytes.
SKILL.md
---
name: runpod-usage
description: >-
Explain Runpod architecture and design choices: Pods versus Serverless, GPU/VRAM,
image construction, storage, networking, and workload development. For creating
or improving a complete template project use runpod-build-template; execute
infrastructure with runpodctl/runpod-mcp and code-first work with runpod-flash.
metadata:
author: runpod
version: "2.0.0" # x-release-please-version
concepts: [pod, serverless-endpoint, network-volume, container-disk, pod-volume-disk, gpu-type, exposed-port, worker]
license: Apache-2.0
---
# Runpod usage (concepts)
Background knowledge for making the right choice before you act. This skill runs
nothing — once you know what to do, execute with **runpod-mcp**/**runpodctl**
(infra), **runpod-flash** (your own code), or **runpod-companion-clis** (models/images/data).
**This skill explains; the golden paths demonstrate.** When the question is really "how do I
do X" rather than "how does X work", the verified end-to-end example is the faster answer —
[runpod/golden-paths/README.md](../runpod/golden-paths/README.md). Read the concept here, then
follow the path.
To author files for a custom template, use
[runpod-build-template](../runpod-build-template/SKILL.md); this lane supplies the
shared concepts, not a competing packaging workflow.
Read the one reference file that matches the question:
| Question | Read |
| --- | --- |
| First-run setup / auth — get + set `RUNPOD_API_KEY`, SSH, companion creds | `reference/getting-started.md` |
| Pods vs serverless, workers, cold starts, FlashBoot, queue vs load-balanced | `reference/concepts.md` |
| **The development loop for ANY workload (start here)** — plan → prefer prebuilt → provision → verify → teardown | `reference/development-loop.md` |
| **Stand up / iterate a workload on a pod** — the pod sub-loop | `reference/pod-workflows.md` |
| **Deploy / iterate a serverless endpoint** — Hub vs flash vs custom, invoke + verify | `reference/endpoint-workflows.md` |
| **Install software on a pod** — package hygiene, `uv`, non-interactive, caching | `reference/on-pod-setup.md` |
| Build a Docker image Runpod can run (handler contract, Dockerfile, `--platform=linux/amd64`) | `reference/docker.md` |
| **How to build an image well** — base image, layering, bake-in vs volume, pod vs serverless (queue/LB) contract | `reference/building-images.md` |
| Where data lives — container disk vs network volume, model caching, S3 access | `reference/storage.md` |
| Which GPU / how much VRAM / **which CUDA version** / cost & availability / data centers | `reference/gpu-selection.md` |
| Why a stopped pod may not restart with a GPU — pods are bound to one machine; what else is per machine | `reference/pods-and-machines.md` |
| Endpoint releases, rolling updates, version overlap, GitHub builds | `reference/endpoint-builds-and-releases.md` |
| Storing API tokens and keys as secrets and referencing them from env vars | `reference/secrets.md` |
| Reaching a pod or endpoint over HTTP (proxy URLs, exposed ports) | `reference/networking.md` |
| Common mistakes and how to avoid them | `reference/gotchas.md` |
## Exact facts: the concept files
The reference files explain. For a precise fact, or a rule an action must satisfy, read
the concept files in `concepts/` next to this file. There is one YAML file per concept
(`pod.yaml`, `network-volume.yaml`, `machine.yaml`, …), each with its fields, states,
relations and rules, and every rule cites the public source it comes from.
```bash
grep -il "volume disk" concepts/*.yaml # find the concept by name or alias
grep -n "statement:" concepts/pod-volume-disk.yaml
```
- `rules[].statement` is the fact. `see:` points at related rules on other concepts.
- A rule with `conflict: true` records two public sources that disagree. Give both readings.
- A `kind: process` file explains a flow step by step. `pod-deployment.yaml` covers deploy,
stop and restart, including why a restart can fail while the data center has stock.