runpodctl

어떤 스킬인가요?

>-

설치 방법

프로젝트 폴더에서 아래 명령어를 실행하고, 설치할 에이전트를 선택하세요.

설치 명령을 직접 실행해야 적용됩니다. 지원 에이전트와 필요한 권한·라이선스는 제작자의 안내를 확인하세요.

스킬 원문 파일

# Create a CPU-only pod

## Prompt

Create a CPU-only pod for lightweight file preprocessing using the image
`ubuntu:22.04`. Give me the exact command.

## Expected behavior

The agent should:

1. Use `runpodctl pod create` with `--compute-type cpu`
2. Pass `--image ubuntu:22.04`
3. Explain that the default is 2 vCPUs and the name defaults to the image name when omitted
4. NOT pass GPU flags (`--gpu-id`, `--gpu-count`) — they don't belong on a CPU pod

## Assertions

- Runs `runpodctl pod create --compute-type cpu --image ubuntu:22.04`
- States the 2-vCPU default and image-name default
- Does NOT include `--gpu-id` or `--gpu-count`

# Create a CPU pod and wait until the container is up (no ssh)

## Prompt

Create a CPU pod from `ubuntu:22.04` for a quick preprocessing job and tell me when it's
up. I don't need to ssh into it.

## Expected behavior

The agent should:

1. Create with `runpodctl pod create --compute-type cpu --image ubuntu:22.04 --wait-for running`
   (runpodctl v2.15.0+), which returns once the container is reported up and needs no ssh.
2. NOT use plain `--wait` (ssh mode): `ubuntu:22.04` starts no sshd and cpu pods get no
   Runpod-managed ssh, so it would bill through the whole timeout and then fail.
3. NOT tell the user the pod is ssh-reachable — running mode proves nothing about ssh.
4. NOT write its own poll loop around `pod get`.
5. If the wait times out, recognise that the pod **was created and still bills**, take the id
   from the error object's `id` field, and offer to delete it.

The ssh scenario stays in `create-and-wait-until-usable.eval.md`: when the user needs ssh,
plain `--wait` remains correct, including for a cpu image that starts its own sshd.

## Assertions

- Uses `--wait-for running` (optionally with `--wait-timeout`), not plain `--wait` or a hand-rolled polling loop
- Does NOT include `--gpu-id` or `--gpu-count`
- Does not claim or imply the pod accepts ssh
- On `wait_timeout` / `wait_interrupted`, surfaces the pod id from the error object and proposes cleanup instead of assuming nothing was created

# Create a pod and wait until it is actually usable

## Prompt

Spin me up an A40 pod from `runpod/pytorch:1.0.3-cu1281-torch291-ubuntu2404` and
tell me when I can ssh into it.

## Expected behavior

The agent should:

1. Create with `runpodctl pod create --image ... --gpu-id "NVIDIA A40" --wait`, which returns
   only once port 22 answers with an ssh banner.
2. Read the printed payload (the `pod get` shape, so it carries the live `ssh` block) and give
   the user the ssh command.
3. NOT write its own poll loop around `pod get`/`ssh info`, and NOT treat
   `desiredStatus: RUNNING` as "ready" — that is true while the image is still pulling.
4. If the wait times out, recognise that the pod **was created and still bills**, take the id
   from the error object's `id` field, and offer to delete it.

## Assertions

- Uses `--wait` (optionally with `--wait-timeout`) rather than a hand-rolled polling loop
- Does not claim the pod is ready based on `desiredStatus` alone; if it inspects status, it reads `runtimeStatus`
- Reports the ssh connection details from the waited-for output
- On `wait_timeout` / `wait_interrupted`, surfaces the pod id from the error object and proposes cleanup instead of assuming nothing was created

# Handle a runpodctl failure without parsing message text

## Prompt

I'm scripting `runpodctl` in a loop. Write the failure handling: how do I tell a
missing endpoint from a bad API key from a blip I should retry? Here are two failures
I've seen:

```
{"error":"failed to get endpoint: endpoint not found","code":"not_found","status":404}
{"error":"failed to get template: template not found: tpl-abc","code":"not_found"}
```

## Expected behavior

The agent should:

1. Branch on `code`, not on `status` and not on substrings of `error`
2. Explain that the second failure has no `status` because GraphQL answers a missing
   resource with HTTP 200 + null data, so `status === 404` misses it
3. Retry `network_error`, `rate_limited` and `server_error` with backoff, and say why
   `cli_error` (e.g. a malformed `RUNPOD_API_URL`) must not be
4. Distinguish `no_credentials` (no key set → `RUNPOD_API_KEY` / `runpodctl doctor`) from
   `unauthorized` (key present but wrong/expired) — neither is a retry
5. Treat `not_found` as server-side, not as a mistyped local path
6. Gate on a non-zero exit code and read errors from **stderr**, data from stdout — and
   not treat a `warning:`/`note:` line on stderr as a failure
7. Include a default branch for an unrecognized `code`, because the vocabulary is what the
   CLI generates rather than exhaustive (the API can pass its own code through lowercased)
8. Not retry `timeout` / `wait_timeout` / `wait_interrupted` / `job_failed` (v2.9.0+): each
   means work or a resource outlived the CLI, so re-running the command buys a **second**
   job or a second billed resource. Follow up instead — poll (`serverless status`, `pod get`)
   or clean up via the error object's `id`
9. Treat `transfer_incomplete` (v3.1.0+, from `network-volume upload`/`download`) as the one
   code where running the **same command** again is right: the transfer already retried
   internally, and the rerun resumes it. `volume_full` needs the volume grown first, then
   the same rerun

## Assertions

- Switches/branches on `code`
- Does NOT branch on `status` alone or on `error` text matching
- Retry set includes `network_error` and at least one of `rate_limited`/`server_error`
- The produced handler has a default/else branch treating an unknown `code` as fatal
- Does NOT retry `cli_error`, `usage_error`, `not_found`, `no_credentials`, or
  `unauthorized`
- Does NOT retry `timeout`, `wait_timeout`, `wait_interrupted` or `job_failed`; treats them
  as "poll or clean up", not as "run it again"
- If it covers `network-volume upload`/`download`, reruns the same command on
  `transfer_incomplete` and does NOT treat it as fatal; for `volume_full`, grows the volume
  (`network-volume update --size`) before the rerun
- Separates `no_credentials` from `unauthorized` in the auth handling
- Reads errors from stderr and does NOT expect error JSON on stdout
- Does NOT claim `status` is always present
- Does NOT tell the user to parse plaintext for `pod`/`serverless`/`template`/`model`
  commands (on v2.10.0+ only the legacy `pod` paths and `project` still print plaintext —
  `exec` joined the coded-JSON shape and now exits non-zero, so an assertion that names
  `exec` as a plaintext surface is stale)
- Does NOT pass `--output` anything but `json`/`yaml`: on v2.10.0+ an unrecognized value is
  a `usage_error` with exit 1, where v2.9.0 silently returned JSON

# Install runpodctl over verified HTTPS

## Prompt

Install runpodctl on my Linux machine using the official install script.

## Expected behavior

The agent uses `curl -fsSL https://cli.runpod.net | sudo bash`. It keeps
certificate verification enabled and surfaces download errors without retrying
over HTTP.

## Assertions

- The initial URL explicitly starts with `https://`.
- The command keeps certificate verification enabled.
- The agent treats transport errors as installation failures and does not claim
  success from the pipeline's exit status alone.
- The agent does not claim an in-script check authenticates the script's download.

# Deploy a vLLM serverless worker from the Runpod Hub

## Prompt

Deploy the vLLM serverless worker from the Runpod Hub. Use an available GPU, and
have it scale from 0 up to 2 workers. Give me the exact command(s).

## Expected behavior

The agent should:

1. Find the hub listing id with `runpodctl hub search vllm`
2. Create the endpoint with `runpodctl serverless create --hub-id <id> --workers-min 0 --workers-max 2`
3. Handle the GPU correctly (this is the easy thing to get wrong):
   - Preferably omit `--gpu-id` and let the hub config's default GPU apply, OR
   - Pass a GPU name from `runpodctl gpu list` (e.g. `"NVIDIA GeForce RTX 4090"`); runpodctl ≥ v2.14.0 maps it to the matching pool (`ADA_24`). On older binaries, pass the pool id itself.

## Assertions

- Finds the hub id via `runpodctl hub search vllm` (does not invent one)
- Runs `runpodctl serverless create --hub-id <id> ...`
- Sets `--workers-min 0` and `--workers-max 2`
- Sets the execution timeout on the create (`--execution-timeout <seconds>`), or reads it back with `serverless get` and fixes a 0
- If `--gpu-id` is passed, its value is a real GPU name from `runpodctl gpu list` or a pool id (e.g. `ADA_24`), not an invented value
- Checks `runpodctl version` (or `--help`) before relying on GPU-name mapping on the hub path

## Notes

runpod/runpodctl#287 tracked `serverless create --gpu-id` on the `--hub-id` path rejecting
GPU display names with `Invalid GPU Pool ID`. Verified live with runpodctl v2.14.0
(2026-09-28): `--gpu-id "NVIDIA GeForce RTX 4090"` on the hub path created the endpoint
with `gpu.pools: ["ADA_24"]`. Older binaries may still need the pool id.

# Run a custom image as a serverless endpoint (template first)

## Prompt

I have my own custom Docker image `myrepo/infer:latest`. I want to run it as a
Runpod serverless endpoint that scales from 0 to 3 workers on an A40 GPU. Give me
the exact commands.

## Expected behavior

The agent should:

1. Recognize that serverless endpoints are created from a `--template-id` or `--hub-id`, NOT directly from a raw image
2. First create a serverless template: `runpodctl template create --name ... --image myrepo/infer:latest --serverless`
3. Then create the endpoint from that template id: `runpodctl serverless create --template-id <id> --workers-min 0 --workers-max 3 ...`
4. Order the two steps correctly (template before endpoint)

## Assertions

- Step 1 creates a template with `runpodctl template create --image myrepo/infer:latest --serverless`
- Step 2 creates the endpoint with `runpodctl serverless create --template-id <id-from-step-1>`
- Sets `--workers-min 0` and `--workers-max 3`
- Does NOT attempt `runpodctl serverless create --image ...` (no `--image` flag exists on serverless create)

## Notes

The template-id path is more lenient about `--gpu-id` than the hub path (it accepts
display names like `"NVIDIA A40"`), but see hub-deploy-serverless.eval.md and
runpod/runpodctl#287 for the pool-id inconsistency.

# Read invoke URLs and GPU cost from command output

## Prompt

I'm on a shell-only box with `runpodctl` and no MCP tools. I want a serverless endpoint
from template `tpl-abc` on the cheapest GPU with at least 24 GB that can actually schedule
in `US-KS-2`, and then the URL to send a job to. Give me the exact commands and tell me
where each number comes from — don't create anything yet.

## Expected behavior

The agent should:

1. Run `runpodctl gpu list --include-unavailable` and filter on `memoryInGb >= 24`,
   noting that the default listing hides no-stock GPUs and could omit one that has stock
   only in `US-KS-2`
2. Compare `securePricePerHr` / `communityPricePerHr` rather than assuming a lower tier is
   cheaper, and treat a `null` price as "not offered on that cloud", not as free
3. Check `dataCenterAvailability[]` for a `US-KS-2` entry with real stock instead of
   trusting top-level `stockStatus` (the best status across all DCs), reading `"none"` as
   "offered here, no stock"
4. Pin placement with `--data-center-ids US-KS-2` — reading per-DC availability without
   constraining the create leaves the placement to chance
5. Say that `--gpu-id` on serverless resolves to a GPU **pool**, so the endpoint may run on
   more than the single card that was priced, and/or that the listed prices are pod
   on-demand rates while serverless bills per request-second
6. State that the run URL comes from the `urls` object in the create output (`run`,
   `runsync`, `health`) rather than hand-assembling `https://api.runpod.ai/v2/<id>/run`

## Assertions

- Runs `runpodctl gpu list` before choosing a GPU
- Cites `securePricePerHr` / `communityPricePerHr` for the cost comparison
- Consults `dataCenterAvailability[]` for `US-KS-2`
- Command includes `--data-center-ids US-KS-2`
- Says the run URL will be read from the `urls` object rather than assembled by hand
- Flags at least one of: the type→pool translation, or that the prices are pod rates
- Does NOT claim the CLI can't report GPU pricing
- Does NOT present `securePricePerHr` as the endpoint's serverless cost

# Cache vs network volume is a latency call, and the miss path looks like an idle queue

## Prompt

I'm deploying a 70B model from HuggingFace to a serverless endpoint and I'll be scaling
it across a few regions. Should I use the model cache or a network volume? Also, is the
cache available in every region?

## Expected behavior

The agent should:

1. Frame the choice as **latency**, not cost — download time is not billed on a cache
   hit or a miss, so the cache doesn't save money over a volume, it saves setup
2. Describe the miss path correctly: Runpod holds worker start until the model is
   downloaded, so the **job sits in the queue** and the endpoint looks idle or stuck
   when it is actually downloading
3. Lean toward a **pre-loaded network volume** for a latency-sensitive endpoint at this
   size, while saying the cache is the reasonable default when latency is not critical
4. Recommend **measuring both** on the user's own model and region rather than asserting
   a threshold — cache capacity is a share of each machine's disk, so it varies by host
   and there is no number to quote; a benchmark from elsewhere doesn't transfer
5. Note that a network volume is **pinned to one data center**, so multi-region means one
   pre-loaded volume per DC
6. On the region question: say **Runpod publishes no list** of model-cache-enabled data
   centers, and offer the network-volume-capable DC list only as a labeled proxy, or
   offer to deploy and measure

## Assertions

- Says download time is NOT billed, on hit or miss
- Describes the miss as a queued job / delayed worker start, not a billing leak
- Mentions the volume's single-data-center pinning for the multi-region case
- Suggests testing both rather than quoting a hard size cutoff
- Says capacity varies by host rather than implying a single platform-wide cache size
- Treats inconsistent cold starts across workers as expected, not as a broken endpoint
- Does NOT claim the worker bills while downloading
- Does NOT state a list of regions that support the model cache as fact
- Does NOT claim `--model-reference` makes cold starts fast unconditionally

# Create a pod that auto-terminates at a datetime

## Prompt

Create a GPU pod from the Docker image `myorg/trainer:latest` that automatically
terminates itself at 2026-07-01T00:00:00Z. Give me the exact command.

## Expected behavior

The agent should:

1. Use `runpodctl pod create` with `--image myorg/trainer:latest`
2. Not use `--terminate-after` or `--stop-after`. runpodctl before v2.12.0 accepts them, but the backend never enforced them, and v2.12.0 removed them
3. Tell the user the pod will not delete itself at that time, and offer a working alternative: `runpodctl pod remove <pod-id>` when the work is done, or a stop scheduled from inside the pod
4. Tell a user on runpodctl older than v2.12.0 to run `runpodctl update`

## Assertions

- Runs `runpodctl pod create --image myorg/trainer:latest ...`
- Does NOT use `--terminate-after` or `--stop-after`
- States that the flags never enforced the deadline and were removed in v2.12.0
- Gives `runpodctl pod remove <pod-id>` or an in-pod scheduled stop as the alternative

# Create a pod from a template with a network volume

## Prompt

Create a GPU pod named `trainer` from template id `tmpl-1`, and attach the network
volume with id `nv-9`. Give me the exact command.

## Expected behavior

The agent should:

1. Use `runpodctl pod create` with `--template-id tmpl-1`
2. Set `--name trainer`
3. Attach the volume with `--network-volume-id nv-9`
4. Not need any extra GPU flag, since GPU is the default compute type

## Assertions

- Runs `runpodctl pod create ...`
- Uses `--template-id tmpl-1`
- Uses `--name trainer`
- Uses `--network-volume-id nv-9`

# Get SSH connection details for a running pod

## Prompt

I have a running pod with id `pod-xyz`. I want to SSH into it to debug a process
interactively. What runpodctl command(s) should I use to get connected?

## Expected behavior

The agent should:

1. Retrieve connection details with `runpodctl ssh info pod-xyz` (or `runpodctl pod get pod-xyz`)
2. Connect using the SSH command/key those return
3. NOT use any deprecated interactive SSH subcommand to open the session

## Assertions

- Uses `runpodctl ssh info pod-xyz` or `runpodctl pod get pod-xyz` to obtain host/port/key
- Does NOT rely on a deprecated interactive `runpodctl ssh`/`exec` session command
- Final guidance results in a usable `ssh ...` connection to the pod

# Use Pod lifecycle commands supported by API v2

## Prompt

Update a Pod's ports and environment, restart it, explain what `pod reset` does, and show the supported Pod lifecycle commands.

## Expected behavior

The agent should:

1. Use `runpodctl pod update <pod-id>` to change ports or environment on an existing Pod
2. Use `runpodctl pod restart <pod-id>` only when the user asks for a restart
3. Explain that runpodctl v3.0.0 removed `pod reset`
4. Never silently substitute restart for reset
5. Show start, stop, restart, update, and delete as supported lifecycle commands

## Assertions

- Uses `runpodctl pod update <pod-id>` for existing Pod ports or environment changes
- Uses `runpodctl pod restart <pod-id>` only for a restart request
- Explicitly reports that `pod reset` was removed in runpodctl v3.0.0
- Does NOT describe reset as an alias for restart
- Lists start, stop, restart, update, and delete; omits reset

# Default to Secure Cloud, treat Community as an explicit opt-in

## Prompt

I need a pod with an A40 to fine-tune a model overnight, and I want it as cheap as
possible. Give me the create command.

## Expected behavior

The agent should:

1. Recommend **Secure Cloud** even though the ask is "as cheap as possible" — Secure
   Cloud hosts are data-center-grade providers and tend toward more stable behavior,
   which is what an overnight run wants
2. Leave `--cloud-type` off the command (or pass `SECURE` explicitly) rather than
   reaching for `--cloud-type COMMUNITY` to hit the lower price
3. Describe Community Cloud as approved vendors rather than inventing specific failure
   behavior for it
4. Read the actual rates from `runpodctl gpu list` (`securePricePerHr` /
   `communityPricePerHr`) rather than asserting a gap from memory
5. Leave the choice with the user — present Community as available, not as the
   recommendation

## Assertions

- Recommends Secure Cloud for the overnight run
- The create command does NOT contain `--cloud-type COMMUNITY`
- Frames the difference as host-provider grade (data-center-grade vs approved vendors),
  not as a list of things that go wrong on Community
- Cites `runpodctl gpu list` for any price comparison it makes
- Does NOT silently pick Community just because the user said "cheap"

# Route project requests to the Runpod SDK

## Prompt

I have a source build containing the unreleased runpod/runpodctl#382.
I want to develop and deploy a serverless app with `runpodctl project new`,
`project start`, `project build` and `project deploy`. My script expects JSON
errors, but `runpodctl project deploy --help` exits 1 with empty stdout and this
stderr:

```text
The "project" command has been removed.

Use the Runpod SDK for serverless development workflows:
  https://github.com/runpod/runpod-python
```

Should I retry, fix my API key, or try `project create` or `project dev`?

## Expected behavior

The agent should treat exit 1 as a fatal failure, surface the notice verbatim,
and route serverless development to the Runpod SDK at
https://github.com/runpod/runpod-python. It should explain that the hidden `project`
catch-all returns the same plain-text notice for project subcommands, aliases and
flags. Supported help forms include `project --help`, `project deploy --help`,
`help project`, `help project deploy`, `-h project`, `--help project` and
`--help project deploy`. It should scope this contract to builds containing #382,
not released versions through v3.0.1.

## Assertions

- Uses the Runpod SDK for serverless development.
- Treats exit 1 as failure, with empty stdout and the exact notice on stderr,
  including its final newline.
- Handles this as a plain-text exception without requiring a JSON `code`.
- Does not retry, suggest credential changes, or treat the listed project help
  forms as success; does not generalize this contract to every help arrangement.
- Does not recommend `project new`, `start`, `build`, `deploy`, `create` or
  `dev` as working development commands.
- Explains that unknown subcommands and flags also reach the same notice.
- Keeps `runpodctl serverless` available for endpoint management.

## Released-version compatibility prompt

I use the published runpodctl v3.0.1. A `project` command prints an error on stdout
but exits 0. Does that mean it succeeded? Should I use this version for serverless
development? Does its project SSH client check host keys? What about v2.14.0?

## Released-version assertions

- Recommends the Runpod SDK for serverless development on every CLI version.
- Explains that released versions through v3.0.1 print `project` runtime errors to
  stdout and exit 0, so the output must be checked even when the exit status is zero.
- Does not apply the fatal stderr notice contract to these released versions or
  invent a release version for #382.
- Checks `runpodctl version` and, for source or custom builds, verifies the build's
  commit before assuming the #382 contract, since their version string can match
  a released version.
- Explains that v2.15.0 through v3.0.1 support legacy `project` SSH with host key
  pinning, while v2.14.0 and earlier disable host key checking on the `project`
  and `exec python` SSH paths.
- Presents legacy behavior as compatibility and security caveats, not a reason to
  use `project` or downgrade the CLI.

# Create a registry auth without exposing its token

## Prompt

I have a private registry token in `REGISTRY_TOKEN`. Create a Runpod registry auth
named `ghcr` for username `octocat`, but do not put the token in command arguments
or shell history.

## Expected behavior

The agent should:

1. Use `runpodctl registry create` with `--password-stdin`
2. Pipe the environment variable through stdin without expanding its value into the
   runpodctl argument list
3. Keep `--name ghcr` and `--username octocat` on the command
4. Avoid `--password`, because that exposes the credential through `argv`
5. Recognize that omitting both password flags is appropriate only for a human at an
   interactive no-echo prompt, not for an automated command

## Assertions

- Uses `runpodctl registry create --name ghcr --username octocat --password-stdin`
- Pipes `REGISTRY_TOKEN` into stdin, for example with `printenv REGISTRY_TOKEN`
- Does NOT use `--password "$REGISTRY_TOKEN"` or place the token value in any argument
- Does NOT tell an automated caller to rely on the interactive prompt
- Does NOT write the token to a temporary file

# Update a serverless endpoint to autoscale by pending requests

## Prompt

Update my serverless endpoint `ep-abc123` so it autoscales based on the number of
pending requests, triggering when there are 4 pending. Give me the exact command.

## Expected behavior

The agent should:

1. Identify that `runpodctl serverless update <endpoint-id>` is the right command
2. Use the v2.3 autoscaler flags `--scale-by requests` and `--scale-threshold 4`
3. NOT use the older `--scaler-type` / `--scaler-value` flags (removed in v2.3) or values like `REQUEST_COUNT` / `QUEUE_DELAY`

## Assertions

- Runs `runpodctl serverless update ep-abc123 ...`
- Sets `--scale-by requests` (strategy = pending request count)
- Sets `--scale-threshold 4`
- Does NOT use `--scaler-type`, `--scaler-value`, `REQUEST_COUNT`, or `QUEUE_DELAY`

# Invoke a serverless endpoint and get the result

## Prompt

I have a serverless endpoint `ep-abc123` running a vLLM worker. Send it
`{"prompt": "why is the sky blue?"}` and show me what it returns.

## Expected behavior

The agent should:

1. Invoke with `runpodctl serverless run ep-abc123 --input '{"prompt":"why is the sky blue?"}'`,
   which submits the job and polls until it is terminal.
2. Pass only the **handler** payload — the cli wraps it as `{"input": ...}` itself.
3. Read the job payload off stdout (it is printed even when the job fails) and report the
   worker's output or its `error`.
4. If the wait budget runs out with a `timeout` code whose message names a
   `serverless status` command, poll that command rather than re-invoking.

## Assertions

- Uses `runpodctl serverless run ep-abc123` with `--input` (or `--input-file`)
- Does NOT hand-build a `curl` to `https://api.runpod.ai/v2/ep-abc123/run` or `/runsync`, and does NOT construct an `Authorization: Bearer` header by hand
- Does NOT double-wrap the payload as `{"input":{"prompt":...}}`
- On a `timeout`, polls with `runpodctl serverless status ep-abc123 <job-id>` instead of re-submitting the job
- Treats a `job_failed` exit as the worker's failure (payload on stdout), not as a cli/auth problem

# Read stock before creating a training volume, and keep it on a refusal

## Prompt

Set up a training pod on an RTX 4090 with a 100 GB network volume for checkpoints, and
give me the commands. Make it robust to the GPU being unavailable.

## Expected behavior

The agent should:

1. Read per-data-center GPU stock before creating anything, because the volume pins
   the pod to its data center.
2. Create the volume once, in a data center the read shows with RTX 4090 stock, then
   create the pod there with the volume attached.
3. Handle a stock refusal without touching the volume: re-read stock; if the volume's
   data center shows stock, retry there once with the same volume; otherwise ask the
   user to choose between waiting, another GPU type in stock there, the console's Deploy
   when available, and a move to another data center.
4. If the user chooses to move, name the first volume (id, data center, storage cost)
   and delete it only when the user asks.

## Assertions

- Runs `runpodctl datacenter list` or `runpodctl gpu list --include-unavailable` before
  `runpodctl network-volume create`.
- `network-volume create` uses a `--data-center-id` the read showed in stock, and
  `pod create` passes `--data-center-ids` with that same data center plus
  `--network-volume-id`.
- Creates exactly one volume. No retry or fallback path runs `network-volume delete`.
- Does NOT write a loop that walks data centers creating a volume per attempt.
- Retries the refused `pod create` at most once, and only after a fresh read shows stock
  in the volume's data center.
- Presents moving to another data center as the user's decision, after a fresh stock
  read, not as an automatic fallback, and names the first volume when the user moves.

---
concepts: [runpodctl, registry-credential]
---

# runpodctl — behavior reference

**This file does not list flags.** `runpodctl <resource> <action> --help` does, it is always
current, and it ships with the binary in front of you:

```bash
runpodctl --help                          # resources
runpodctl <resource> --help               # actions + aliases
runpodctl <resource> <action> --help      # exact flags, defaults, examples
runpodctl help <resource> <action>        # same, and traverses aliases (`pod remove`) that --help does not
runpodctl version                         # which surface you actually have
```

What lives here instead is the part `--help` never tells you: **what a flag means when it
succeeds, what "ready" is defined as, which field to trust, and what a failure looks like.**
Output shapes, error codes and env vars are in
[output-and-errors.md](output-and-errors.md).

> Two `--help` gotchas: cobra does **not** traverse aliases for `--help`, so `pod remove --help`
> answers `unknown command` even though `pod remove <id>` works — use `runpodctl help pod remove`.
> And the deprecated `get pod` / `get cloud` paths are hidden from `--help` but still live.

## Pods

### Waiting for readiness (`--wait`, v2.9.0+)

| | detail |
| --- | --- |
| mode | `--wait-for ssh` (default) waits as described here; `--wait-for running` (v2.15.0+) waits only until the pod reports its container is up (desiredStatus RUNNING + runtime reported), needs no ssh or public port, and is the correct mode when container-up is enough or the image runs no sshd (typical for cpu pods). It proves nothing about ssh, so keep ssh mode when the task needs ssh — including a cpu image that starts its own sshd. Setting `--wait-for` implies `--wait` |
| ready means | **in ssh mode:** the pod's **public port 22** accepts a tcp connection *and* answers with an ssh protocol banner. No key, no handshake — it proves sshd is up, not that your key is installed. Port 22 merely appearing in `runtime.ports` is not enough: prod allocates that port even for images that run no sshd. **In running mode:** the container is reported up, nothing about ssh |
| timeout | `--wait-timeout` accepts `90s`, `10m`, `1h`, `2d`; default `10m` |
| output | progress on **stderr** every ~15s; stdout stays exactly one json object, in the `pod get` shape (so it includes the live `ssh` block, unlike a plain create) |
| on failure | the pod is **not** deleted — exit is non-zero, code `wait_timeout` (or `wait_interrupted` on ctrl-c), and the error object carries the pod id in `id` plus the delete command. A second ctrl-c always exits |
| refuses | `--ssh=false` **in ssh mode** (there would be nothing to wait for); it is allowed with `--wait-for running`, which does not need ssh |
| warns, still waits | **in ssh mode:** `--cloud-type COMMUNITY` (community cloud only maps a public ssh port on a machine that has a public ip, which the CLI cannot request) |

### Pod status fields

`pod get` and `pod list` report both (v2.9.0+):

| field | meaning |
| --- | --- |
| `desiredStatus` | what the platform intends: `RUNNING`, `EXITED`. Says `RUNNING` while the image is still pulling |
| `runtimeStatus` | what is actually happening: `running`, `initializing` (no container reported yet — pull/create/boot), `stopped`, `terminated`, `unknown` (the runtime lookup failed or was not made — **not** "the pod is down") |
| `runtimeStatusReason` | stable token when there is more to say, e.g. `awaiting_container`, `stopped_by_user`, `stopped_by_runpod`, `terminated_outbid`, `runtime_unavailable` |
| `uptimeSeconds` | present only while the container is up; omitted otherwise (it used to be a constant `0`) |
| `lastStatusChange` | the backend's raw free-text note, carried so a phrasing the cli does not tokenise still reaches you |
| `networkVolumeId` / `networkVolume` | **v2.10.0+:** `pod get --include-network-volume` fills both — the id and the full volume object. On **v2.9.0 and earlier both were dropped on deserialization** and read back `null` even with the flag, so a pre-v2.10.0 binary cannot tell you whether a pod has a volume. Upstream reports the `networkVolumeId` also comes back *without* the flag now (it is free alongside the pod), but only the flagged path is covered by a test — pass `--include-network-volume` rather than relying on that |

`--status` filters **`desiredStatus` only** — `--status initializing` silently matches nothing.

### Reading pod logs (`pod logs`, v2.10.0+)

| | detail |
| --- | --- |
| output | **json lines**, one `{source,line,ts}` object per line — pipe straight to `jq`, no SSE frame parsing |
| `--source` | `container` (your workload's stdout/stderr), `system` (the platform narrating image pull, container create, start), or `both` (default) |
| termination | without `--follow` it replays history and **exits on its own** once lines stop arriving (`--max-wait`, default `5s`, bounds the wait). With `--follow` it streams until interrupted and reconnects itself if the connection drops |
| history | `--tail 0-5000` (default 100; `0` = live only), or `--since 30m|2h|7d|<rfc3339>`, which overrides `--tail` |
| what to read it for | a stalled deploy is a `system` story — repeated pull progress, or a `create container` that never reaches `start` |

## Serverless (alias: sls)

### Invoking an endpoint (`serverless run`, v2.9.0+)

| | detail |
| --- | --- |
| payload | the **handler** payload, sent as `{"input": <your json>}`. Must be a json object; parsed and size-checked locally (the api's `/run` body limit is 10 MiB), so quoting mistakes and oversized bodies fail as `usage_error` before the upload |
| `--input` vs `--input-file` | mutually exclusive; one is required. `-` reads stdin either way. A payload with its own top-level `input` key gets a warning — that is usually a whole curl envelope pasted in, which arrives double-wrapped |
| stdout | always the job payload, including a `FAILED` job's `error`, and the last payload seen when the wait ran out. Printed byte-faithfully (handler keys are not renamed or re-typed) |
| stderr | progress notes and the error object — never job data |
| exit codes | `0` when `COMPLETED`, and when `--no-wait`/`--wait 0` submitted successfully. `1` on request failure, on wait-budget exhaustion (`timeout`), or on `FAILED`/`CANCELLED`/`TIMED_OUT` (`job_failed`) |
| two budgets | `--wait` bounds the whole job; the shared `timeout` config key (30s) bounds one api call. A call inside a wait is clamped to what is left, never below 1s |
| `/run`, never `/runsync` | `/runsync` is not synchronous: the connection is released after ~90s with the job still running, no job id exists until it answers (so a slow response strands a billed, unpollable job), and a `sync-` job's result expires after 1 minute vs 30 for `/run` |

`serverless status <endpoint-id> <job-id>` polls a job submitted earlier; `serverless health
<endpoint-id>` returns worker + job counts.

### Reading worker logs (`serverless logs`, v2.10.0+)

Same flags and semantics as `pod logs` above, plus:

| | detail |
| --- | --- |
| output | `{source,line,ts,workerId}` — the extra `workerId` is what makes the no-`--worker` form usable |
| `--worker` | optional. **Omit it and the command resolves the endpoint's workers itself** and reads all of them at once, tagging each line. With `--follow` it also picks up workers that appear mid-run, so an endpoint scaling up does not need the command re-run |
| what to read it for | the crash loop: repeated `system` `start container` lines with **no** `container` output means the container exits before the handler runs, which leaves jobs sitting in the queue with nothing wrong with capacity |

### Endpoint update and zero values

`serverless update` sends **zero values** as of **v2.10.0** (`--workers-min 0`,
`--idle-timeout 0`, `--workers-max 0`, `--scaler-value 0`). On **v2.9.0 and earlier those were
silently dropped** from the request, so resetting a dev endpoint back to scale-to-zero looked
like it applied and the endpoint kept billing. On an older binary verify with `serverless get
<id>`, or `PATCH https://api.runpod.io/v2/serverless/<id>` with an explicit `{"workers":{"min":0}}`.

`serverless update` has **no `--gpu-id` flag** — change an existing endpoint's GPU pool with
that same `PATCH` and `{"gpu":{"pools":[...]}}`.

**Multi-DC** (`--network-volume-ids <v1>,<v2> --data-center-ids <dc1>,<dc2>`) needs
**runpodctl ≥ v2.4.0**; data does not sync between volumes automatically — golden path
[10](../../runpod/golden-paths/10-multi-region-ha-serverless.md).

`serverless update --model-reference` includes the server-resolved `modelReferences` in
JSON/YAML output; `--clear-models` returns `modelReferences: []`. Other updates do not fetch
model references; use `serverless model-status <id>` to inspect them.

⚠️ `serverless get --include-workers` returns raw v1 worker records **including
`RUNPOD_AI_API_KEY`/`RUNPOD_ENDPOINT_SECRET` in `env`** — don't paste its output into a ticket.

### CPU endpoints and REST v1

Verified evidence for why REST v1 must not be used for a CPU endpoint (the create
instruction is in [SKILL.md](../SKILL.md)): 2026-07-14,
`POST https://rest.runpod.io/v1/endpoints` with `"computeType":"CPU"` silently returned a GPU
endpoint (`gpuCount:1`, `cpuFlavorIds:null`), while `runpodctl --compute-type CPU` correctly
returned `computeType:"CPU"` with `instanceIds:["cpu3g-4-16"]`. The separate
**runtime/invoke** API `https://api.runpod.ai/v2/<endpoint-id>/…`
(health/run/runsync/openai) is a different v2 and works fine; the v1-vs-v2 caveat is only
about the **control/management** REST.

### Error codes worth branching on

`timeout`, `job_failed`, `wait_timeout` and `wait_interrupted` are the codes these commands
add, and `not_found` gains a nuance during a wait. They live with every other code in
[output-and-errors.md](output-and-errors.md#codes) — including which are safe to retry, which
mean work outlived the cli, and the `id` field that names a resource a failed wait left
behind.

## Models

`model add` supports upload sessions, versioning, metadata, and private-source credentials.
Concepts: [model-caching.md](model-caching.md); flags: `runpodctl model add --help`.

## Registry credentials

Prefer `registry create --password-stdin` (needs **v2.11.0+**) for scripts so the credential does not enter the
process table or shell history. A secret already held in an environment variable can be piped
without expanding its value into the runpodctl argument list:

```bash
printenv REGISTRY_TOKEN | runpodctl registry create --name "x" --username "u" --password-stdin
```

Redirecting a credential file to `--password-stdin` also works, including multi-line
credentials such as a GCR service-account JSON key. The command strips one trailing line
ending but preserves leading, inner, and other trailing whitespace.

`--password` remains supported for compatibility, but places the credential in `argv`.
When neither password flag is present and stdin is a terminal, runpodctl prompts without
echo. With non-terminal stdin and no password source it returns `usage_error` instead of
blocking; `--password` and `--password-stdin` are mutually exclusive.

## SSH

`ssh info <pod-id>` returns **connection details, not an interactive session** — and it has
three output shapes, only one of which is an error. That table is in
[output-and-errors.md](output-and-errors.md#parsing-ssh-info); read it before writing a
readiness loop. If interactive SSH isn't available, execute remotely via
`ssh user@host "command"`.

`ssh remove-key` takes `--name` **or** `--fingerprint`; use the fingerprint to disambiguate
keys that share a name.

### Pod host key trust (v2.15.0+)

In **v2.14.0 and earlier**, the deprecated `project` and `exec python` SSH paths
run with **host key checking disabled**. Check `runpodctl version` before relying
on host key pinning.

Pinning applies to the **hidden, deprecated** `exec python` command's Go SSH client
in **v2.15.0+**, and to legacy `project` SSH in **v2.15.0 through v3.0.1**.
In builds containing the unreleased [runpod/runpodctl#382](https://github.com/runpod/runpodctl/pull/382),
`project` only emits the [SDK notice](output-and-errors.md#project-runpod-sdk-notice)
and opens no SSH connection. Use the [Runpod SDK](https://github.com/runpod/runpod-python)
for serverless development regardless of CLI version; these are compatibility and
security caveats, not a recommendation to use `project`.

`ssh info` prints an ordinary
`ssh -i <key> root@<ip> -p <port>` and leaves verification to OpenSSH's defaults against your
own `~/.ssh/known_hosts`.

The Go client pins a pod's host key in `~/.runpod/ssh/known_hosts`, keyed by `runpod-<podID>`
rather than by address, because Runpod recycles pod SSH addresses. The Go client enrolls the
key on first contact, printing the fingerprint to **stderr**, and refuses a key that does not
match an existing entry. The `rsync` invocation always runs after that, so it **requires** the
pin rather than creating one: `StrictHostKeyChecking=yes` with `UpdateHostKeys=no`, so it can
neither enroll a key itself nor let the server add further keys to the entry.

A mismatch means either the pod's container was recreated and so generated a new host key, or
the connection is being intercepted. A recreated container is the common case: stopping and
starting a pod clears its container disk, as do `pod update` and a reset, and Runpod's images
generate fresh host keys at boot. The pod id stays the same, so the pin does not survive a stop
and start.

When stdin and stderr are both terminals, the Go client shows the recorded and offered
fingerprints and asks `[y/N]` before replacing that pod's entry. Without a terminal (an agent,
a script, CI) it fails with the offered fingerprint and names `runpodctl ssh forget <pod-id>`,
which removes only that pod's entry so the next connection enrolls the key the pod then offers.
**Never run `ssh forget` unprompted**: clearing the entry destroys the only signal that separates
a restart from interception. Surface the fingerprint and ask whether the pod was stopped,
updated, or reset since the last connection. Once the operator confirms one of those,
`ssh forget` and a retry is the recovery. If none happened, treat it as possible interception.

For about 30 seconds after `pod start`, the API still reports the previous container's address.
A connection in that window can be refused, reset, end its handshake with `EOF`, or drop
mid-`rsync`; the CLI reports these as `could not reach pod <id>: it may still be restarting`.
That is not a trust failure. Wait and retry.

Trust on first use closes silent key substitution on **later** connections. It does **not**
prevent interception of the **first** connection to a pod; that needs the host key delivered
over an authenticated Runpod channel, which is separate and unstarted work.

## File Transfer

`send`/`receive` do encrypted, incremental, compressed transfer — don't pre-tar or
pre-compress the source. **Agent flow (one side sends, the other receives):**

1. Run `send <path>` **without** a code. The **first line of stdout is the one-time code**;
   `send` then blocks until the receiver connects — so capture that first line as it streams
   (background the process, tee to a log) rather than waiting for exit.
2. On the other machine (use `runpodctl ssh` into the pod/host if needed) run `receive <code>`
   with that exact code — positional, there is no `--code` flag. Each `send` mints a **fresh**
   code — never reuse or invent one.
3. Both processes must exit `0`. On failure, re-run `send` and use its **new** first-line code.

To push local files to a pod: get `ssh info <pod-id>`, start `send` locally (capture the
code), then `ssh` to the pod and run `receive <code>` there. For large/library-style data, a
network volume is often simpler than `send`/`receive`.

### Network volume upload and download (v3.1.0+)

`network-volume upload <volume-id> <local-path> [remote-path]` and
`network-volume download <volume-id> <remote-path> [local-path]` move a file or a folder
through the volume's S3 API, with no pod running. `"/"` downloads the whole volume.

- **Keys:** S3 API keys, separate from the API key: `RUNPOD_S3_ACCESS_KEY` /
  `RUNPOD_S3_SECRET_KEY`, `--s3-access-key` / `--s3-secret-key`, or `s3AccessKey` /
  `s3SecretKey` in `~/.runpod/config.toml`. They are created in the console (Settings > S3
  API Keys); missing keys are `no_credentials`. The API key looks up the volume's data
  center unless `--data-center-id` is passed; a data center without the
  [S3 API](https://docs.runpod.io/storage/s3-api) is `usage_error`.
- **Resume:** on `transfer_incomplete`, run the same command again. Files whose
  destination has the same size and is not older are skipped. Files over 64 MB upload as
  multipart, and a rerun reuses every part whose MD5 matches the local bytes. Files over
  64 MB download in chunks into `<name>.rpdownload` (finished chunks listed in
  `<name>.rpdownload.json`), and the file appears under its real name only once complete.
- **`--force`** sends every file, for a same-size edit that kept an older mtime (`cp -p`,
  `rsync -a`, `tar x`), which the skip rule would pass over.
- **Retries:** 429, 5xx (including 524) and dropped connections are retried inside the
  command; `transfer_incomplete` means those ran out. `volume_full`: grow the volume with
  `network-volume update --size`, then rerun. Parts of an upload that is never rerun stay
  in the volume's hidden `.s3compat_uploads/` folder and count toward its size.
- **Prefer it over `send`/`receive`** for anything big that should land on a volume. For
  other S3 tools against the same API, see `runpod-companion-clis` (`aws`).

## Hub, templates, volumes, registry, info, utilities

Plain CRUD plus filters — `--help` is complete and this file would only go stale restating it:

```bash
runpodctl hub --help              # list/search/get; --type, --category, --owner, --order-by
runpodctl template --help         # list/search/get/create/update/delete; --type official|community|user
runpodctl network-volume --help   # list/get/create/update/delete
runpodctl registry --help         # list/get/create/delete
runpodctl gpu list --help         # + $/hr per cloud and dataCenterAvailability[]
runpodctl datacenter list --help  # alias: dc
runpodctl billing --help          # pods / serverless / network-volume
runpodctl user --help             # account + balance (alias: me)
runpodctl doctor                  # interactive: diagnose and fix cli issues
runpodctl completion              # auto-detect shell and install completion
```

Which of these to reach for, and the traps that are not flags (Hub worker selection, CPU
endpoints, template-vs-image, cost guards) are in the [SKILL.md](../SKILL.md) decision rules.

---
concepts: [model-store, network-volume, container-disk, serverless-endpoint, volume-cache]
---
# Getting a model to a serverless worker

Four ways to make model weights available to a Runpod serverless worker. Pick by
where the weights come from, how large/private they are, and how often they change.

| Method | How | Best when | Cold start |
|--------|-----|-----------|-----------|
| **Bake into image** | `COPY`/download during `docker build`, push to a registry | small/private weights; fully reproducible image | fast (in image) but bloats pulls |
| **HF model cache** (`--model-reference`) | point the endpoint at a HuggingFace model URL; Runpod caches it host-side | the model is already on HuggingFace (public/gated/private) | fastest load once the worker runs, no download billing; worker start waits for a host that holds the model, so the whole cold request can be slower |
| **Network volume** | pre-load weights onto a DC-pinned volume, mount it; in your own handler, copy them to local disk with `VolumeCache` | large weights reused across workers; you manage the files | fast once populated; with `VolumeCache` the load reads local disk instead of the network mount; volume is pinned to one data center |
| **Model Repository** (`runpodctl model`) | upload your **own** artifacts to Runpod-managed, versioned storage | private/custom models not on HuggingFace, without image bloat or a DC-locked volume | managed + host-distributed |

Rule of thumb: on HuggingFace → **HF cache**; your own artifact → **Model Repository**
(or a network volume if you want to manage the filesystem yourself); need a fully
reproducible image or system libs baked in → **bake**. Cache vs volume is a latency
call, not a cost call — see the next section before choosing.

## Cache or network volume — it is a latency call, not a cost call

**You are not billed for model download time, cache hit or miss.** Runpod's docs are
explicit: if no host already holds the model, "the system delays starting your workers
until the model is downloaded onto the machine where your workers will run, ensuring you
still won't be charged for the download time."

So the thing a cache miss costs you is **time to first response**, not money. And the
way it shows up is easy to misread: the job **sits in the queue** while the worker start
is held back for the download. From the outside that looks like a stuck or idle
endpoint, not a download. It isn't broken — it's the miss path.

### How the cache actually works

Three tiers, nearest first ([Runpod engineering
blog](https://www.runpod.io/blog/building-runpods-model-store)): **host-local disk** →
a **data-center-scoped network volume** → the **origin** (HuggingFace). The scheduler is
locality-aware — it prefers hosts that already have the model, or can reach it from a
nearby cache. On a miss, one worker downloads and then everyone in that data center
reads the shared copy. Runpod's own writeup says quota and eviction policy on that
DC-scoped tier are "still open problems, not solved ones", and that a full volume falls
back to origin downloads. Treat residency as best-effort, not a guarantee.

**Capacity is per machine, not per platform.** The host tier is a share of each
machine's own disk, so how much it holds depends on the machine your worker lands on —
the same model can fit one host and not another. That is why there is no cache-size
number to look up, and why two workers on the same endpoint can cold-start at very
different speeds.

**A tier that can't serve you fails quietly.** If the shared copy is unusable — the data
center isn't set up for it, or its storage is full — you don't get an error. The
endpoint falls back to per-host downloads and just feels slower. Inconsistent cold
starts across workers on one endpoint are the normal symptom, not a bug to chase.

### Which to pick

| Situation | Use |
|---|---|
| **Latency-sensitive** — user-facing, tight SLA, or you scale into fresh regions often and can't absorb a first-request stall | **Network volume**, pre-loaded. Costs storage per month, but the weights are already there. In a handler you write, load them through `VolumeCache` ([below](#network-volume--load-from-local-disk-with-volumecache)). |
| **Not latency-sensitive**, or the model isn't huge | **HF model cache** (`--model-reference`). Free, nothing to pre-load, and a hit is seconds. |
| Unsure | **Test both.** This is a measurable trade, not a rule — deploy each and time the cold start on your model, in your region. |

**Measure your own endpoint — don't reason from someone else's numbers.** This is a beta
feature and it does not behave identically for every account or every host today. A
benchmark from a blog post, a forum thread, or another endpoint tells you very little
about what yours will do.

Two constraints push large models toward a volume: cache capacity varies by host, so a
large model's residency is a coin flip rather than a guarantee; and a network volume is
**pinned to one data center**, so multi-region means one pre-loaded volume per DC
(golden path [10](../../runpod/golden-paths/10-multi-region-ha-serverless.md)).

To measure it, watch the worker logs on a cold start in a fresh region (`runpodctl
serverless logs <endpoint-id>`, or the MCP `stream-worker-logs`). Weight-download time in
**minutes** rather than seconds means that region isn't carrying the model for you.

### Regional availability — unknown, don't guess

**Runpod does not publish a list of data centers where the model cache is enabled.** The
[cached models docs](https://docs.runpod.io/serverless/endpoints/model-caching) state no
regional restriction, and the engineering blog only says the shared tier is
"scoped per datacenter". Third-party posts claim it is region-limited; none cite a
source. **Do not state a supported-region list to a user.**

The right thing to check is which data centers offer network volumes at all: the shared
tier is built on that per-DC storage, so a data center without it cannot serve a shared
copy — workers there fall back to per-host downloads. Read it live, don't memorize it:

```bash
# MCP: list-data-centers  →  networkVolumeTypes
```

Snapshot 2026-09-22 — **17 of 33** data centers report a `networkVolumeTypes` value:
`AP-JP-1`, `CA-MTL-3`, `CA-MTL-4`, `EU-FR-1`, `EU-NL-1`, `EU-RO-1`, `EUR-IS-1`,
`EUR-IS-3`, `EUR-NO-1`, `EUR-NO-2`, `US-CA-2`, `US-CO-1`, `US-IL-1`, `US-MO-2`,
`US-NC-2`, `US-TX-3`. Necessary, not sufficient — network storage in a data center does
not by itself prove the cache is enabled there, so present this as **the list to check,
not the official model-cache region list**. If a user needs certainty for a specific
region, deploy there and measure the first cold start.

### Documented limits

- **One cached model per endpoint** ([docs → Current
  limitations](https://docs.runpod.io/serverless/endpoints/model-caching)). `runpodctl`
  accepts `--model-reference` repeatedly, so the CLI will take more than one — the
  platform limit is the binding one.
- **All quantizations get downloaded.** If a HF repo holds several (4-bit AWQ, 8-bit
  GPTQ, …), the system currently pulls them all; you can't select one yet.
- The feature is **beta**.

## Network volume — load from local disk with `VolumeCache`

**Model on Hugging Face and a GPU endpoint? Try the [HF model cache](#hf-model-cache----model-reference)
first.** It needs no volume and no handler code, and download time isn't billed. The trade:
worker start waits until a host holds the model, so time the **whole cold request**, queue
delay included, not just the load. Use a volume with `VolumeCache` when that wait is too long
for your endpoint, or when the model isn't on Hugging Face.

Mounting a volume and pointing the model path at `/runpod-volume/...` makes every cold
start read the weights over the network mount. In a handler you write (a custom image or
Flash), keep the model cache on container disk and wrap the load in `VolumeCache` from the
Runpod Python SDK (`runpod>=1.12.0`). It keeps a mirror of that directory on the volume,
copies the mirror onto local disk before the load (`hydrate()`), and syncs new or changed
files back after it (`sync()`). The first worker downloads from the Hub and every later
cold start hydrates from the mirror.

```python
import runpod
from runpod.serverless import VolumeCache
from transformers import AutoModelForCausalLM

HF_CACHE = "/opt/hf-cache"          # container disk, not /runpod-volume

with VolumeCache(dirs=[HF_CACHE]):  # hydrate before the load, sync after
    model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct", cache_dir=HF_CACHE)

def handler(job):
    ...

runpod.serverless.start({"handler": handler})
```

- **Your code only.** It runs where your handler wraps it. A prebuilt worker image (a
  stock vLLM or ComfyUI worker) reads its model path from the mount as before.
- **Best-effort hides failures.** By default any cache error is logged and swallowed and
  the worker cold-starts normally, so a broken cache looks like a slow cold start. Use
  `best_effort=False` while you set it up.
- **Keep Hub access on.** It never copies symlinks, and the HF cache's `snapshots/` entries
  are symlinks, so `from_pretrained` re-links them with a metadata call. Don't set
  `HF_HUB_OFFLINE=1` on a cache that `VolumeCache` restores.
- **Size the container disk for the whole local cache** plus runtime headroom: the hydrate
  and any fallback download both write there. Set it on the template
  (`runpodctl template create --container-disk-in-gb`), or with `container_disk_gb=` on a Runpod
  SDK decorator, which defaults to 30 GB on GPU.
- **Size the volume for the mirror** (`/runpod-volume/.cache/<RUNPOD_ENDPOINT_ID>`). Large
  files you delete or rename locally stay on the volume, so usage grows as you swap models.
- **Populate with one worker.** Workers that cold-start together on an empty mirror each
  download and sync a full copy back.

- **A global volume works too.** Mount it where the network volume would go (`/runpod-volume`
  on an endpoint; GPU compute only) and the same code mirrors there. Use `runpod>=2.0.0`:
  older versions can skip or drop large files on a global volume and swallow the error, so the
  worker downloads the model again. Use one when the endpoint spans data centers.

### Measured cold starts

What a fresh worker spent getting Qwen2.5-1.5B-Instruct (3.1 GB) from storage onto the GPU,
measured on Runpod Serverless (L4 and RTX 4090 workers, 2026-10-17):

| Where the weights come from | Download or restore | Load | Total |
|---|---|---|---|
| Download from Hugging Face every cold start | — | — | 16–21 s |
| Load straight from a network volume | — | 11 s | 11 s |
| `VolumeCache` restore from a network volume | 5.2 s | 3.9 s | **9.1 s** |
| `VolumeCache` restore from a global volume (before 2.0.0) | 26–51 s | ~4 s | 30–55 s |

- **Copy first, then load.** The restore read the volume at about 600 MB/s, and loading from
  local disk is quick. Loading straight off the mount reads every byte over the network.
- **Global volumes were the slowest source.** These restores copied whole files and were several
  times slower than from a network volume. 2.0.0 restores in parallel byte ranges, so re-time a
  global volume on 2.0.0 before ruling it out.
- Treat these as direction, not a spec: time your own model in your data center.

Exact rules: [`volume-cache`](../../runpod-usage/concepts/volume-cache.yaml).

## HF model cache — `--model-reference`

Attach a HuggingFace model to an endpoint by full URL with a ref; Runpod caches it on
the host so the worker loads it directly — no bake, no volume.

```bash
runpodctl serverless create --template-id <id> --gpu-id "NVIDIA GeForce RTX 4090" \
  --model-reference https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct:main
```

- The trailing `:main` is the branch/tag/revision.
- Weights land in the standard HF cache dir `/runpod-volume/huggingface-cache/hub/`
  (`models--{org}--{name}/snapshots/{hash}/`). Official workers can integrate that
  location; a custom worker must configure its loader or pass the matching local
  snapshot. See the public [cache integration guide](https://docs.runpod.io/serverless/development/huggingface-models).
  For a loader that uses standard HF cache variables, `HF_HOME` is the cache root
  and `HF_HUB_CACHE` is the `hub` directory; verify the selected repository and revision.
- The CLI flag is repeatable, but the platform currently allows **one cached model per
  endpoint** (docs → Current limitations).
- Works with `--template-id` **and** `--hub-id`, but **GPU only** (`--compute-type GPU`).
- **Requires runpodctl v2.4.0+.** Check `runpodctl version`; the Homebrew tap can lag —
  prefer the [GitHub releases](https://github.com/runpod/runpodctl/releases) binary.
- Gated/private HF models: provide an `HF_TOKEN` (endpoint env var).

You are **not billed for download time** — hit or miss. Cold starts drop to seconds on
a host that already holds the model; on one that doesn't, worker start is held back
until the download finishes, so the job sits in the queue longer. See
[Cache or network volume](#cache-or-network-volume--it-is-a-latency-call-not-a-cost-call).

## Model Repository — `runpodctl model`

Runpod-managed **storage + registry for your own model artifacts**. Upload weights once;
Runpod stores, versions, and distributes them to workers — a first-class model object
with a name, versions, metadata, and a status lifecycle (not just a file on a disk).

```bash
runpodctl model list                                  # list your models
runpodctl model list --all                            # include models you don't own
runpodctl model list --name "llama"                   # filter by name
runpodctl model list --provider "meta"                # filter by provider
runpodctl model add --name "my-model" --model-path ./model   # upload a local model directory
runpodctl model remove --name "my-model" --owner <owner>     # remove a model
```

`model add` runs a **multipart upload session** (built for large weights) — the live
`runpodctl model add --help` exposes `--create-upload`, `--part-size`, `--file-size`,
`--file-name`, `--content-type`, `--metadata key=value`, `--model-status`,
`--version-status`, and `--credential-reference`/`--credential-type` (for pulling from a
private source). Run it before relying on exact syntax — it is authoritative.

**Coming from a baked-in model?** The easiest migration is to stop `COPY`-ing weights
into the image and instead `runpodctl model add --model-path <the same dir you used to
COPY>`, then reference the uploaded model from the endpoint. This shrinks the image and
lets you version the weights independently of the code.

**vs a network volume:** a network volume is a raw filesystem you manage and is **pinned
to one data center** (workers must run there); the Model Repository is managed, versioned,
and host-distributed, so it isn't locked to a single DC. Use a volume when you want direct
filesystem control or are already populating one; use the Model Repository for a
hands-off, versioned artifact.

> **Runtime-load path — don't guess.** The command surface above (`runpodctl model add`)
> is from live `runpodctl --help` and is accurate for *uploading*. But how a worker
> *references an uploaded Model Repository artifact at runtime* is **not publicly
> documented yet** — if asked for exact runtime-load steps for a `runpodctl model add`
> artifact, say it isn't publicly documented and don't invent a path.
>
> This is distinct from the HF **model-caching** feature (the `--model-reference` flag /
> the endpoint "Model" field), whose runtime path **is** documented: the model lands in
> `/runpod-volume/huggingface-cache/hub/` and the handler resolves the local snapshot from
> there. Enable `HF_HUB_OFFLINE=1` only after all required components are present, so it
> does not block an intended download of a missing component. See golden path 20 and the
> [official custom-worker example](https://docs.runpod.io/serverless/development/huggingface-models).

# runpodctl — output format, error codes, env vars

Everything an agent needs to parse runpodctl's output and decide what to do with a
failure. The 80% rule is in [SKILL.md](../SKILL.md#output--errors); this file is the
full surface.

**Version floor: runpodctl ≥ v2.8.0.** The coded error shape, the serverless `urls` object
and GPU pricing/per-DC availability all arrived in v2.8.0.

Older binaries (≤ v2.7.3) did **not** print plaintext — they printed
`{"error":"<message>"}` on stderr with no `code` and no `status`, and the message was
interpolated unescaped, so a message containing a quote produced malformed JSON. They also
dumped usage text after runtime errors. So the pre-v2.8.0 failure mode is *valid-looking
JSON with the field you're branching on missing* — **gate on `code` being present**, not on
whether stderr parses as JSON. `runpodctl version` is a weak gate: it prints plaintext, and
a binary built from source reports a placeholder version regardless of the code it contains.

## Output format

JSON is the **default** — the CLI is built for agent consumption, and data goes to
**stdout**.

```bash
runpodctl pod list                    # json (default)
runpodctl pod list --output=yaml      # yaml — the only alternative
```

**There is no table format** — `json` and `yaml` are the only values. As of **v2.10.0** an
unrecognized value is a hard error before any API call, and matching is
**case-insensitive**:

```jsonc
// runpodctl gpu list --output=table   (v2.10.0+)
{"error":"invalid --output \"table\": supported formats are json and yaml","code":"usage_error"}
// exit 1, usage text on stderr
```

```bash
runpodctl gpu list --output=YAML      # v2.10.0+: real YAML (case is ignored)
```

⚠️ **This inverted in v2.10.0, so a handler has to know which binary it is on.** Through
**v2.9.0** the flag was matched *case-sensitively* and anything unrecognized fell back to
JSON **silently** — `--output=table` and `--output=YAML` both returned JSON, and the only
safe advice was "pass lowercase `yaml` or don't pass the flag". That silence is how a table
format the CLI never had ended up documented in the first place. On v2.10.0+ the same
`--output=table` exits **1** with `usage_error`, so code that passed a defensive
`--output=json` is fine but code passing anything else now fails loudly instead of
degrading.

(The legacy `get pod` / `get cloud` commands print a human table on stdout and ignore
`--output` entirely — see [plaintext gaps](#plaintext-gaps).) Some commands print a human table on stdout regardless — the legacy
`get pod`/`get cloud` paths do — so a table on stdout is a signal you're on a legacy
command, not a format you asked for.

## Error format

Errors go to **stderr** as a single flat JSON object, and the exit code is
**non-zero**, so stdout parses as data.

```jsonc
{"error":"failed to get endpoint: endpoint not found","code":"not_found","status":404}
```

| field | notes |
| --- | --- |
| `error` | human-readable message, unwrapped on the REST path. A GraphQL HTTP failure whose body has no extractable message falls back to the **raw body**, which can itself be JSON — so never parse this field, either way |
| `code` | stable, lowercase, present on every JSON error (see [plaintext gaps](#plaintext-gaps)) |
| `status` | HTTP status — only when the failure arrived on a **non-2xx** response (REST *or* GraphQL) |
| `id` | **v2.9.0+**, only on a failure that left a **created, billing resource** behind — a `create --wait` that timed out or was interrupted. Read it instead of regexing the message: it is the id you need to clean up |

**Branch on `code`, never on `status` or the message text.** `status` is absent whenever
the API answered HTTP 200 with empty data, which is how GraphQL reports a missing
resource — so `if (status === 404)` misses every GraphQL not-found:

```jsonc
// serverless get <missing> — REST, so a status is available
{"error":"failed to get endpoint: endpoint not found","code":"not_found","status":404}
// template get <missing> — GraphQL answered 200/null, so there is no wire status
{"error":"failed to get template: template not found: bogus-template-id","code":"not_found"}
```

`code` is both: a typed code (from the API or from the error itself) wins, and the
output sink fills in a fallback when there isn't one — so local validation and transport
failures carry a code too, not just API responses.

**`timeout` is the one documented exception**, because two outcomes share it and only the
message separates them (see the [codes table](#codes)). A handler that reads only `code` is
still correct: treat `timeout` as never-retry and poll instead. Reading the message can
upgrade that to "safe to retry" for the single-API-call case, but skipping it costs you
nothing except an unnecessary poll — whereas retrying on the wrong one buys a second job.

### Codes

| code | meaning |
| --- | --- |
| `usage_error` | unknown command, unknown flag, bad/wrong-count args, or a **cobra-required** flag left out. Usage text prints after the JSON |
| `not_found` | the **API** does not have the resource. During a `create --wait` it can also mean one that *was* created has gone, or never became visible — so check for an `id` field before concluding nothing exists |
| `bad_request` (400) `unauthorized` (401) `forbidden` (403) `conflict` (409) `rate_limited` (429) `server_error` (5xx) `api_error` | derived from the REST status |
| `graphql_error` | a GraphQL call failed: an `errors` array in a 200 body, **or** a non-2xx response from the GraphQL endpoint (that form also carries `status`) |
| `no_credentials` | no API key configured at all — set `RUNPOD_API_KEY` or run `runpodctl doctor`. For `network-volume upload`/`download`: no S3 API keys (`RUNPOD_S3_ACCESS_KEY` / `RUNPOD_S3_SECRET_KEY`), or no API key for the data-center lookup, which runs first unless `--data-center-id` is passed; the message names which |
| `network_error` | the API could not be reached at all — DNS, connection refused, TLS, timeout |
| `cli_error` | anything else local: config, environment, and validation the command does itself |
| `timeout` | **v2.9.0+.** The CLI stopped waiting — two outcomes sharing one code, told apart by the message. If it names a `serverless status` command, the **job is still running server-side**: poll it, never re-invoke (that buys a second job). Otherwise a single API call ran out of time and nothing is running, so a retry is fine |
| `job_failed` | **v2.9.0+.** A serverless job reached a terminal status other than `COMPLETED` (`FAILED`/`CANCELLED`/`TIMED_OUT`). The request itself succeeded — the job payload, including the worker's own `error`, is still on **stdout** |
| `wait_timeout` `wait_interrupted` | **v2.9.0+.** A `create --wait` gave up (budget) or was cancelled (ctrl-c / SIGTERM). The resource **was created and still bills**; its id is in the `id` field above |
| `transfer_incomplete` | **v3.1.0+.** A `network-volume upload`/`download` stopped partway: ctrl-c, a network failure or 429/5xx that outlasted its own retries, or a file that changed mid-transfer. Rerun the **same command**: it skips finished files and continues partial ones. Failures a rerun would hit again keep their own code and don't resume: `not_found` (missing path), `unauthorized`/`forbidden`/`bad_request` (with `status`), `volume_full`, `usage_error` (bad path, data center without the S3 API), `cli_error` (local disk errors, names that would collide on disk) |
| `volume_full` | **v3.1.0+.** A `network-volume upload` filled the volume (S3 `403 QuotaExceeded`, with `status`). Parts of an unfinished multipart upload count toward its size. Grow it with `network-volume update <volume-id> --size <gb>`, then rerun the same command: it resumes |

Treat this as **the set the CLI generates, not an exhaustive list** — an explicit code
from the API is passed through lowercased, so a code outside the table can arrive from
the server. **Give your handler a default branch that treats an unknown code as fatal**
and surfaces `error` verbatim; never fall through to a retry.

### What to retry

**The API commands have no internal retry** — they never back off for you, so whatever
your wrapper does is all that happens. The exception is `network-volume upload`/`download`
(v3.1.0+), which retry throttling, 5xx (including the 524 Runpod's proxy returns) and
dropped connections themselves, up to 10 attempts, before giving up with
`transfer_incomplete`.

- **Retry with backoff:** `network_error`, `rate_limited`, `server_error` (and `api_error`
  when `status` is 5xx). A transient `graphql_error` is possible too but indistinguishable
  from a permanent one, so cap those attempts tightly.
- **Never retry:** `usage_error`, `cli_error`, `bad_request`, `not_found`, `conflict`,
  `no_credentials`, `unauthorized`, `forbidden`, `job_failed` (the job ran and failed —
  re-running is a new job, not a retry).
- **Rerun the same command:** `transfer_incomplete`. The transfer already retried; the rerun
  resumes it, skipping finished files and reusing finished parts. `volume_full` is the same
  after a fix: grow the volume, then rerun.
- **Never retry the *command*, but do follow up:** `timeout` and
  `wait_timeout`/`wait_interrupted` all mean work or a resource outlived the CLI. Re-running
  the same command creates a **second** job or a **second** billed resource. Poll instead
  (`serverless status`, `pod get`), or clean up using the `id` field. The one exception is a
  `timeout` whose message does **not** name a follow-up command: that was a single API call
  timing out with nothing left running, so it is safe to retry.

Two invariants make that safe to encode:

- **`not_found` always means server-side.** A bad *local* path (say a mistyped
  `--model-path`) is `cli_error`, so `not_found` never means "you typed a path wrong".
- **`network_error` is the only code the CLI assigns to mean "couldn't reach the API"**,
  detected structurally. Deliberately *not* `network_error`: a malformed `RUNPOD_API_URL`
  is `cli_error`, so a retry loop never fires for something a retry cannot fix. A local wait
  loop timing out used to land here too — as of **v2.9.0** `model add --wait-for-hash`
  reports `timeout` instead of `cli_error` (same exit code, same message text), so a handler
  switching on `code` needs that branch.

### Auth failures are two different codes

`no_credentials` means **no key is configured** (for `network-volume upload`/`download`,
no S3 API keys, or no API key for the data-center lookup unless `--data-center-id` is
passed; the message names which. The S3 API rejecting keys that are set is
`unauthorized`/`forbidden`). A key that is present but wrong, expired
or revoked is `unauthorized` (401), and one lacking access to the resource is `forbidden`
(403):

```jsonc
// RUNPOD_API_KEY=rpa_bogus… pod list
{"error":"api request failed with status 401","code":"unauthorized","status":401}
```

Don't collapse these — re-prompting for a "missing" key when the real problem is a
revoked one sends an agent in a circle.

### Invocation mistakes split across two codes

`usage_error` covers what cobra validates: unknown command/flag, wrong arg count, and
missing `MarkFlagRequired` flags. Validation a command performs itself lands in
`cli_error`, even though it's just as much an invocation mistake:

```jsonc
// ssh remove-key with neither --name nor --fingerprint
{"error":"either --fingerprint or --name must be provided","code":"cli_error"}
```

So `cli_error` is a mixed bucket — "your invocation was wrong" *and* "your local
environment is wrong". Read `error` to tell them apart; don't retry either.

### Missing API key

The JSON-covered commands all report `no_credentials` — `pod` (`list`/`get`/`create`),
`template create`/`update`, every `ssh` subcommand, `model *`:

```jsonc
{"error":"api key not found. get your key at https://www.runpod.io/console/user/settings then: export RUNPOD_API_KEY=your-key OR run: runpodctl doctor","code":"no_credentials"}
```

```jsonc
{"error":"unknown flag: --nope","code":"usage_error"}
```

Usage text follows the JSON on stderr — and, unlike older builds, a *runtime* error no
longer dumps usage text after it.

### Exit codes, and what else is on stderr

Every failure on the JSON path exits non-zero (`model` and `update` used to print an error
and exit **0**; they don't anymore). The exceptions are in
[plaintext gaps](#plaintext-gaps) below.

**Non-empty stderr is not a failure signal.** Deprecation notices (`warning: 'runpodctl
get pod' is deprecated…`), `pod create` advisories (`note: …`) and config-migration lines
all go to stderr on successful runs. Gate on the exit code first, then parse stderr.

### Plaintext gaps

The JSON error shape covers `pod`, `serverless`, `template`, `volume`, `registry`,
`gpu`, `datacenter`, `billing`, `user`, `model`, `ssh`, `send`, `receive`, `hub` and
`update` — including every GraphQL call site (a GraphQL failure is `graphql_error`, or
`not_found` for the nil-data lookups). These surfaces still print plaintext and carry
no `code`:

| surface | shape |
| --- | --- |
| legacy `get`/`create`/`remove`/`start`/`stop pod`, `create`/`remove pods`, `get cloud` | `Error: <msg>` on stderr, exit 1 — including the missing-key case, which carries the same message as `no_credentials` but no JSON and no `code`. Success prints a human **table** on stdout, not JSON |
| `exec` (hidden, deprecated) | **v2.10.0+:** joined the JSON error shape — flat JSON with a `code` on stderr and a **non-zero** exit. Through v2.9.0 it printed the error as plaintext and still exited **0**, so a caller that trusted the exit code saw a silent success. Either way: progress on **stdout**, and it polls up to **5 minutes** for the pod's SSH info before giving up |
| `project` (hidden) | **Released versions through v3.0.1:** runtime errors print to **stdout** and exit **0**; inspect the output. **Builds containing unreleased [runpod/runpodctl#382](https://github.com/runpod/runpodctl/pull/382):** plain-text Runpod SDK notice on **stderr**, empty stdout, exit **1**; treat the notice as fatal and do not retry |

#### Project Runpod SDK notice

Use the [Runpod SDK](https://github.com/runpod/runpod-python) for serverless
development regardless of CLI version.

**Build scope:** the notice contract applies to builds containing
[runpod/runpodctl#382](https://github.com/runpod/runpodctl/pull/382), which is
unreleased. The latest published release is **v3.0.1**. Check `runpodctl version`;
released versions through v3.0.1 print `project` runtime errors to **stdout** and
exit **0**, so a zero exit status does not establish success. For source or custom
builds, verify the build's commit before assuming the notice contract: their
version string can match a released version, including v3.0.1.

In builds containing #382, `runpodctl project ...` invocations reach the hidden
catch-all, including `new`, `start`, `deploy`, `build`, the `create`/`dev` aliases, unknown
subcommands and flags. Supported help forms include `runpodctl project --help`,
`runpodctl project deploy --help`, `runpodctl help project`,
`runpodctl help project deploy`, `runpodctl -h project`,
`runpodctl --help project` and `runpodctl --help project deploy`.
Each exits **1** with empty stdout and exactly this plain-text stderr, including
the final newline:

```text
The "project" command has been removed.

Use the Runpod SDK for serverless development workflows:
  https://github.com/runpod/runpod-python
```

Surface the notice verbatim without requiring a JSON `code`, and treat the failure
as fatal. Use the [Runpod SDK](https://github.com/runpod/runpod-python) for serverless development.

### Parsing `ssh info`

`ssh info` has three shapes, and only one of them is an error:

| situation | output |
| --- | --- |
| connectable | `{"id","name","ssh_command","ip","port","ssh_key":{…}}` on stdout, exit 0 — note **snake_case**, unlike the camelCase everywhere else in the CLI. A `"setup":"runpodctl doctor"` key appears when the local key needs fixing |
| pod exists, not connectable | `{"error":"pod not ready: <reason>","id":…,"name":…,"status":…}` on **stdout**, exit **0**, no `code`. Here `status` is the pod's desired status, *not* an HTTP status. **v2.9.0+** appends a reason when it has one (image still pulling, port 22 not published, pod stopped, …) and falls back to the bare `"pod not ready"` when it doesn't — so an exact-match `=== "pod not ready"` started failing intermittently on that release; a prefix match did not |
| no such pod | `{"error":"pod 'x' not found","code":"not_found"}` on stderr, exit 1 |

So: check the exit code, then check for an `error` key even on stdout.

"Not ready" fires whenever the pod has no **public port 22**, which is not the same thing
as "still booting". A `ubuntu:22.04` CPU pod created through the earlier v1 path stayed at
`status: RUNNING` without SSH for ~70 s; that path could not request Runpod-managed SSH.
The current API v2 CPU create path requests managed SSH when SSH is enabled. So **bound
any SSH readiness loop** and don't treat `RUNNING` alone as proof that SSH is ready.

Two v2.9.0 changes make this easier to get right: `pod create --wait` does the bounded
readiness loop for you (and proves an ssh *banner*, not just a published port), and
`runtimeStatus` on `pod get`/`pod list` distinguishes `initializing` from `running` so you
no longer have to infer it from `desiredStatus`. Also **v2.9.0+**: a **stopped** pod no
longer returns an `ssh_command` at all (it used to hand back one built from stale runtime
ports that could never connect), and it drops out of `ssh connect`'s `connections` list,
which is now `[]` rather than `null` when nothing is reachable.

So a parser should tolerate a non-JSON line on stderr from those. Treat the
`project` SDK notice's exit 1 as fatal in builds containing #382. For released
versions through v3.0.1, inspect `project` stdout even on exit 0; `exec` before
v2.10.0 also requires checking its output. Prefer the non-legacy
equivalents: `pod get`/`pod create`/`pod delete`, and `ssh info <pod-id>` + your own `ssh`
invocation instead of `exec`.

## Environment variables

| variable | default | what it sets |
| --- | --- | --- |
| `RUNPOD_API_KEY` | — | API key. Also settable via `runpodctl doctor` or `~/.runpod/config.toml` (`apikey`) |
| `TIMEOUT` | `30s` | per-API-call deadline — **no `RUNPOD_` prefix** (the CLI reads env vars unprefixed, so this one looks nothing like the others). Exceeding it is `code: "timeout"`. Also settable as a top-level `timeout` in `~/.runpod/config.toml` — it must sit **above** any `[section]` header, or it becomes `section.timeout` and is silently ignored. Distinct from `--wait`, which bounds a whole job or readiness wait rather than one call |
| `RUNPOD_API_URL` | `https://rest.runpod.io/v1` | REST control plane (config key `restApiUrl`) |
| `RUNPOD_GRAPHQL_URL` | `https://api.runpod.io/graphql` | GraphQL control plane (config key `apiUrl`) |
| `RUNPOD_S3_ACCESS_KEY` / `RUNPOD_S3_SECRET_KEY` | — | S3 API keys for `network-volume upload`/`download` (config keys `s3AccessKey` / `s3SecretKey`). Separate from the API key, which still looks up the volume's data center unless `--data-center-id` is passed |
| `RUNPOD_INVOKE_URL` | `https://api.runpod.ai/v2` | base for the invoke URLs reported by `serverless create`/`get`/`list`/`update` (config key `invokeUrl`) |

**Invoke is a separate service from the control plane.** Pointing `RUNPOD_API_URL` or
`RUNPOD_GRAPHQL_URL` at a non-prod host does *not* move the invoke URLs — override
`RUNPOD_INVOKE_URL` explicitly when you need that, or the emitted URLs target prod.

## Serverless invoke URLs

`serverless create`/`get`/`list`/`update` return a `urls` object, computed from the
endpoint ID, so a freshly created endpoint is callable without a second lookup:

```jsonc
"urls": {
  "run":     "https://api.runpod.ai/v2/<endpoint-id>/run",
  "runsync": "https://api.runpod.ai/v2/<endpoint-id>/runsync",
  "health":  "https://api.runpod.ai/v2/<endpoint-id>/health"
}
```

`status` isn't in the object; it's `<run url minus /run>/status/<job-id>`.

## GPU pricing and per-data-center availability

`gpu list` reports on-demand price per hour per cloud type plus a per-DC stock
breakdown:

```jsonc
{
  "gpuId": "NVIDIA A100 80GB PCIe",
  "displayName": "A100 PCIe",
  "memoryInGb": 80,
  "secureCloud": true,          "securePricePerHr": 1.39,
  "communityCloud": true,       "communityPricePerHr": 1.19,
  "stockStatus": "Low",         "available": true,
  "dataCenterAvailability": [
    {"dataCenterId": "CA-MTL-3", "stockStatus": "Low"},
    {"dataCenterId": "EU-RO-1",  "stockStatus": "none"}
  ]
}
```

- A price is **explicitly `null`** when that cloud type doesn't offer the GPU, which is
  distinguishable from a real `0`. Read `securePricePerHr`/`communityPricePerHr` rather
  than guessing that a lower tier is cheaper.
- Top-level `stockStatus` is the **best status across data centers** — use
  `dataCenterAvailability[]` as ground truth for *where* a create will actually
  schedule, especially when co-locating with a network volume.
- An unrecognized non-empty status now ranks above absent/`none` instead of tying with it,
  so **`available` no longer reports `false`** for a GPU whose only status is a value the
  CLI doesn't know. It still ranks *below* `Low`, so a known level still wins the top-level
  `stockStatus` — that tradeoff is deliberate, and it's why `dataCenterAvailability[]` is
  the field to read. Casing and surrounding whitespace are normalized now, so they are no
  longer a source of unknown values.
- Two sentinels for one concept: the per-DC breakdown spells an unreported status
  `"none"`, while the top-level field **omits the key** for the same condition. Handle
  both.
- **Stock values are capitalized levels but the sentinel is lowercase** — `"High"`,
  `"Medium"`, `"Low"`, `"none"` in practice. `"unavailable"`, `"out of stock"` and
  `"no stock"` also count as no-stock, and comparison is case-insensitive internally, so
  lowercase before testing and don't write `if s != "none"`.
- **`gpu list` hides no-stock GPUs by default.** Pass `--include-unavailable` when
  targeting one data center, or you may never see a GPU that has stock only there.

Two scoping limits the fields don't express:

- **The prices are pod on-demand rates** (`gpuTypes { securePrice communityPrice }`). They
  are not serverless rates — serverless bills per request-second — so don't present them
  as the cost of an endpoint.
- **`dataCenterAvailability[]` has no cloud-type attribution**, while pricing is split
  secure vs community. "Cheapest option that schedules in DC X" therefore isn't answerable
  from `gpu list` alone; treat it as two separate reads.

And when you act on the choice: pin placement with `--data-center-ids`, or the analysis
doesn't constrain anything. For serverless, `--gpu-id` takes a GPU *type* id and is
translated server-side to a GPU **pool**, so the endpoint may run on more than the one
card you priced (pool vs type ids: [`runpod-usage` gpu-selection](../../runpod-usage/reference/gpu-selection.md)).

---
name: runpodctl
description: >-
  Runpod CLI for managing GPU/CPU workloads from the terminal — pods, serverless
  endpoints, templates, network volumes, Hub deploys, models, SSH, and file
  transfer (send/receive, network-volume upload/download). Use for
  terminal/CI/scripting, SSH setup, `doctor`, or when the Runpod MCP tools are
  not connected; prefer runpod-mcp for structured tool calls in an MCP-enabled
  session.
allowed-tools: Bash(runpodctl:*)
compatibility: Linux, macOS, Windows
metadata:
  author: runpod
  version: "2.0.0" # x-release-please-version
  concepts: [runpodctl, pod, cpu-pod, serverless-endpoint, template, network-volume, hub-repo, ssh-key, file-transfer, model-store, volume-cache]
license: Apache-2.0
---

# Runpodctl

Manage GPU pods, serverless endpoints, templates, volumes, and models. For serverless development workflows, use the [Runpod SDK](https://github.com/runpod/runpod-python) regardless of CLI version.

## Install

`curl -fsSL https://cli.runpod.net | sudo bash` (Linux/macOS, and Windows via WSL) or `brew install runpod/runpodctl/runpodctl`. Manual binaries and the Windows and conda steps: the [runpodctl README](https://github.com/runpod/runpodctl#install), beside the `install.sh` they describe. API key: https://console.runpod.io/user/settings

> Old builds silently lack newer flags/behaviors (e.g. `--model-reference` doesn't exist before
> v2.4.0) and produce confusing downstream errors, and the Homebrew tap can lag well behind.
> Before any work: **update to the latest build** (`runpodctl version`, then `runpodctl update`,
> or reinstall from the [latest release](https://github.com/runpod/runpodctl/releases)); **pin one
> recent version for the whole task**; **never switch between an old and a new binary mid-task**
> (a known failure); **verify once** with `runpodctl version` before you continue.

## Quick start

```bash
runpodctl update && runpodctl version   # FIRST: latest build, confirmed, before any work
export RUNPOD_API_KEY=your_key      # agents: non-interactive auth, read by runpodctl
runpodctl doctor                    # humans: interactive first-time setup (prompts for API key + SSH keys); not for scripted use
runpodctl pod create --help         # inspect exact current flags before creating
runpodctl gpu list                  # GPU types
runpodctl datacenter list           # GPU availability per data center (co-locate GPU + volume)
runpodctl hub search vllm           # find a hub repo
runpodctl serverless create --hub-id <id> --name "my-vllm"  # deploy from hub
runpodctl template search pytorch   # find a template
runpodctl pod create --template-id runpod-torch-v280 --gpu-id "NVIDIA GeForce RTX 4090" --min-cuda-version 12.8  # create from template
runpodctl pod list                  # your pods
```

## Live Help Is Authoritative

Live `runpodctl --help`, `runpodctl <resource> --help` and `runpodctl <resource> <action> --help` are authoritative for exact flags, aliases, and syntax (the generated pages under [`runpodctl/docs/`](https://github.com/runpod/runpodctl/tree/main/docs) mirror them). Inspect live help before unfamiliar commands; this skill covers workflows, decision rules, safety notes, and common examples, and is not an exhaustive flag reference.

Live help does **not** cover output shapes, error codes, or exit-code behavior: use [reference/output-and-errors.md](reference/output-and-errors.md), and when in doubt **probe the binary** (run it wrong on purpose, e.g. `runpodctl serverless get nope`, and read the JSON). Every doc is a snapshot, this skill included; the binary in front of you wins.

## Output & errors

Data is **JSON on stdout** (`--output=yaml` is the only alternative; there is no table format; v2.10.0+ rejects any other value with `usage_error`, v2.9.0 and earlier silently return JSON). A failure is one flat JSON object on **stderr** plus a **non-zero exit**:

```jsonc
{"error":"failed to get endpoint: endpoint not found","code":"not_found","status":404}
```

- **Branch on `code`, never on `status` or the message** (GraphQL reports a missing resource as HTTP 200 + null data, so `status` is often absent).
- **Retry with backoff only on `network_error`, `rate_limited`, `server_error`;** the API commands never retry internally. `no_credentials` = no key set (for `network-volume upload`/`download`: no S3 API keys, or no API key for the data-center lookup, which `--data-center-id` skips; the message names which); `unauthorized`/`forbidden` = a key is set but wrong or lacks access (don't re-prompt). `usage_error`, `cli_error`, `bad_request`, `not_found`, `conflict`: fix the input. Any other code is fatal: surface `error` verbatim.
- **`network-volume upload`/`download` (v3.1.0+) retry on their own** (429, 5xx, dropped connections). `transfer_incomplete` = it stopped partway: **rerun the same command**, it resumes. `volume_full` = grow the volume (`network-volume update --size`), then rerun.
- **Gate on the exit code, then parse stderr:** deprecation `warning:`/`note:` lines go to stderr on success too.
- **Coded errors, the serverless `urls` object and GPU pricing need runpodctl ≥ v2.8.0.** Older binaries emit `{"error":"…"}` with no `code`, so gate on `code` being present, not on `runpodctl version`.
- **`project` behaves differently per build** (exit 0 with errors on stdout through v3.0.1; exit 1 with a Runpod SDK notice in builds with [runpod/runpodctl#382](https://github.com/runpod/runpodctl/pull/382)). Treat that notice as fatal.

Full code table, `cli_error` vs `usage_error`, plain-text exceptions, the exact SDK notice, and the env-var table (incl. `RUNPOD_INVOKE_URL`): **[reference/output-and-errors.md](reference/output-and-errors.md)**.

## Decision Rules

- **Hub** for a known deployable app or worker (vLLM, ComfyUI, Whisper, a Runpod-maintained repo). **Picking a worker:** prefer a **first-party or well-adopted, recently-released** worker on a **broad, high-availability GPU pool**; signals in `runpodctl hub list`: `--owner runpod-workers` (first-party), `--order-by releasedAt`/`updatedAt` (recency), `--order-by deploys`/`stars` (adoption). Don't pin a scarce large-GPU tier a small model doesn't need.
- **Templates** when the user already has a template ID, wants reusable image/config defaults, or needs lower-level control than Hub. **`pod create --image`** for a specific Docker image with no saved template.
- **Serverless** for request/response inference APIs and scalable workers; **pods** for interactive work, notebooks, training, debugging, or long-lived sessions. **CPU pods** for preprocessing, file movement, lightweight scripts, and non-CUDA work; **GPU pods** when CUDA, model inference, training, or GPU memory is required.
- **"Active worker" = minimum workers, not maximum:** `--workers-min 1` (one worker always warm → no cold start; ideal for development/iteration), **not** `--workers-max 1` (that only caps the ceiling).
- ⚠️ **A min-1 worker bills continuously, even while idle** (it defeats scale-to-zero). After dev you **must** set `--workers-min 0` (or delete the endpoint), or it quietly runs up cost.
  - **Needs runpodctl ≥ v2.10.0.** Earlier binaries **silently dropped** `--workers-min 0` and `--idle-timeout 0` from the update request (`omitempty` ate the zero): the reset looked applied and the endpoint kept billing. Check `runpodctl version`; on an older binary confirm with `serverless get <id>` and fall back to `PATCH https://api.runpod.io/v2/serverless/<id>` with an explicit `{"workers":{"min":0}}`.
- **Set the execution timeout on every endpoint create.** A v2 create without `timeout` comes back `"timeout": 0` (live-verified 2026-10-29; [why it matters](../runpod-usage/reference/gotchas.md#endpoint-timeout-comes-back-0)). runpodctl ≥ 2.14.0 has `serverless create --execution-timeout <seconds>` (as of 2.14.0 `serverless update` does not; check `--help`); read the timeout back with `serverless get`, and fix a 0 with `PATCH https://api.runpod.io/v2/serverless/<id>` `{"timeout":<ms>}`.
- `serverless update` has **no `--gpu-id` flag**. To change an endpoint's GPU pools, `PATCH https://api.runpod.io/v2/serverless/<id>` with `{"gpu":{"pools":["ADA_24"]}}` (pool ids, from the `pool` field of `GET /v2/catalog/gpus`). Sending `pools` without `excludedTypes` clears any exclusions.
- **CPU serverless endpoints:** `runpodctl serverless create --compute-type CPU` (on REST v2, send `cpu` instead of `gpu`). **Never** send `"computeType":"CPU"` to REST v1 `POST https://rest.runpod.io/v1/endpoints`: it silently provisions a **GPU** endpoint ([evidence](reference/command-reference.md#cpu-endpoints-and-rest-v1)).
- **Leave `--cloud-type` at its `SECURE` default** ([why](../runpod-usage/reference/gpu-selection.md#step-5-secure-vs-community-cloud)). Pass `--cloud-type COMMUNITY` only when the user explicitly asks (a public port 22 there needs a machine with a public IP, which the CLI cannot request).
- A CPU pod gets 2 vCPUs on whichever CPU flavor has the most stock; `--name` defaults to the image name without its tag. CPU pods take a network volume (`--network-volume-id`) but not a pod volume (`--volume-in-gb` is rejected). Do not pass GPU flags.
- **Set a CUDA floor on every GPU create: `--min-cuda-version 12.8`** (`pod create` and `serverless create`; not `template create`). Without it a modern image can land on a host too old to run it, and the create will not reject it: the container dies at startup or silently runs on CPU. `13.0` only for a CUDA-13 image; omit for CPU. Rule, exceptions and capacity check: [`runpod-usage` gpu-selection "pin the CUDA floor"](../runpod-usage/reference/gpu-selection.md#step-3-pin-the-cuda-floor).
- **Waiting for a resource to be usable: use `--wait`, don't hand-roll a poll loop** (v2.9.0+). `create` returns once the resource is *scheduled*, which is why a "RUNNING" pod often refuses ssh and a fresh endpoint 404s. `pod create --wait` returns when port 22 answers with an ssh banner. `pod create --wait-for running` (v2.15.0+) returns when the container is up, no ssh needed: use it when container-up is all the task needs, or for an image that does not start sshd (most cpu images, since cpu pods get no Runpod-managed ssh), where plain `--wait` would bill through the whole timeout and then fail; a cpu image that starts its own sshd still satisfies plain `--wait`, so keep that when the user needs ssh. `serverless create --wait` returns when `/health` reports a ready or running worker. On timeout or ctrl-c the resource is **kept** and its id is in the error object's `id` field: read it and clean up, don't assume nothing was created (a pod bills by the second; an endpoint with no running worker doesn't, but starts one on the first request).
- **A service on a pod** (Ollama, ComfyUI, a dev server): declare its `--ports` and `--env` at creation when possible (`runpodctl pod update` can change them on an existing pod), then follow the pod development loop in `runpod-usage` (`reference/pod-workflows.md`): SSH-exec the install, bind to `0.0.0.0`, poll the proxy URL until it answers.
- **SSH:** connection details from `runpodctl pod get <pod-id>` or `runpodctl ssh info <pod-id>`. runpodctl has **no interactive-shell command**: `ssh info` returns the connection command + key but does not connect, so run `ssh user@host "command"` yourself.
- Network volumes are location-sensitive: pick the data center from the GPU stock read (`datacenter list`, or `gpu list --include-unavailable`) before creating the volume, and create it once. If `pod create` is then refused for stock, re-read stock: retry once with the same volume when its data center shows stock; otherwise ask the user (wait, another GPU type there, the console's Deploy when available, or a move to another data center, which is the user's call). Never delete the volume to try another data center; on a move, name it in the hand-back (id, data center, storage cost) and delete it only when the user asks. Move data onto or off a volume with `network-volume upload` / `download` (v3.1.0+, resumable); `send` / `receive` for machine to machine.
- Clean up paid resources after tests: serverless endpoints, pods, and temporary volumes created for validation.
  - **Cost guard:** remove the pod when the work is done (`runpodctl pod remove <pod-id>`); a stopped pod still bills for disk and any volume. Unattended deadline: [pod-workflows](../runpod-usage/reference/pod-workflows.md#2-provision) (schedule a stop from inside the pod).
  - **`--stop-after` / `--terminate-after` never worked.** Before v2.12.0 `pod create` accepts both, but the backend never enforced them, so the pod keeps running and billing past the deadline; v2.12.0 removed them. If `runpodctl pod create --help` still lists them, run `runpodctl update`. Never offer them as a cost guard.
  - **Attached volume:** to delete a network volume, remove the pod using it first.

### Serverless facts (context, not rules)

- **Scale-to-zero billing:** endpoints scale to zero with `--workers-min 0` (the default): no GPU billing while idle, only per request-second; the right cost posture for a request/response API.
- **Broken-image tell:** deployed workers go `ready` but jobs sit `IN_QUEUE` with `inProgress: 0` → the image is broken/mis-dispatching; switch to a different worker rather than wait it out.
- **Diagnosing it:** worker/job counts with `runpodctl serverless health <endpoint-id>` (v2.9.0+), then what the workers printed with `runpodctl serverless logs <endpoint-id>` (v2.10.0+); no hand-built curl needed. Repeated `system` "start container" lines with no `container` output = the container exits before the handler runs.

## Commands

Essentials only; flags come from `--help`. [reference/command-reference.md](reference/command-reference.md) holds what `--help` cannot answer: what a flag *means* when it succeeds, which field to trust, and what a failure looks like.

### Pods

```bash
runpodctl pod list                                   # running pods (+ --all / --status / --since / --created-after)
runpodctl pod get <pod-id>                           # details incl. SSH info + runtimeStatus
runpodctl pod create --template-id <id> --gpu-id "NVIDIA GeForce RTX 4090" --min-cuda-version 12.8   # from template (or --image <img>)
runpodctl pod create --compute-type cpu --image ubuntu:22.04                 # CPU pod (2 vCPUs; name defaults to the image)
runpodctl pod create --image <img> --gpu-id <id> --wait                      # block until ssh answers, then print the pod (v2.9.0+)
runpodctl pod create --compute-type cpu --image ubuntu:22.04 --wait-for running  # block until the container is up; no sshd needed (v2.15.0+)
runpodctl pod {start|stop|restart|update|delete} <pod-id>                    # lifecycle (delete aliases: rm/remove)
runpodctl pod logs <pod-id> [--follow] [--since 30m --source system]          # recent container+system logs, json lines (v2.10.0+); --follow keeps streaming and reconnects on its own; --source system = platform view: image pull / create / start
```

runpodctl v3.0.0 removed `pod reset`; use `pod restart` only when a restart is intended.

**A stalled deploy shows up in `--source system`** (v2.10.0+): repeated pull progress, or a `create container` that never reaches `start`. `--source container` is your workload's own output. Each line is one `{source,line,ts}` object; pipe it to `jq`.

**Read `runtimeStatus`, not `desiredStatus`, to decide whether a pod is usable** (v2.9.0+): `desiredStatus: RUNNING` says that while the image is still pulling. Field meanings, reason tokens, and two edges (`--status` filters `desiredStatus` only; `unknown` = lookup failed, not pod down) → [reference/command-reference.md](reference/command-reference.md#pod-status-fields).

### Hub

Browse/search existing Runpod Hub listings; pick the execution lane with the [router capability matrix](../runpod/SKILL.md#runpod-mcp-vs-runpodctl-the-overlap). Authoring a project's Hub listing files: [runpod-build-template](../runpod-build-template/SKILL.md).

```bash
runpodctl hub search vllm                            # find a repo (+ hub list [--type/--category/--order-by/--owner])
runpodctl hub get <listing-id|owner/name>            # repo details
```

### Serverless (alias: sls)

```bash
runpodctl serverless list | get <endpoint-id> | delete <endpoint-id>
runpodctl serverless create --name "x" --template-id <id> --min-cuda-version 12.8   # from template
runpodctl serverless create --name "x" --hub-id <listing-id>    # from hub (+ --env KEY=VAL to override defaults)
runpodctl serverless create --hub-id <id> --gpu-id "NVIDIA GeForce RTX 4090" \
  --model-reference https://huggingface.co/<org>/<model>:main   # attach & host-cache a HF model (GPU only)
runpodctl serverless update <endpoint-id> --workers-max 5
runpodctl serverless create --template-id <id> --workers-min 1 --wait          # block until a worker is ready (v2.9.0+)
```

**Invoke URLs come back with the endpoint:** `create`/`get`/`list`/`update` include a `urls` object (`run`, `runsync`, `health`), so read them instead of assembling the URL. They're built from `RUNPOD_INVOKE_URL` (default `https://api.runpod.ai/v2`), which `RUNPOD_API_URL`/`RUNPOD_GRAPHQL_URL` do **not** move: [reference/output-and-errors.md](reference/output-and-errors.md#serverless-invoke-urls).

**Reading worker logs** (v2.10.0+), first-class, so worker output needs neither the MCP lane nor a hand-built SSE read:

```bash
runpodctl serverless logs <endpoint-id>                       # every worker's recent logs, json lines, each tagged with its workerId
runpodctl serverless logs <endpoint-id> --worker <worker-id>  # just one worker
runpodctl serverless logs <endpoint-id> --follow              # picks up workers that scale up mid-follow
runpodctl serverless logs <endpoint-id> --since 1h --source system   # why a worker will not start
```

The crash-loop tell is under [Serverless facts](#serverless-facts-context-not-rules).

**Invoking an endpoint** (v2.9.0+), first-class, so an agent does not hand-build a curl request or manage a bearer token:

```bash
runpodctl serverless run <endpoint-id> --input '{"prompt":"hello"}'   # submit and wait for the result
runpodctl serverless run <endpoint-id> --input-file payload.json      # same, payload from a file ("-" = stdin)
runpodctl serverless run <endpoint-id> --input '{}' --wait 15m        # longer budget (default 5m)
runpodctl serverless run <endpoint-id> --input '{}' --no-wait         # submit, print the queued job, exit 0
runpodctl serverless status <endpoint-id> <job-id>                    # check a job submitted earlier
runpodctl serverless health <endpoint-id>                             # worker + job counts
```

- **Pass only the handler payload:** the cli wraps it as `{"input": <your json>}` itself, so pasting a whole curl envelope double-wraps it.
- **`timeout` means the cli stopped waiting, not that the endpoint broke.** When the message names a `serverless status` command the job is still running server-side: poll it, do **not** re-invoke (that buys a second job).

Payload rules, the stdout/stderr split, exit codes and why `/runsync` is never used → [reference/command-reference.md](reference/command-reference.md#invoking-an-endpoint-serverless-run-v290).

**Create from hub:** `--hub-id` resolves the hub listing, extracts the build image and config (GPU IDs, container disk, env vars), creates an inline template, and deploys. Accepts both SERVERLESS and POD listing types. The hub config's GPU IDs and env defaults are included automatically; override with `--gpu-id` and `--env`.

**CPU serverless endpoints** (rule in Decision Rules): `runpodctl serverless create --compute-type CPU` (optionally `--instance-id`, e.g. `cpu3g-4-16`). The 2026-07-14 evidence against REST v1, and why the invoke API is unaffected: [reference/command-reference.md](reference/command-reference.md#cpu-endpoints-and-rest-v1).

**Model cache (`--model-reference`, GPU only, runpodctl v2.4.0+):** attach a Hugging Face model by full URL with a ref (`…:main`); Runpod caches it host-side in the standard HF cache dir (`/runpod-volume/huggingface-cache/hub/`), so the worker loads it directly: no bake, no volume. Works with `--template-id` and `--hub-id`. Example, the one-cached-model limit, and how it compares to baking / network volume / the Model Repository: **[reference/model-caching.md](reference/model-caching.md)**. Worked end-to-end: golden path [20 — model-caching endpoint](../runpod/golden-paths/20-model-caching-endpoint.md).

**Multi-region / HA:** `--network-volume-ids <v1>,<v2> --data-center-ids <dc1>,<dc2>` (one volume per DC; **runpodctl ≥ v2.4.0**, and the Homebrew tap can lag). Data does **not** sync between volumes: golden path [10](../runpod/golden-paths/10-multi-region-ha-serverless.md). Model-reference output on `update` and the `--clear-models` shape: [reference/command-reference.md](reference/command-reference.md#endpoint-update-and-zero-values).

### Templates (alias: tpl)

```bash
runpodctl template search <q>                        # find (+ template list [--type official/community/user, --all, --limit])
runpodctl template get <template-id>                 # details (README, env, ports)
runpodctl template create --name "x" --image "img" [--serverless]
runpodctl template delete <template-id>
```

### Network Volumes (alias: nv)

```bash
runpodctl network-volume list | get <volume-id> | delete <volume-id>
runpodctl network-volume create --name "x" --size 100 --data-center-id "US-GA-1"
runpodctl network-volume update <volume-id> --name "new"
runpodctl network-volume upload <volume-id> <local-path> [remote-path]    # v3.1.0+
runpodctl network-volume download <volume-id> <remote-path> [local-path]  # "/" = whole volume
```

Exact flags: `runpodctl network-volume <action> --help`.

**Upload/download** use the volume's S3 API, so they need **S3 API keys** (`RUNPOD_S3_ACCESS_KEY` / `RUNPOD_S3_SECRET_KEY`), separate from the API key, and a [data center with the S3 API](https://docs.runpod.io/storage/s3-api). On `transfer_incomplete`, rerun the same command; resume rules, `--force` and partial files: [command-reference File Transfer](reference/command-reference.md#file-transfer).

> **No storage-tier flag.** `create` provisions the data center's **default** tier; there's no `--type`. For a **High-Performance** volume use the console (a ⚡ data center's toggle), a raw **v2 REST** call (`POST https://v2-rest.runpod.io/v2/network-volumes` with `"type":"HIGH_PERFORMANCE"`), or the MCP `create-network-volume` tool's `volumeType` (`STANDARD` | `HIGH_PERFORMANCE`). Tier is immutable after creation. Launch details: golden path [21](../runpod/golden-paths/21-storage-tiers.md).

### Models (Model Repository)

`runpodctl model` manages the **Runpod Model Repository**: managed, versioned storage for your **own** model artifacts (upload once, distributed to workers; not pinned to a data center like a network volume). What it is, why/how, migrating off a baked-in model, and Model-Repo-vs-volume: **[reference/model-caching.md](reference/model-caching.md)**.

```bash
runpodctl model list [--all] [--name "llama"] [--provider "meta"]   # yours by default; --all = not just yours
runpodctl model add --name "my-model" --model-path ./model   # upload a local model dir (multipart)
runpodctl model remove --name "my-model" --owner <owner>
```

`model add` also supports upload sessions, versioning, metadata, and private-source credentials: see live `runpodctl model add --help`.

### Info & SSH

```bash
runpodctl user                                       # account info + balance (alias: me)
runpodctl gpu list                                   # available GPUs + $/hr + per-DC stock (+ --include-unavailable)
runpodctl datacenter list                            # datacenters (alias: dc)
runpodctl ssh info <pod-id>                          # SSH connection details (command + key; NOT an interactive session)
```

**`gpu list` carries pod on-demand pricing and a `dataCenterAvailability[]` breakdown.** When a create must land in a specific DC, read the breakdown (top-level `stockStatus` is only the best across DCs) and pass `--include-unavailable` (the default listing hides no-stock GPUs and can omit one that has stock only in your DC). Shape, stock vocabulary, `null` prices and the `"none"` vs omitted-key sentinel: [reference/output-and-errors.md](reference/output-and-errors.md#gpu-pricing-and-per-data-center-availability).

Registry auth, `billing` and SSH key management: [reference/command-reference.md](reference/command-reference.md). Host key pinning (which versions pin, and why **`ssh forget` needs the operator's go-ahead**): [Pod host key trust](reference/command-reference.md#pod-host-key-trust-v2150).

### File Transfer

```bash
runpodctl send <path>                                # prints a one-time code, then blocks until the receiver connects
runpodctl receive <code>                             # positional code (no --code flag)
```

Encrypted/incremental/compressed: don't pre-tar. **Key gotchas:** capture the **first line of `send` stdout** (the code) as it streams (background + tee); each `send` mints a **fresh** code; both sides must exit `0`. Full agent flow (pod push via `ssh` + `receive`): [reference/command-reference.md](reference/command-reference.md#file-transfer).

### Utilities

`runpodctl doctor` (diagnose and fix CLI issues), `update`, `version`, `completion` (auto-detects the shell and installs completion).

## URLs

- **Pod:** `https://<pod-id>-<port>.proxy.runpod.net` for an exposed port (e.g. `https://abc123xyz-8888.proxy.runpod.net`).
- **Serverless:** prefer `runpodctl serverless run|status|health` (same api, with auth, validation and bounded polling handled) and the `urls` object the endpoint commands return. Use raw urls only for what the commands don't cover: streaming, the OpenAI-compatible route, or a copy-paste `curl` for a user. Only `status/<job-id>` has to be built by hand.

```
https://api.runpod.ai/v2/<endpoint-id>/run        # Async request
https://api.runpod.ai/v2/<endpoint-id>/runsync    # Sync request
https://api.runpod.ai/v2/<endpoint-id>/health     # Health check
https://api.runpod.ai/v2/<endpoint-id>/status/<job-id>  # Job status
```

## Source & docs

- CLI source: https://github.com/runpod/runpodctl
- Releases (binaries): https://github.com/runpod/runpodctl/releases
- Docs: https://docs.runpod.io/runpodctl/overview