evals/cpu-pod-create.eval.md
# Create a CPU-only pod
## Prompt
Create a CPU-only pod for lightweight file preprocessing using the image
`ubuntu:22.04`. Give me the exact command.
## Expected behavior
The agent should:
1. Use `runpodctl pod create` with `--compute-type cpu`
2. Pass `--image ubuntu:22.04`
3. NOT pass any GPU flags (`--gpu-id`, `--gpu-count`) — they don't belong on a CPU pod
## Assertions
- Runs `runpodctl pod create --compute-type cpu --image ubuntu:22.04`
- Does NOT include `--gpu-id` or `--gpu-count`
evals/create-and-wait-until-usable.eval.md
# Create a pod and wait until it is actually usable
## Prompt
Spin me up an A40 pod from `runpod/pytorch:1.0.3-cu1281-torch291-ubuntu2404` and
tell me when I can ssh into it.
## Expected behavior
The agent should:
1. Create with `runpodctl pod create --image ... --gpu-id "NVIDIA A40" --wait`, which returns
only once port 22 answers with an ssh banner.
2. Read the printed payload (the `pod get` shape, so it carries the live `ssh` block) and give
the user the ssh command.
3. NOT write its own poll loop around `pod get`/`ssh info`, and NOT treat
`desiredStatus: RUNNING` as "ready" — that is true while the image is still pulling.
4. If the wait times out, recognise that the pod **was created and still bills**, take the id
from the error object's `id` field, and offer to delete it.
## Assertions
- Uses `--wait` (optionally with `--wait-timeout`) rather than a hand-rolled polling loop
- Does not claim the pod is ready based on `desiredStatus` alone; if it inspects status, it reads `runtimeStatus`
- Reports the ssh connection details from the waited-for output
- On `wait_timeout` / `wait_interrupted`, surfaces the pod id from the error object and proposes cleanup instead of assuming nothing was created
evals/error-handling.eval.md
# Handle a runpodctl failure without parsing message text
## Prompt
I'm scripting `runpodctl` in a loop. Write the failure handling: how do I tell a
missing endpoint from a bad API key from a blip I should retry? Here are two failures
I've seen:
```
{"error":"failed to get endpoint: endpoint not found","code":"not_found","status":404}
{"error":"failed to get template: template not found: tpl-abc","code":"not_found"}
```
## Expected behavior
The agent should:
1. Branch on `code`, not on `status` and not on substrings of `error`
2. Explain that the second failure has no `status` because GraphQL answers a missing
resource with HTTP 200 + null data, so `status === 404` misses it
3. Retry `network_error`, `rate_limited` and `server_error` with backoff, and say why
`cli_error` (e.g. a malformed `RUNPOD_API_URL`) must not be
4. Distinguish `no_credentials` (no key set → `RUNPOD_API_KEY` / `runpodctl doctor`) from
`unauthorized` (key present but wrong/expired) — neither is a retry
5. Treat `not_found` as server-side, not as a mistyped local path
6. Gate on a non-zero exit code and read errors from **stderr**, data from stdout — and
not treat a `warning:`/`note:` line on stderr as a failure
7. Include a default branch for an unrecognized `code`, because the vocabulary is what the
CLI generates rather than exhaustive (the API can pass its own code through lowercased)
8. Not retry `timeout` / `wait_timeout` / `wait_interrupted` / `job_failed` (v2.9.0+): each
means work or a resource outlived the CLI, so re-running the command buys a **second**
job or a second billed resource. Follow up instead — poll (`serverless status`, `pod get`)
or clean up via the error object's `id`
## Assertions
- Switches/branches on `code`
- Does NOT branch on `status` alone or on `error` text matching
- Retry set includes `network_error` and at least one of `rate_limited`/`server_error`
- The produced handler has a default/else branch treating an unknown `code` as fatal
- Does NOT retry `cli_error`, `usage_error`, `not_found`, `no_credentials`, or
`unauthorized`
- Does NOT retry `timeout`, `wait_timeout`, `wait_interrupted` or `job_failed`; treats them
as "poll or clean up", not as "run it again"
- Separates `no_credentials` from `unauthorized` in the auth handling
- Reads errors from stderr and does NOT expect error JSON on stdout
- Does NOT claim `status` is always present
- Does NOT tell the user to parse plaintext for `pod`/`serverless`/`template`/`model`
commands (on v2.10.0+ only the legacy `pod` paths and `project` still print plaintext —
`exec` joined the coded-JSON shape and now exits non-zero, so an assertion that names
`exec` as a plaintext surface is stale)
- Does NOT pass `--output` anything but `json`/`yaml`: on v2.10.0+ an unrecognized value is
a `usage_error` with exit 1, where v2.9.0 silently returned JSON
evals/hub-deploy-serverless.eval.md
# Deploy a vLLM serverless worker from the Runpod Hub
## Prompt
Deploy the vLLM serverless worker from the Runpod Hub. Use an available GPU, and
have it scale from 0 up to 2 workers. Give me the exact command(s).
## Expected behavior
The agent should:
1. Find the hub listing id with `runpodctl hub search vllm`
2. Create the endpoint with `runpodctl serverless create --hub-id <id> --workers-min 0 --workers-max 2`
3. Handle the GPU correctly (this is the easy thing to get wrong):
- Preferably omit `--gpu-id` and let the hub config's default GPU apply, OR
- If specifying `--gpu-id`, use a GPU **pool ID** (e.g. `AMPERE_48`, `ADA_24`, `HOPPER_141`) — NOT a display name like `"NVIDIA A40"` from `runpodctl gpu list`. On the `--hub-id` path the API rejects display names with `Invalid GPU Pool ID`.
## Assertions
- Finds the hub id via `runpodctl hub search vllm` (does not invent one)
- Runs `runpodctl serverless create --hub-id <id> ...`
- Sets `--workers-min 0` and `--workers-max 2`
- If `--gpu-id` is passed at all, its value is a GPU pool ID (e.g. `AMPERE_48`), NOT a `gpu list` display name like `"NVIDIA A40"`
- Does NOT pass a `gpu list` display name to `--gpu-id` on the hub path
## Notes
This encodes the gotcha found via live testing and tracked upstream as
runpod/runpodctl#287: `serverless create --gpu-id` on the `--hub-id` path requires
GPU pool IDs, while `gpu list` (and the `--help` text) surface display names. Until
that is reconciled, the safe answer is to omit `--gpu-id` or use a pool ID.
evals/image-to-template-to-serverless.eval.md
# Run a custom image as a serverless endpoint (template first)
## Prompt
I have my own custom Docker image `myrepo/infer:latest`. I want to run it as a
Runpod serverless endpoint that scales from 0 to 3 workers on an A40 GPU. Give me
the exact commands.
## Expected behavior
The agent should:
1. Recognize that serverless endpoints are created from a `--template-id` or `--hub-id`, NOT directly from a raw image
2. First create a serverless template: `runpodctl template create --name ... --image myrepo/infer:latest --serverless`
3. Then create the endpoint from that template id: `runpodctl serverless create --template-id <id> --workers-min 0 --workers-max 3 ...`
4. Order the two steps correctly (template before endpoint)
## Assertions
- Step 1 creates a template with `runpodctl template create --image myrepo/infer:latest --serverless`
- Step 2 creates the endpoint with `runpodctl serverless create --template-id <id-from-step-1>`
- Sets `--workers-min 0` and `--workers-max 3`
- Does NOT attempt `runpodctl serverless create --image ...` (no `--image` flag exists on serverless create)
## Notes
The template-id path is more lenient about `--gpu-id` than the hub path (it accepts
display names like `"NVIDIA A40"`), but see hub-deploy-serverless.eval.md and
runpod/runpodctl#287 for the pool-id inconsistency.
evals/invoke-urls-and-gpu-pricing.eval.md
# Read invoke URLs and GPU cost from command output
## Prompt
I'm on a shell-only box with `runpodctl` and no MCP tools. I want a serverless endpoint
from template `tpl-abc` on the cheapest GPU with at least 24 GB that can actually schedule
in `US-KS-2`, and then the URL to send a job to. Give me the exact commands and tell me
where each number comes from — don't create anything yet.
## Expected behavior
The agent should:
1. Run `runpodctl gpu list --include-unavailable` and filter on `memoryInGb >= 24`,
noting that the default listing hides no-stock GPUs and could omit one that has stock
only in `US-KS-2`
2. Compare `securePricePerHr` / `communityPricePerHr` rather than assuming a lower tier is
cheaper, and treat a `null` price as "not offered on that cloud", not as free
3. Check `dataCenterAvailability[]` for a `US-KS-2` entry with real stock instead of
trusting top-level `stockStatus` (the best status across all DCs), reading `"none"` as
"offered here, no stock"
4. Pin placement with `--data-center-ids US-KS-2` — reading per-DC availability without
constraining the create leaves the placement to chance
5. Say that `--gpu-id` on serverless resolves to a GPU **pool**, so the endpoint may run on
more than the single card that was priced, and/or that the listed prices are pod
on-demand rates while serverless bills per request-second
6. State that the run URL comes from the `urls` object in the create output (`run`,
`runsync`, `health`) rather than hand-assembling `https://api.runpod.ai/v2/<id>/run`
## Assertions
- Runs `runpodctl gpu list` before choosing a GPU
- Cites `securePricePerHr` / `communityPricePerHr` for the cost comparison
- Consults `dataCenterAvailability[]` for `US-KS-2`
- Command includes `--data-center-ids US-KS-2`
- Says the run URL will be read from the `urls` object rather than assembled by hand
- Flags at least one of: the type→pool translation, or that the prices are pod rates
- Does NOT claim the CLI can't report GPU pricing
- Does NOT present `securePricePerHr` as the endpoint's serverless cost
evals/pod-auto-terminate.eval.md
# Create a pod that auto-terminates at a datetime
## Prompt
Create a GPU pod from the Docker image `myorg/trainer:latest` that automatically
terminates itself at 2026-07-01T00:00:00Z. Give me the exact command.
## Expected behavior
The agent should:
1. Use `runpodctl pod create` with `--image myorg/trainer:latest`
2. Use the `--terminate-after` flag with the given datetime
3. Choose `--terminate-after` (deletes the pod) over `--stop-after` (only stops it), since the user asked for termination
4. Recognize this flag exists rather than declaring it impossible
## Assertions
- Runs `runpodctl pod create --image myorg/trainer:latest ...`
- Uses `--terminate-after 2026-07-01T00:00:00Z`
- Does NOT use `--stop-after` for this request
- Does NOT claim auto-termination is unsupported
evals/pod-from-template-with-volume.eval.md
# Create a pod from a template with a network volume
## Prompt
Create a GPU pod named `trainer` from template id `tmpl-1`, and attach the network
volume with id `nv-9`. Give me the exact command.
## Expected behavior
The agent should:
1. Use `runpodctl pod create` with `--template-id tmpl-1`
2. Set `--name trainer`
3. Attach the volume with `--network-volume-id nv-9`
4. Not need any extra GPU flag, since GPU is the default compute type
## Assertions
- Runs `runpodctl pod create ...`
- Uses `--template-id tmpl-1`
- Uses `--name trainer`
- Uses `--network-volume-id nv-9`
evals/pod-ssh-connect.eval.md
# Get SSH connection details for a running pod
## Prompt
I have a running pod with id `pod-xyz`. I want to SSH into it to debug a process
interactively. What runpodctl command(s) should I use to get connected?
## Expected behavior
The agent should:
1. Retrieve connection details with `runpodctl ssh info pod-xyz` (or `runpodctl pod get pod-xyz`)
2. Connect using the SSH command/key those return
3. NOT use any deprecated interactive SSH subcommand to open the session
## Assertions
- Uses `runpodctl ssh info pod-xyz` or `runpodctl pod get pod-xyz` to obtain host/port/key
- Does NOT rely on a deprecated interactive `runpodctl ssh`/`exec` session command
- Final guidance results in a usable `ssh ...` connection to the pod
evals/registry-password-stdin.eval.md
# Create a registry auth without exposing its token
## Prompt
I have a private registry token in `REGISTRY_TOKEN`. Create a Runpod registry auth
named `ghcr` for username `octocat`, but do not put the token in command arguments
or shell history.
## Expected behavior
The agent should:
1. Use `runpodctl registry create` with `--password-stdin`
2. Pipe the environment variable through stdin without expanding its value into the
runpodctl argument list
3. Keep `--name ghcr` and `--username octocat` on the command
4. Avoid `--password`, because that exposes the credential through `argv`
5. Recognize that omitting both password flags is appropriate only for a human at an
interactive no-echo prompt, not for an automated command
## Assertions
- Uses `runpodctl registry create --name ghcr --username octocat --password-stdin`
- Pipes `REGISTRY_TOKEN` into stdin, for example with `printenv REGISTRY_TOKEN`
- Does NOT use `--password "$REGISTRY_TOKEN"` or place the token value in any argument
- Does NOT tell an automated caller to rely on the interactive prompt
- Does NOT write the token to a temporary file
evals/serverless-autoscale-by-requests.eval.md
# Update a serverless endpoint to autoscale by pending requests
## Prompt
Update my serverless endpoint `ep-abc123` so it autoscales based on the number of
pending requests, triggering when there are 4 pending. Give me the exact command.
## Expected behavior
The agent should:
1. Identify that `runpodctl serverless update <endpoint-id>` is the right command
2. Use the v2.3 autoscaler flags `--scale-by requests` and `--scale-threshold 4`
3. NOT use the older `--scaler-type` / `--scaler-value` flags (removed in v2.3) or values like `REQUEST_COUNT` / `QUEUE_DELAY`
## Assertions
- Runs `runpodctl serverless update ep-abc123 ...`
- Sets `--scale-by requests` (strategy = pending request count)
- Sets `--scale-threshold 4`
- Does NOT use `--scaler-type`, `--scaler-value`, `REQUEST_COUNT`, or `QUEUE_DELAY`
evals/serverless-invoke-job.eval.md
# Invoke a serverless endpoint and get the result
## Prompt
I have a serverless endpoint `ep-abc123` running a vLLM worker. Send it
`{"prompt": "why is the sky blue?"}` and show me what it returns.
## Expected behavior
The agent should:
1. Invoke with `runpodctl serverless run ep-abc123 --input '{"prompt":"why is the sky blue?"}'`,
which submits the job and polls until it is terminal.
2. Pass only the **handler** payload — the cli wraps it as `{"input": ...}` itself.
3. Read the job payload off stdout (it is printed even when the job fails) and report the
worker's output or its `error`.
4. If the wait budget runs out with a `timeout` code whose message names a
`serverless status` command, poll that command rather than re-invoking.
## Assertions
- Uses `runpodctl serverless run ep-abc123` with `--input` (or `--input-file`)
- Does NOT hand-build a `curl` to `https://api.runpod.ai/v2/ep-abc123/run` or `/runsync`, and does NOT construct an `Authorization: Bearer` header by hand
- Does NOT double-wrap the payload as `{"input":{"prompt":...}}`
- On a `timeout`, polls with `runpodctl serverless status ep-abc123 <job-id>` instead of re-submitting the job
- Treats a `job_failed` exit as the worker's failure (payload on stdout), not as a cli/auth problem
reference/command-reference.md
# runpodctl — behavior reference
**This file does not list flags.** `runpodctl <resource> <action> --help` does, it is always
current, and it ships with the binary in front of you:
```bash
runpodctl --help # resources
runpodctl <resource> --help # actions + aliases
runpodctl <resource> <action> --help # exact flags, defaults, examples
runpodctl help <resource> <action> # same, and traverses aliases (`pod remove`) that --help does not
runpodctl version # which surface you actually have
```
What lives here instead is the part `--help` never tells you: **what a flag means when it
succeeds, what "ready" is defined as, which field to trust, and what a failure looks like.**
Output shapes, error codes and env vars are in
[output-and-errors.md](output-and-errors.md).
> Two `--help` gotchas: cobra does **not** traverse aliases for `--help`, so `pod remove --help`
> answers `unknown command` even though `pod remove <id>` works — use `runpodctl help pod remove`.
> And the deprecated `get pod` / `get cloud` paths are hidden from `--help` but still live.
## Pods
### Waiting for readiness (`--wait`, v2.9.0+)
| | detail |
| --- | --- |
| ready means | the pod's **public port 22** accepts a tcp connection *and* answers with an ssh protocol banner. No key, no handshake — it proves sshd is up, not that your key is installed. Port 22 merely appearing in `runtime.ports` is not enough: prod allocates that port even for images that run no sshd |
| timeout | `--wait-timeout` accepts `90s`, `10m`, `1h`, `2d`; default `10m` |
| output | progress on **stderr** every ~15s; stdout stays exactly one json object, in the `pod get` shape (so it includes the live `ssh` block, unlike a plain create) |
| on failure | the pod is **not** deleted — exit is non-zero, code `wait_timeout` (or `wait_interrupted` on ctrl-c), and the error object carries the pod id in `id` plus the delete command. A second ctrl-c always exits |
| refuses | `--ssh=false` (there would be nothing to wait for) |
| warns, still waits | `--compute-type CPU` (cpu pods are created over rest, which cannot request Runpod-managed ssh, so only an image that starts its own sshd becomes reachable) and `--cloud-type COMMUNITY` without `--public-ip` (community cloud only maps a public ssh port on a machine that has a public ip) |
### Pod status fields
`pod get` and `pod list` report both (v2.9.0+):
| field | meaning |
| --- | --- |
| `desiredStatus` | what the platform intends: `RUNNING`, `EXITED`. Says `RUNNING` while the image is still pulling |
| `runtimeStatus` | what is actually happening: `running`, `initializing` (no container reported yet — pull/create/boot), `stopped`, `terminated`, `unknown` (the runtime lookup failed or was not made — **not** "the pod is down") |
| `runtimeStatusReason` | stable token when there is more to say, e.g. `awaiting_container`, `stopped_by_user`, `stopped_by_runpod`, `terminated_outbid`, `runtime_unavailable` |
| `uptimeSeconds` | present only while the container is up; omitted otherwise (it used to be a constant `0`) |
| `lastStatusChange` | the backend's raw free-text note, carried so a phrasing the cli does not tokenise still reaches you |
| `networkVolumeId` / `networkVolume` | **v2.10.0+:** `pod get --include-network-volume` fills both — the id and the full volume object. On **v2.9.0 and earlier both were dropped on deserialization** and read back `null` even with the flag, so a pre-v2.10.0 binary cannot tell you whether a pod has a volume. Upstream reports the `networkVolumeId` also comes back *without* the flag now (it is free alongside the pod), but only the flagged path is covered by a test — pass `--include-network-volume` rather than relying on that |
`--status` filters **`desiredStatus` only** — `--status initializing` silently matches nothing.
### Reading pod logs (`pod logs`, v2.10.0+)
| | detail |
| --- | --- |
| output | **json lines**, one `{source,line,ts}` object per line — pipe straight to `jq`, no SSE frame parsing |
| `--source` | `container` (your workload's stdout/stderr), `system` (the platform narrating image pull, container create, start), or `both` (default) |
| termination | without `--follow` it replays history and **exits on its own** once lines stop arriving (`--max-wait`, default `5s`, bounds the wait). With `--follow` it streams until interrupted and reconnects itself if the connection drops |
| history | `--tail 0-5000` (default 100; `0` = live only), or `--since 30m|2h|7d|<rfc3339>`, which overrides `--tail` |
| what to read it for | a stalled deploy is a `system` story — repeated pull progress, or a `create container` that never reaches `start` |
## Serverless (alias: sls)
### Invoking an endpoint (`serverless run`, v2.9.0+)
| | detail |
| --- | --- |
| payload | the **handler** payload, sent as `{"input": <your json>}`. Must be a json object; parsed and size-checked locally (the api's `/run` body limit is 10 MiB), so quoting mistakes and oversized bodies fail as `usage_error` before the upload |
| `--input` vs `--input-file` | mutually exclusive; one is required. `-` reads stdin either way. A payload with its own top-level `input` key gets a warning — that is usually a whole curl envelope pasted in, which arrives double-wrapped |
| stdout | always the job payload, including a `FAILED` job's `error`, and the last payload seen when the wait ran out. Printed byte-faithfully (handler keys are not renamed or re-typed) |
| stderr | progress notes and the error object — never job data |
| exit codes | `0` when `COMPLETED`, and when `--no-wait`/`--wait 0` submitted successfully. `1` on request failure, on wait-budget exhaustion (`timeout`), or on `FAILED`/`CANCELLED`/`TIMED_OUT` (`job_failed`) |
| two budgets | `--wait` bounds the whole job; the shared `timeout` config key (30s) bounds one api call. A call inside a wait is clamped to what is left, never below 1s |
| `/run`, never `/runsync` | `/runsync` is not synchronous: the connection is released after ~90s with the job still running, no job id exists until it answers (so a slow response strands a billed, unpollable job), and a `sync-` job's result expires after 1 minute vs 30 for `/run` |
`serverless status <endpoint-id> <job-id>` polls a job submitted earlier; `serverless health
<endpoint-id>` returns worker + job counts.
### Reading worker logs (`serverless logs`, v2.10.0+)
Same flags and semantics as `pod logs` above, plus:
| | detail |
| --- | --- |
| output | `{source,line,ts,workerId}` — the extra `workerId` is what makes the no-`--worker` form usable |
| `--worker` | optional. **Omit it and the command resolves the endpoint's workers itself** and reads all of them at once, tagging each line. With `--follow` it also picks up workers that appear mid-run, so an endpoint scaling up does not need the command re-run |
| what to read it for | the crash loop: repeated `system` `start container` lines with **no** `container` output means the container exits before the handler runs, which leaves jobs sitting in the queue with nothing wrong with capacity |
### Endpoint update and zero values
`serverless update` sends **zero values** as of **v2.10.0** (`--workers-min 0`,
`--idle-timeout 0`, `--workers-max 0`, `--scaler-value 0`). On **v2.9.0 and earlier those were
silently dropped** from the request, so resetting a dev endpoint back to scale-to-zero looked
like it applied and the endpoint kept billing. On an older binary verify with `serverless get
<id>`, or `PATCH https://rest.runpod.io/v1/endpoints/<id>` with an explicit `{"workersMin":0}`.
`serverless update` has **no `--gpu-id` flag** — change an existing endpoint's GPU pool with
that same `PATCH` and `{"gpuTypeIds":[...]}`.
**Multi-DC** (`--network-volume-ids <v1>,<v2> --data-center-ids <dc1>,<dc2>`) needs
**runpodctl ≥ v2.4.0**; data does not sync between volumes automatically — golden path
[10](../../runpod/golden-paths/10-multi-region-ha-serverless.md).
⚠️ `serverless get --include-workers` returns raw v1 worker records **including
`RUNPOD_AI_API_KEY`/`RUNPOD_ENDPOINT_SECRET` in `env`** — don't paste its output into a ticket.
### Error codes worth branching on
`timeout`, `job_failed`, `wait_timeout` and `wait_interrupted` are the codes these commands
add, and `not_found` gains a nuance during a wait. They live with every other code in
[output-and-errors.md](output-and-errors.md#codes) — including which are safe to retry, which
mean work outlived the cli, and the `id` field that names a resource a failed wait left
behind.
## Models
`model add` supports upload sessions, versioning, metadata, and private-source credentials.
Concepts: [model-caching.md](model-caching.md); flags: `runpodctl model add --help`.
## Registry credentials
Prefer `registry create --password-stdin` (needs **v2.11.0+**) for scripts so the credential does not enter the
process table or shell history. A secret already held in an environment variable can be piped
without expanding its value into the runpodctl argument list:
```bash
printenv REGISTRY_TOKEN | runpodctl registry create --name "x" --username "u" --password-stdin
```
Redirecting a credential file to `--password-stdin` also works, including multi-line
credentials such as a GCR service-account JSON key. The command strips one trailing line
ending but preserves leading, inner, and other trailing whitespace.
`--password` remains supported for compatibility, but places the credential in `argv`.
When neither password flag is present and stdin is a terminal, runpodctl prompts without
echo. With non-terminal stdin and no password source it returns `usage_error` instead of
blocking; `--password` and `--password-stdin` are mutually exclusive.
## SSH
`ssh info <pod-id>` returns **connection details, not an interactive session** — and it has
three output shapes, only one of which is an error. That table is in
[output-and-errors.md](output-and-errors.md#parsing-ssh-info); read it before writing a
readiness loop. If interactive SSH isn't available, execute remotely via
`ssh user@host "command"`.
`ssh remove-key` takes `--name` **or** `--fingerprint`; use the fingerprint to disambiguate
keys that share a name.
## File Transfer
`send`/`receive` do encrypted, incremental, compressed transfer — don't pre-tar or
pre-compress the source. **Agent flow (one side sends, the other receives):**
1. Run `send <path>` **without** a code. The **first line of stdout is the one-time code**;
`send` then blocks until the receiver connects — so capture that first line as it streams
(background the process, tee to a log) rather than waiting for exit.
2. On the other machine (use `runpodctl ssh` into the pod/host if needed) run `receive <code>`
with that exact code — positional, there is no `--code` flag. Each `send` mints a **fresh**
code — never reuse or invent one.
3. Both processes must exit `0`. On failure, re-run `send` and use its **new** first-line code.
To push local files to a pod: get `ssh info <pod-id>`, start `send` locally (capture the
code), then `ssh` to the pod and run `receive <code>` there. For large/library-style data, a
network volume or the S3 API is often simpler than `send`/`receive`.
## Hub, templates, volumes, registry, info, utilities
Plain CRUD plus filters — `--help` is complete and this file would only go stale restating it:
```bash
runpodctl hub --help # list/search/get; --type, --category, --owner, --order-by
runpodctl template --help # list/search/get/create/update/delete; --type official|community|user
runpodctl network-volume --help # list/get/create/update/delete
runpodctl registry --help # list/get/create/delete
runpodctl gpu list --help # + $/hr per cloud and dataCenterAvailability[]
runpodctl datacenter list --help # alias: dc
runpodctl billing --help # pods / serverless / network-volume
runpodctl user --help # account + balance (alias: me)
runpodctl doctor # interactive: diagnose and fix cli issues
runpodctl completion # auto-detect shell and install completion
```
Which of these to reach for, and the traps that are not flags (Hub worker selection, CPU
endpoints, template-vs-image, cost guards) are in the [SKILL.md](../SKILL.md) decision rules.
reference/model-caching.md
# Getting a model to a serverless worker
Four ways to make model weights available to a Runpod serverless worker. Pick by
where the weights come from, how large/private they are, and how often they change.
| Method | How | Best when | Cold start |
|--------|-----|-----------|-----------|
| **Bake into image** | `COPY`/download during `docker build`, push to a registry | small/private weights; fully reproducible image | fast (in image) but bloats pulls |
| **HF model cache** (`--model-reference`) | point the endpoint at a HuggingFace model URL; Runpod caches it host-side | the model is already on HuggingFace (public/gated/private) | fastest — host-cached, no download billing |
| **Network volume** | pre-load weights onto a DC-pinned volume, mount it | large weights reused across workers; you manage the files | fast once populated; volume is pinned to one data center |
| **Model Repository** (`runpodctl model`) | upload your **own** artifacts to Runpod-managed, versioned storage | private/custom models not on HuggingFace, without image bloat or a DC-locked volume | managed + host-distributed |
Rule of thumb: on HuggingFace → **HF cache**; your own artifact → **Model Repository**
(or a network volume if you want to manage the filesystem yourself); need a fully
reproducible image or system libs baked in → **bake**.
## HF model cache — `--model-reference`
Attach a HuggingFace model to an endpoint by full URL with a ref; Runpod caches it on
the host so the worker loads it directly — no bake, no volume.
```bash
runpodctl serverless create --template-id <id> --gpu-id "NVIDIA GeForce RTX 4090" \
--model-reference https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct:main
```
- The trailing `:main` is the branch/tag/revision.
- Weights land in the standard HF cache dir `/runpod-volume/huggingface-cache/hub/`
(`models--{org}--{name}/snapshots/{hash}/`), so anything that reads the HF cache
(Transformers, vLLM, …) picks it up automatically.
- Repeatable — pass it multiple times to attach multiple models.
- Works with `--template-id` **and** `--hub-id`, but **GPU only** (`--compute-type GPU`).
- **Requires runpodctl v2.4.0+.** Check `runpodctl version`; the Homebrew tap can lag —
prefer the [GitHub releases](https://github.com/runpod/runpodctl/releases) binary.
- Gated/private HF models: provide an `HF_TOKEN` (endpoint env var).
You are **not billed for download time** with the cache, and cold starts drop to
seconds because workers start on hosts that already hold the model.
## Model Repository — `runpodctl model`
Runpod-managed **storage + registry for your own model artifacts**. Upload weights once;
Runpod stores, versions, and distributes them to workers — a first-class model object
with a name, versions, metadata, and a status lifecycle (not just a file on a disk).
```bash
runpodctl model list # list your models
runpodctl model list --all # include models you don't own
runpodctl model list --name "llama" # filter by name
runpodctl model list --provider "meta" # filter by provider
runpodctl model add --name "my-model" --model-path ./model # upload a local model directory
runpodctl model remove --name "my-model" --owner <owner> # remove a model
```
`model add` runs a **multipart upload session** (built for large weights) — the live
`runpodctl model add --help` exposes `--create-upload`, `--part-size`, `--file-size`,
`--file-name`, `--content-type`, `--metadata key=value`, `--model-status`,
`--version-status`, and `--credential-reference`/`--credential-type` (for pulling from a
private source). Run it before relying on exact syntax — it is authoritative.
**Coming from a baked-in model?** The easiest migration is to stop `COPY`-ing weights
into the image and instead `runpodctl model add --model-path <the same dir you used to
COPY>`, then reference the uploaded model from the endpoint. This shrinks the image and
lets you version the weights independently of the code.
**vs a network volume:** a network volume is a raw filesystem you manage and is **pinned
to one data center** (workers must run there); the Model Repository is managed, versioned,
and host-distributed, so it isn't locked to a single DC. Use a volume when you want direct
filesystem control or are already populating one; use the Model Repository for a
hands-off, versioned artifact.
> **Runtime-load path — don't guess.** The command surface above (`runpodctl model add`)
> is from live `runpodctl --help` and is accurate for *uploading*. But how a worker
> *references an uploaded Model Repository artifact at runtime* is **not publicly
> documented yet** — if asked for exact runtime-load steps for a `runpodctl model add`
> artifact, say it isn't publicly documented and don't invent a path.
>
> This is distinct from the HF **model-caching** feature (the `--model-reference` flag /
> the endpoint "Model" field), whose runtime path **is** documented: the model lands in
> `/runpod-volume/huggingface-cache/hub/` and the handler resolves the local snapshot from
> there (load offline with `HF_HUB_OFFLINE=1`). See golden path 20 and the official
> example `runpod-workers/model-store-cache-example`.
reference/output-and-errors.md
# runpodctl — output format, error codes, env vars
Everything an agent needs to parse runpodctl's output and decide what to do with a
failure. The 80% rule is in [SKILL.md](../SKILL.md#output--errors); this file is the
full surface.
**Version floor: runpodctl ≥ v2.8.0.** The coded error shape, the serverless `urls` object
and GPU pricing/per-DC availability all arrived in v2.8.0.
Older binaries (≤ v2.7.3) did **not** print plaintext — they printed
`{"error":"<message>"}` on stderr with no `code` and no `status`, and the message was
interpolated unescaped, so a message containing a quote produced malformed JSON. They also
dumped usage text after runtime errors. So the pre-v2.8.0 failure mode is *valid-looking
JSON with the field you're branching on missing* — **gate on `code` being present**, not on
whether stderr parses as JSON. `runpodctl version` is a weak gate: it prints plaintext, and
a binary built from source reports a placeholder version regardless of the code it contains.
## Output format
JSON is the **default** — the CLI is built for agent consumption, and data goes to
**stdout**.
```bash
runpodctl pod list # json (default)
runpodctl pod list --output=yaml # yaml — the only alternative
```
**There is no table format** — `json` and `yaml` are the only values. As of **v2.10.0** an
unrecognized value is a hard error before any API call, and matching is
**case-insensitive**:
```jsonc
// runpodctl gpu list --output=table (v2.10.0+)
{"error":"invalid --output \"table\": supported formats are json and yaml","code":"usage_error"}
// exit 1, usage text on stderr
```
```bash
runpodctl gpu list --output=YAML # v2.10.0+: real YAML (case is ignored)
```
⚠️ **This inverted in v2.10.0, so a handler has to know which binary it is on.** Through
**v2.9.0** the flag was matched *case-sensitively* and anything unrecognized fell back to
JSON **silently** — `--output=table` and `--output=YAML` both returned JSON, and the only
safe advice was "pass lowercase `yaml` or don't pass the flag". That silence is how a table
format the CLI never had ended up documented in the first place. On v2.10.0+ the same
`--output=table` exits **1** with `usage_error`, so code that passed a defensive
`--output=json` is fine but code passing anything else now fails loudly instead of
degrading.
(The legacy `get pod` / `get cloud` commands print a human table on stdout and ignore
`--output` entirely — see [plaintext gaps](#plaintext-gaps).) Some commands print a human table on stdout regardless — the legacy
`get pod`/`get cloud` paths do — so a table on stdout is a signal you're on a legacy
command, not a format you asked for.
## Error format
Errors go to **stderr** as a single flat JSON object, and the exit code is
**non-zero**, so stdout parses as data.
```jsonc
{"error":"failed to get endpoint: endpoint not found","code":"not_found","status":404}
```
| field | notes |
| --- | --- |
| `error` | human-readable message, unwrapped on the REST path. A GraphQL HTTP failure whose body has no extractable message falls back to the **raw body**, which can itself be JSON — so never parse this field, either way |
| `code` | stable, lowercase, present on every JSON error (see [plaintext gaps](#plaintext-gaps)) |
| `status` | HTTP status — only when the failure arrived on a **non-2xx** response (REST *or* GraphQL) |
| `id` | **v2.9.0+**, only on a failure that left a **created, billing resource** behind — a `create --wait` that timed out or was interrupted. Read it instead of regexing the message: it is the id you need to clean up |
**Branch on `code`, never on `status` or the message text.** `status` is absent whenever
the API answered HTTP 200 with empty data, which is how GraphQL reports a missing
resource — so `if (status === 404)` misses every GraphQL not-found:
```jsonc
// serverless get <missing> — REST, so a status is available
{"error":"failed to get endpoint: endpoint not found","code":"not_found","status":404}
// template get <missing> — GraphQL answered 200/null, so there is no wire status
{"error":"failed to get template: template not found: bogus-template-id","code":"not_found"}
```
`code` is both: a typed code (from the API or from the error itself) wins, and the
output sink fills in a fallback when there isn't one — so local validation and transport
failures carry a code too, not just API responses.
**`timeout` is the one documented exception**, because two outcomes share it and only the
message separates them (see the [codes table](#codes)). A handler that reads only `code` is
still correct: treat `timeout` as never-retry and poll instead. Reading the message can
upgrade that to "safe to retry" for the single-API-call case, but skipping it costs you
nothing except an unnecessary poll — whereas retrying on the wrong one buys a second job.
### Codes
| code | meaning |
| --- | --- |
| `usage_error` | unknown command, unknown flag, bad/wrong-count args, or a **cobra-required** flag left out. Usage text prints after the JSON |
| `not_found` | the **API** does not have the resource. During a `create --wait` it can also mean one that *was* created has gone, or never became visible — so check for an `id` field before concluding nothing exists |
| `bad_request` (400) `unauthorized` (401) `forbidden` (403) `conflict` (409) `rate_limited` (429) `server_error` (5xx) `api_error` | derived from the REST status |
| `graphql_error` | a GraphQL call failed: an `errors` array in a 200 body, **or** a non-2xx response from the GraphQL endpoint (that form also carries `status`) |
| `no_credentials` | no API key configured at all — set `RUNPOD_API_KEY` or run `runpodctl doctor` |
| `network_error` | the API could not be reached at all — DNS, connection refused, TLS, timeout |
| `cli_error` | anything else local: config, environment, and validation the command does itself |
| `timeout` | **v2.9.0+.** The CLI stopped waiting — two outcomes sharing one code, told apart by the message. If it names a `serverless status` command, the **job is still running server-side**: poll it, never re-invoke (that buys a second job). Otherwise a single API call ran out of time and nothing is running, so a retry is fine |
| `job_failed` | **v2.9.0+.** A serverless job reached a terminal status other than `COMPLETED` (`FAILED`/`CANCELLED`/`TIMED_OUT`). The request itself succeeded — the job payload, including the worker's own `error`, is still on **stdout** |
| `wait_timeout` `wait_interrupted` | **v2.9.0+.** A `create --wait` gave up (budget) or was cancelled (ctrl-c / SIGTERM). The resource **was created and still bills**; its id is in the `id` field above |
Treat this as **the set the CLI generates, not an exhaustive list** — an explicit code
from the API is passed through lowercased, so a code outside the table can arrive from
the server. **Give your handler a default branch that treats an unknown code as fatal**
and surfaces `error` verbatim; never fall through to a retry.
### What to retry
**runpodctl has no internal retry** — it never backs off for you, so whatever your
wrapper does is all that happens.
- **Retry with backoff:** `network_error`, `rate_limited`, `server_error` (and `api_error`
when `status` is 5xx). A transient `graphql_error` is possible too but indistinguishable
from a permanent one, so cap those attempts tightly.
- **Never retry:** `usage_error`, `cli_error`, `bad_request`, `not_found`, `conflict`,
`no_credentials`, `unauthorized`, `forbidden`, `job_failed` (the job ran and failed —
re-running is a new job, not a retry).
- **Never retry the *command*, but do follow up:** `timeout` and
`wait_timeout`/`wait_interrupted` all mean work or a resource outlived the CLI. Re-running
the same command creates a **second** job or a **second** billed resource. Poll instead
(`serverless status`, `pod get`), or clean up using the `id` field. The one exception is a
`timeout` whose message does **not** name a follow-up command: that was a single API call
timing out with nothing left running, so it is safe to retry.
Two invariants make that safe to encode:
- **`not_found` always means server-side.** A bad *local* path (say a mistyped
`--model-path`) is `cli_error`, so `not_found` never means "you typed a path wrong".
- **`network_error` is the only code the CLI assigns to mean "couldn't reach the API"**,
detected structurally. Deliberately *not* `network_error`: a malformed `RUNPOD_API_URL`
is `cli_error`, so a retry loop never fires for something a retry cannot fix. A local wait
loop timing out used to land here too — as of **v2.9.0** `model add --wait-for-hash`
reports `timeout` instead of `cli_error` (same exit code, same message text), so a handler
switching on `code` needs that branch.
### Auth failures are two different codes
`no_credentials` means **no key is configured**. A key that is present but wrong, expired
or revoked is `unauthorized` (401), and one lacking access to the resource is `forbidden`
(403):
```jsonc
// RUNPOD_API_KEY=rpa_bogus… pod list
{"error":"api request failed with status 401","code":"unauthorized","status":401}
```
Don't collapse these — re-prompting for a "missing" key when the real problem is a
revoked one sends an agent in a circle.
### Invocation mistakes split across two codes
`usage_error` covers what cobra validates: unknown command/flag, wrong arg count, and
missing `MarkFlagRequired` flags. Validation a command performs itself lands in
`cli_error`, even though it's just as much an invocation mistake:
```jsonc
// ssh remove-key with neither --name nor --fingerprint
{"error":"either --fingerprint or --name must be provided","code":"cli_error"}
```
So `cli_error` is a mixed bucket — "your invocation was wrong" *and* "your local
environment is wrong". Read `error` to tell them apart; don't retry either.
### Missing API key
The JSON-covered commands all report `no_credentials` — `pod` (`list`/`get`/`create`),
`template create`/`update`, every `ssh` subcommand, `model *`:
```jsonc
{"error":"api key not found. get your key at https://www.runpod.io/console/user/settings then: export RUNPOD_API_KEY=your-key OR run: runpodctl doctor","code":"no_credentials"}
```
```jsonc
{"error":"unknown flag: --nope","code":"usage_error"}
```
Usage text follows the JSON on stderr — and, unlike older builds, a *runtime* error no
longer dumps usage text after it.
### Exit codes, and what else is on stderr
Every failure on the JSON path exits non-zero (`model` and `update` used to print an error
and exit **0**; they don't anymore). The exceptions are in
[plaintext gaps](#plaintext-gaps) below.
**Non-empty stderr is not a failure signal.** Deprecation notices (`warning: 'runpodctl
get pod' is deprecated…`), `pod create` advisories (`note: …`) and config-migration lines
all go to stderr on successful runs. Gate on the exit code first, then parse stderr.
### Plaintext gaps
The JSON error shape covers `pod`, `serverless`, `template`, `volume`, `registry`,
`gpu`, `datacenter`, `billing`, `user`, `model`, `ssh`, `send`, `receive`, `hub` and
`update` — including every GraphQL call site (a GraphQL failure is `graphql_error`, or
`not_found` for the nil-data lookups). These surfaces still print plaintext and carry
no `code`:
| surface | shape |
| --- | --- |
| legacy `get`/`create`/`remove`/`start`/`stop pod`, `create`/`remove pods`, `get cloud` | `Error: <msg>` on stderr, exit 1 — including the missing-key case, which carries the same message as `no_credentials` but no JSON and no `code`. Success prints a human **table** on stdout, not JSON |
| `exec` (hidden, deprecated) | **v2.10.0+:** joined the JSON error shape — flat JSON with a `code` on stderr and a **non-zero** exit. Through v2.9.0 it printed the error as plaintext and still exited **0**, so a caller that trusted the exit code saw a silent success. Either way: progress on **stdout**, and it polls up to **5 minutes** for the pod's SSH info before giving up |
| `project` (hidden) | prints errors to **stdout** and exits **0** — the last surface where the exit code cannot be trusted |
### Parsing `ssh info`
`ssh info` has three shapes, and only one of them is an error:
| situation | output |
| --- | --- |
| connectable | `{"id","name","ssh_command","ip","port","ssh_key":{…}}` on stdout, exit 0 — note **snake_case**, unlike the camelCase everywhere else in the CLI. A `"setup":"runpodctl doctor"` key appears when the local key needs fixing |
| pod exists, not connectable | `{"error":"pod not ready: <reason>","id":…,"name":…,"status":…}` on **stdout**, exit **0**, no `code`. Here `status` is the pod's desired status, *not* an HTTP status. **v2.9.0+** appends a reason when it has one (image still pulling, port 22 not published, pod stopped, …) and falls back to the bare `"pod not ready"` when it doesn't — so an exact-match `=== "pod not ready"` started failing intermittently on that release; a prefix match did not |
| no such pod | `{"error":"pod 'x' not found","code":"not_found"}` on stderr, exit 1 |
So: check the exit code, then check for an `error` key even on stdout.
"Not ready" fires whenever the pod has no **public port 22**, which is not the same thing
as "still booting". A pod whose image never starts an sshd reports not-ready indefinitely —
verified by creating a `ubuntu:22.04` CPU pod and polling for ~70 s at `status: RUNNING`
throughout. So **bound any SSH readiness loop** and don't treat `RUNNING` as "SSH is
coming"; if it never arrives, the image is the problem, not the wait.
Two v2.9.0 changes make this easier to get right: `pod create --wait` does the bounded
readiness loop for you (and proves an ssh *banner*, not just a published port), and
`runtimeStatus` on `pod get`/`pod list` distinguishes `initializing` from `running` so you
no longer have to infer it from `desiredStatus`. Also **v2.9.0+**: a **stopped** pod no
longer returns an `ssh_command` at all (it used to hand back one built from stale runtime
ports that could never connect), and it drops out of `ssh connect`'s `connections` list,
which is now `[]` rather than `null` when nothing is reachable.
So a parser should tolerate a non-JSON line on stderr from those, and must not rely on
the exit code for `project` (nor for `exec` before v2.10.0). Prefer the non-legacy
equivalents: `pod get`/`pod create`/`pod delete`, and `ssh info <pod-id>` + your own `ssh`
invocation instead of `exec`.
## Environment variables
| variable | default | what it sets |
| --- | --- | --- |
| `RUNPOD_API_KEY` | — | API key. Also settable via `runpodctl doctor` or `~/.runpod/config.toml` (`apikey`) |
| `TIMEOUT` | `30s` | per-API-call deadline — **no `RUNPOD_` prefix** (the CLI reads env vars unprefixed, so this one looks nothing like the others). Exceeding it is `code: "timeout"`. Also settable as a top-level `timeout` in `~/.runpod/config.toml` — it must sit **above** any `[section]` header, or it becomes `section.timeout` and is silently ignored. Distinct from `--wait`, which bounds a whole job or readiness wait rather than one call |
| `RUNPOD_API_URL` | `https://rest.runpod.io/v1` | REST control plane (config key `restApiUrl`) |
| `RUNPOD_GRAPHQL_URL` | `https://api.runpod.io/graphql` | GraphQL control plane (config key `apiUrl`) |
| `RUNPOD_INVOKE_URL` | `https://api.runpod.ai/v2` | base for the invoke URLs reported by `serverless create`/`get`/`list`/`update` (config key `invokeUrl`) |
**Invoke is a separate service from the control plane.** Pointing `RUNPOD_API_URL` or
`RUNPOD_GRAPHQL_URL` at a non-prod host does *not* move the invoke URLs — override
`RUNPOD_INVOKE_URL` explicitly when you need that, or the emitted URLs target prod.
## Serverless invoke URLs
`serverless create`/`get`/`list`/`update` return a `urls` object, computed from the
endpoint ID, so a freshly created endpoint is callable without a second lookup:
```jsonc
"urls": {
"run": "https://api.runpod.ai/v2/<endpoint-id>/run",
"runsync": "https://api.runpod.ai/v2/<endpoint-id>/runsync",
"health": "https://api.runpod.ai/v2/<endpoint-id>/health"
}
```
`status` isn't in the object; it's `<run url minus /run>/status/<job-id>`.
## GPU pricing and per-data-center availability
`gpu list` reports on-demand price per hour per cloud type plus a per-DC stock
breakdown:
```jsonc
{
"gpuId": "NVIDIA A100 80GB PCIe",
"displayName": "A100 PCIe",
"memoryInGb": 80,
"secureCloud": true, "securePricePerHr": 1.39,
"communityCloud": true, "communityPricePerHr": 1.19,
"stockStatus": "Low", "available": true,
"dataCenterAvailability": [
{"dataCenterId": "CA-MTL-3", "stockStatus": "Low"},
{"dataCenterId": "EU-RO-1", "stockStatus": "none"}
]
}
```
- A price is **explicitly `null`** when that cloud type doesn't offer the GPU, which is
distinguishable from a real `0`. Read `securePricePerHr`/`communityPricePerHr` rather
than guessing that a lower tier is cheaper.
- Top-level `stockStatus` is the **best status across data centers** — use
`dataCenterAvailability[]` as ground truth for *where* a create will actually
schedule, especially when co-locating with a network volume.
- An unrecognized non-empty status now ranks above absent/`none` instead of tying with it,
so **`available` no longer reports `false`** for a GPU whose only status is a value the
CLI doesn't know. It still ranks *below* `Low`, so a known level still wins the top-level
`stockStatus` — that tradeoff is deliberate, and it's why `dataCenterAvailability[]` is
the field to read. Casing and surrounding whitespace are normalized now, so they are no
longer a source of unknown values.
- Two sentinels for one concept: the per-DC breakdown spells an unreported status
`"none"`, while the top-level field **omits the key** for the same condition. Handle
both.
- **Stock values are capitalized levels but the sentinel is lowercase** — `"High"`,
`"Medium"`, `"Low"`, `"none"` in practice. `"unavailable"`, `"out of stock"` and
`"no stock"` also count as no-stock, and comparison is case-insensitive internally, so
lowercase before testing and don't write `if s != "none"`.
- **`gpu list` hides no-stock GPUs by default.** Pass `--include-unavailable` when
targeting one data center, or you may never see a GPU that has stock only there.
Two scoping limits the fields don't express:
- **The prices are pod on-demand rates** (`gpuTypes { securePrice communityPrice }`). They
are not serverless rates — serverless bills per request-second — so don't present them
as the cost of an endpoint.
- **`dataCenterAvailability[]` has no cloud-type attribution**, while pricing is split
secure vs community. "Cheapest option that schedules in DC X" therefore isn't answerable
from `gpu list` alone; treat it as two separate reads.
And when you act on the choice: pin placement with `--data-center-ids`, or the analysis
doesn't constrain anything. For serverless, `--gpu-id` takes a GPU *type* id and is
translated server-side to a GPU **pool**, so the endpoint may run on more than the one
card you priced (pool vs type ids: [`runpod-usage` gpu-selection](../../runpod-usage/reference/gpu-selection.md)).
SKILL.md
---
name: runpodctl
description: >-
Runpod CLI for managing GPU/CPU workloads from the terminal — pods, serverless
endpoints, templates, network volumes, Hub deploys, models, SSH, and file
transfer (send/receive). Use for terminal/CI/scripting, Hub browse/deploy, SSH
setup, `doctor`, or when the Runpod MCP tools are not connected. For structured
tool calls in an MCP-enabled session, prefer runpod-mcp.
allowed-tools: Bash(runpodctl:*)
compatibility: Linux, macOS
metadata:
author: runpod
version: "1.2.0" # x-release-please-version
license: Apache-2.0
---
# Runpodctl
Manage GPU pods, serverless endpoints, templates, volumes, and models.
## Install
`curl -sSL https://cli.runpod.net | bash` (Linux/macOS, and Windows via WSL) or `brew install runpod/runpodctl/runpodctl`. Manual binaries and the Windows and conda steps live in the [runpodctl README](https://github.com/runpod/runpodctl#install), beside the `install.sh` they describe. The command surface comes from the binary — `runpodctl <resource> <action> --help` — or the generated pages under [`runpodctl/docs/`](https://github.com/runpod/runpodctl/tree/main/docs).
> Old runpodctl builds silently lack newer flags/behaviors (e.g. `--model-reference` doesn't
> exist before v2.4.0) and produce confusing downstream errors — and the Homebrew tap can lag
> well behind. So, before any work:
>
> - **Update to the latest build** — check `runpodctl version`, then run `runpodctl update`
> (or reinstall from the [latest release](https://github.com/runpod/runpodctl/releases)).
> - **Pin to one recent version for the whole task.**
> - **Never switch between an old and a new binary mid-task** (that flip-flop is a known failure).
> - **Verify once** — `runpodctl version` shows the current build before you continue.
## Quick start
```bash
runpodctl update # FIRST: get on the latest build — old versions cause confusing errors
runpodctl version # confirm the current version before doing any work
export RUNPOD_API_KEY=your_key # Non-interactive auth (agents) — runpodctl reads this
runpodctl doctor # Interactive first-time setup (API key + SSH) — for humans
runpodctl --help # See current top-level commands
runpodctl pod create --help # Inspect exact current flags before creating
runpodctl gpu list # See available GPU types
runpodctl datacenter list # GPU availability per data center (use to co-locate GPU + volume)
runpodctl hub search vllm # Find a hub repo
runpodctl serverless create --hub-id <id> --name "my-vllm" # Deploy from hub
runpodctl template search pytorch # Find a template
runpodctl pod create --template-id runpod-torch-v21 --gpu-id "NVIDIA GeForce RTX 4090" # Create from template
runpodctl pod list # List your pods
```
> Auth: an agent should `export RUNPOD_API_KEY=...` (non-interactive). `runpodctl
> doctor` is interactive (prompts) and also sets up SSH keys — good for a human's
> first run, not for scripted use.
API key: https://console.runpod.io/user/settings
## Live Help Is Authoritative
Live `runpodctl --help` output is authoritative for exact flags, aliases, and command syntax. Use this skill for workflows, decision rules, safety notes, and common examples.
```bash
runpodctl --help
runpodctl <resource> --help
runpodctl <resource> <action> --help
```
Before using unfamiliar commands, inspect live help first. Do not rely on this skill as an exhaustive flag reference.
**What live help does *not* cover:** output shapes, error codes, and exit-code behavior. `--help` lists flags; it never shows you what a failure looks like. For those, use [reference/output-and-errors.md](reference/output-and-errors.md) — and when in doubt, **probe the binary**: run the command wrong on purpose (`runpodctl serverless get nope`) and read the JSON it emits. Every doc is a snapshot, this skill included; the binary in front of you wins.
## Output & errors
Data is **JSON on stdout** (`--output=yaml` is the only alternative — there is no table
format; anything else silently returns JSON). A failure from the resource commands is a
single flat JSON object on **stderr** plus a **non-zero exit**:
```jsonc
{"error":"failed to get endpoint: endpoint not found","code":"not_found","status":404}
```
**Branch on `code`, never on `status` or the message.** `status` is there only when the
failure arrived on a non-2xx response — GraphQL reports a missing resource as HTTP 200 +
null data, so `if status == 404` misses every GraphQL not-found.
| `code` | what to do |
| --- | --- |
| `network_error` | **retry with backoff** — the only code meaning "couldn't reach the API" |
| `rate_limited` `server_error` | **retry with backoff** — 429/5xx from the API |
| `usage_error` `cli_error` `bad_request` `not_found` `conflict` | don't retry, fix the input |
| `no_credentials` | no key set: `export RUNPOD_API_KEY=…` or `runpodctl doctor` |
| `unauthorized` `forbidden` | a key **is** set but is wrong/expired or lacks access — don't retry, don't re-prompt for a missing key |
| anything else | treat as fatal, surface `error` verbatim — the API can pass through its own code |
**runpodctl never retries internally**; nothing backs off for you.
- **`not_found` always means the API lacks the resource**, never a mistyped local path
(that's `cli_error`).
- **`cli_error` is a mixed bucket:** local environment problems *and* invocation mistakes
the command validates itself (e.g. `ssh remove-key` with neither `--name` nor
`--fingerprint`). Only cobra-enforced required flags are `usage_error`.
- **`usage_error`** = unknown command/flag, bad args, missing cobra-required flag; usage
text follows the JSON. Runtime errors no longer print usage.
- **Non-empty stderr does not mean failure** — deprecation `warning:` and `note:` lines go
to stderr on success too. Gate on the exit code, then parse stderr.
Coded errors, the serverless `urls` object and GPU pricing all need **runpodctl ≥ v2.8.0**.
Older binaries emit `{"error":"…"}` with **no `code` and no `status`** — still JSON-shaped,
so a `switch (err.code)` silently gets `undefined` rather than failing loudly. **Gate on
`code` being present**, not on JSON-vs-plaintext; `runpodctl version` is unreliable for
this (plaintext, and a source build reports a placeholder version).
Full code table, the surfaces that still print plaintext (`exec`, legacy `pod`
commands, `project`), and the env-var table (incl. `RUNPOD_INVOKE_URL`):
**[reference/output-and-errors.md](reference/output-and-errors.md)**.
## Decision Rules
- Use Hub when the user wants a known deployable app or worker such as vLLM, ComfyUI, Whisper, or a Runpod-maintained repo.
- **Picking a worker:** prefer a **first-party or well-adopted, recently-released** worker on a **broad, high-availability GPU pool**. Observable signals via `runpodctl hub list`: `--owner runpod-workers` (first-party), `--order-by releasedAt`/`updatedAt` (recency), `--order-by deploys`/`stars` (adoption). Don't pin a scarce large-GPU tier a small model doesn't need.
- **"Active worker" = minimum workers, not maximum.** If a user asks for an "active worker," they mean `--workers-min 1` (keep one worker always warm → no cold start), **not** `--workers-max 1` (that only caps the ceiling). A warm min-1 worker is ideal for development/iteration.
- ⚠️ **A min-1 worker bills continuously, even while idle** (it defeats scale-to-zero). When you set `--workers-min 1` for dev, you **must** set it back to `--workers-min 0` (or delete the endpoint) when done — otherwise it quietly runs up cost.
- **Needs runpodctl ≥ v2.10.0.** On earlier binaries `--workers-min 0` and `--idle-timeout 0` were **silently dropped** from the update request (`omitempty` ate the zero), so the reset looked like it applied and the endpoint kept billing. Check `runpodctl version`; on an older binary confirm with `serverless get <id>` and fall back to `PATCH https://rest.runpod.io/v1/endpoints/<id>` with an explicit `{"workersMin":0}`.
- `serverless update` has **no `--gpu-id` flag**. To change an existing endpoint's GPU pool, call `PATCH https://rest.runpod.io/v1/endpoints/<id>` with `{"gpuTypeIds":[...]}` directly.
- **CPU serverless endpoints:** always create them with `runpodctl serverless create --compute-type CPU` — **not** the MCP server, whose v2 `create-endpoint` requires `gpuPoolIds` and has no CPU concept. **Never** use the public control REST `POST https://rest.runpod.io/v1/endpoints` with `"computeType":"CPU"` — it silently provisions a **GPU** endpoint instead (verified evidence in the Serverless command section below).
- Use templates when the user already has a template ID, wants reusable image/config defaults, or needs lower-level control than Hub.
- Use direct pod creation with `--image` when the user has a specific Docker image and does not need a saved template.
- Use serverless for request/response inference APIs and scalable workers; use pods for interactive work, notebooks, training, debugging, or long-lived sessions.
- Use CPU pods for preprocessing, file movement, lightweight scripts, and non-CUDA work. Use GPU pods when CUDA, model inference, training, or GPU memory is required.
- Do not pass GPU flags when creating CPU pods. Check `runpodctl pod create --help` for the current valid flag set.
- **Waiting for a resource to be usable: use `--wait`, don't hand-roll a poll loop** (v2.9.0+). `create` returns as soon as the resource is *scheduled*, which is why a "RUNNING" pod often refuses ssh and a fresh endpoint 404s. `pod create --wait` returns when port 22 answers with an ssh banner; `serverless create --wait` when `/health` reports a ready or running worker. On timeout or ctrl-c the resource is **kept**, and its id is in the error object's `id` field — read that and clean up, don't assume nothing was created (a pod bills by the second; an endpoint with no running worker doesn't, but will start one on the first request).
- Standing up a **service on a pod** (Ollama, ComfyUI, a dev server)? Declare its `--ports` and `--env` **at creation** (they can't be added to a running pod without a reset), then follow the pod development loop in the `runpod-usage` skill (`reference/pod-workflows.md`) — SSH-exec the install, bind to `0.0.0.0`, and poll the proxy URL until it answers.
- For SSH, use `runpodctl pod get <pod-id>` or `runpodctl ssh info <pod-id>` to retrieve connection details. runpodctl has **no interactive-shell command** — `ssh info` returns the connection command + key but does not connect. Run commands over SSH yourself with `ssh user@host "command"`.
- Network volumes are location-sensitive. Check datacenter availability before attaching volumes, and use `send` / `receive` or S3-compatible storage for migrations.
- Clean up paid resources after tests: delete serverless endpoints, pods, and temporary volumes created for validation.
- **Cost guard on creation:** use `--terminate-after` (deletes the pod); `--stop-after` only *stops* it, so disk/volume keep billing.
- **Attached volume:** to delete a network volume, remove the pod using it first.
### Serverless facts (context, not rules)
- **Scale-to-zero billing:** serverless endpoints scale to zero with `--workers-min 0` (the default) — no GPU billing while idle, only per request-second; this is the right cost posture for a request/response API.
- **Broken-image tell:** if deployed workers go `ready` but jobs sit `IN_QUEUE` with `inProgress: 0`, the image is broken/mis-dispatching — the fix is to switch to a different worker rather than wait it out.
- **Diagnosing it:** read the worker/job counts with `runpodctl serverless health <endpoint-id>` (v2.9.0+), then read what the workers actually printed with `runpodctl serverless logs <endpoint-id>` (v2.10.0+) — no hand-built curl needed. Repeated `system` "start container" lines with no `container` output means the container exits before the handler runs.
## Commands
Essentials below. **For flags, ask the binary** — `runpodctl <resource> <action> --help`, which is current by construction. [reference/command-reference.md](reference/command-reference.md) holds the part `--help` cannot answer: what a flag *means* when it succeeds, which field to trust, and what a failure looks like.
### Pods
```bash
runpodctl pod list # running pods (+ --all / --status / --since / --created-after)
runpodctl pod get <pod-id> # details incl. SSH info + runtimeStatus
runpodctl pod create --template-id <id> --gpu-id "NVIDIA GeForce RTX 4090" # from template
runpodctl pod create --image <img> --gpu-id "NVIDIA GeForce RTX 4090" # from image
runpodctl pod create --compute-type cpu --image ubuntu:22.04 # CPU pod (lowercase `cpu`; serverless uses `CPU`)
runpodctl pod create --image <img> --gpu-id <id> --wait # block until ssh answers, then print the pod (v2.9.0+)
runpodctl pod {start|stop|restart|reset|update|delete} <pod-id> # lifecycle (delete aliases: rm/remove)
runpodctl pod logs <pod-id> # recent container+system logs, json lines (v2.10.0+)
runpodctl pod logs <pod-id> --follow # keep streaming, reconnects on its own
runpodctl pod logs <pod-id> --since 30m --source system # platform view: image pull / create / start
```
**A stalled deploy shows up in `--source system`** (v2.10.0+): repeated pull progress, or a
`create container` that never reaches `start`. Use `--source container` for your workload's own
output. Each line is one `{source,line,ts}` object, so pipe it straight to `jq`.
**Read `runtimeStatus`, not `desiredStatus`, to decide whether a pod is usable** (v2.9.0+):
`desiredStatus: RUNNING` says that while the image is still pulling. Field meanings, reason
tokens, and two edges (`--status` filters `desiredStatus` only; `unknown` = lookup failed, not
pod down) → [reference/command-reference.md](reference/command-reference.md#pod-status-fields).
### Hub
Browse/search the Runpod Hub (curated deployable repos).
```bash
runpodctl hub search vllm # find a repo (+ hub list [--type/--category/--order-by/--owner])
runpodctl hub get <listing-id|owner/name> # repo details
```
### Serverless (alias: sls)
```bash
runpodctl serverless list | get <endpoint-id> | delete <endpoint-id>
runpodctl serverless create --name "x" --template-id <id> # from template
runpodctl serverless create --name "x" --hub-id <listing-id> # from hub (+ --env KEY=VAL to override defaults)
runpodctl serverless create --hub-id <id> --gpu-id "NVIDIA GeForce RTX 4090" \
--model-reference https://huggingface.co/<org>/<model>:main # attach & host-cache a HF model (GPU only)
runpodctl serverless update <endpoint-id> --workers-max 5
runpodctl serverless create --template-id <id> --workers-min 1 --wait # block until a worker is ready (v2.9.0+)
```
**Invoke URLs come back with the endpoint.** `create`/`get`/`list`/`update` include a
`urls` object (`run`, `runsync`, `health`), so a freshly created endpoint is callable
without a second lookup — read them instead of assembling the URL yourself. They're
built from `RUNPOD_INVOKE_URL` (default `https://api.runpod.ai/v2`), which
`RUNPOD_API_URL`/`RUNPOD_GRAPHQL_URL` do **not** move: [reference/output-and-errors.md](reference/output-and-errors.md#serverless-invoke-urls).
**Reading worker logs** (v2.10.0+) — also first-class, so worker output no longer requires the
MCP lane or a hand-built SSE read:
```bash
runpodctl serverless logs <endpoint-id> # every worker's recent logs, json lines
runpodctl serverless logs <endpoint-id> --worker <worker-id> # just one worker
runpodctl serverless logs <endpoint-id> --follow # picks up workers that scale up mid-follow
runpodctl serverless logs <endpoint-id> --since 1h --source system # why a worker will not start
```
Logs belong to a **worker**, not the endpoint, so without `--worker` this reads them all at once and
tags each line with its `workerId`. **The crash-loop tell:** repeated `system` "start container" lines
with no `container` output means the container exits before the handler runs — jobs then sit in the
queue with nothing wrong with capacity.
**Invoking an endpoint** (v2.9.0+) — first-class commands, so an agent does not hand-build a
curl request or manage a bearer token:
```bash
runpodctl serverless run <endpoint-id> --input '{"prompt":"hello"}' # submit and wait for the result
runpodctl serverless run <endpoint-id> --input-file payload.json # same, payload from a file ("-" = stdin)
runpodctl serverless run <endpoint-id> --input '{}' --wait 15m # longer budget (default 5m)
runpodctl serverless run <endpoint-id> --input '{}' --no-wait # submit, print the queued job, exit 0
runpodctl serverless status <endpoint-id> <job-id> # check a job submitted earlier
runpodctl serverless health <endpoint-id> # worker + job counts
```
- **Pass only the handler payload** — the cli wraps it as `{"input": <your json>}` itself, so
pasting a whole curl envelope double-wraps it.
- **`timeout` means the cli stopped waiting, not that the endpoint broke.** When the message
names a `serverless status` command the job is still running server-side — poll it, do
**not** re-invoke (that buys a second job).
Payload rules, the stdout/stderr split, exit codes and why `/runsync` is never used →
[reference/command-reference.md](reference/command-reference.md#invoking-an-endpoint-serverless-run-v290).
**Create from hub:** `--hub-id` resolves the hub listing, extracts the build image and config (GPU IDs, container disk, env vars), creates an inline template, and deploys. Accepts both SERVERLESS and POD listing types. GPU IDs and env var defaults from the hub config are included automatically; override with `--gpu-id` and `--env`.
**CPU serverless endpoints** (the always/never rule is in Decision Rules above): create with `runpodctl serverless create --compute-type CPU` (optionally `--instance-id`, e.g. `cpu3g-4-16`). Verified evidence for why the public REST must not be used: 2026-07-14, `POST https://rest.runpod.io/v1/endpoints` with `"computeType":"CPU"` silently returned a GPU endpoint (`gpuCount:1`, `cpuFlavorIds:null`), while `runpodctl --compute-type CPU` correctly returned `computeType:"CPU"` with `instanceIds:["cpu3g-4-16"]`. The MCP server is **not** an alternative here: its v2 `create-endpoint` requires `gpuPoolIds` and the v2 spec has no `computeType`/`cpuFlavor` field at all (verified 2026-07-29). The public control REST is v1-only (`rest.runpod.io/v2` just redirects to docs). The separate **runtime/invoke** API `https://api.runpod.ai/v2/<endpoint-id>/…` (health/run/runsync/openai) is a different v2 and works fine — the v1-vs-v2 caveat here is only about the **control/management** REST.
**Model cache (`--model-reference`):** Attach a Hugging Face model to the endpoint by full URL with a ref, e.g. `https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct:main`. Runpod caches it host-side in the standard HF cache dir (`/runpod-volume/huggingface-cache/hub/`), so the worker loads it directly — no bake, no volume. Repeatable; works with `--template-id`/`--hub-id`, GPU only, **runpodctl v2.4.0+**. Full mechanics + how it compares to baking / network volume / the Model Repository: **[reference/model-caching.md](reference/model-caching.md)**. Worked end-to-end: golden path [20 — model-caching endpoint](../runpod/golden-paths/20-model-caching-endpoint.md).
**Multi-region / high-availability (`--network-volume-ids`):** attach **multiple** network
volumes (one per data center) so workers spread across DCs instead of being pinned to one —
`runpodctl serverless create --template-id <t> --network-volume-ids <v1>,<v2> --data-center-ids <dc1>,<dc2> …`.
**Requires runpodctl ≥ v2.4.0** (older versions don't support multi-volume attach). Check
`runpodctl version`; the Homebrew tap can lag, so prefer the
[GitHub releases](https://github.com/runpod/runpodctl/releases) binary. Data does **not**
sync between volumes automatically — see golden path
[10 — multi-region HA serverless](../runpod/golden-paths/10-multi-region-ha-serverless.md).
For exact serverless flags, run `runpodctl serverless <action> --help`.
### Templates (alias: tpl)
```bash
runpodctl template search <q> # find (+ template list [--type official/community/user, --all, --limit])
runpodctl template get <template-id> # details (README, env, ports)
runpodctl template create --name "x" --image "img" [--serverless]
runpodctl template delete <template-id>
```
### Network Volumes (alias: nv)
```bash
runpodctl network-volume list # List all volumes
runpodctl network-volume get <volume-id> # Get volume details
runpodctl network-volume create --name "x" --size 100 --data-center-id "US-GA-1" # Create volume
runpodctl network-volume update <volume-id> --name "new" # Update volume
runpodctl network-volume delete <volume-id> # Delete volume
```
For exact network volume flags, run `runpodctl network-volume <action> --help`.
> **No storage-tier flag.** `create` provisions the data center's **default** tier — there's
> no `--type`. To get a **High-Performance** volume, use the console (a ⚡ data center's toggle)
> or a raw **v2 REST** call (`POST https://v2-rest.runpod.io/v2/network-volumes` with
> `"type":"HIGH_PERFORMANCE"`) — or the MCP `create-network-volume` tool, which takes
> `volumeType` (`STANDARD` | `HIGH_PERFORMANCE`). Tier is immutable after creation. Launch details: golden path [21](../runpod/golden-paths/21-storage-tiers.md).
### Models (Model Repository)
`runpodctl model` manages the **Runpod Model Repository** — managed, versioned storage
for your **own** model artifacts (upload once, distributed to workers; not pinned to a
data center like a network volume). What it is, why/how, migrating off a baked-in model,
and Model-Repo-vs-volume: **[reference/model-caching.md](reference/model-caching.md)**.
```bash
runpodctl model list # List your models
runpodctl model list --all # List all models (not just yours)
runpodctl model list --name "llama" # Filter by name
runpodctl model list --provider "meta" # Filter by provider
runpodctl model add --name "my-model" --model-path ./model # Upload a local model dir (multipart)
runpodctl model remove --name "my-model" --owner <owner> # Remove a model
```
`model add` supports upload sessions, versioning, metadata, and private-source credentials — see live `runpodctl model add --help`.
### Info & SSH
```bash
runpodctl user # account info + balance (alias: me)
runpodctl gpu list # available GPUs + $/hr + per-DC stock (+ --include-unavailable)
runpodctl datacenter list # datacenters (alias: dc)
runpodctl ssh info <pod-id> # SSH connection details (command + key; NOT an interactive session)
```
**`gpu list` carries pricing and placement data** — `securePricePerHr` /
`communityPricePerHr` (explicitly `null` when that cloud doesn't offer the GPU) and a
`dataCenterAvailability[]` breakdown. Read the breakdown, not just top-level
`stockStatus` (which is only the *best* status across DCs), when a create has to
schedule in a specific DC — and pass `--include-unavailable`, since the default listing
hides no-stock GPUs and can omit one that has stock only in the DC you want. The prices
are **pod on-demand** rates. Shape, stock-value vocabulary and the `"none"` vs
omitted-key sentinel:
[reference/output-and-errors.md](reference/output-and-errors.md#gpu-pricing-and-per-data-center-availability).
`ssh info` gives connection details, not a session — if interactive SSH isn't available, run `ssh user@host "command"`. **Registry auth, `billing` history, and SSH key management** (`ssh add-key`/`remove-key`) are in [reference/command-reference.md](reference/command-reference.md).
### File Transfer
```bash
runpodctl send <path> # prints a one-time code, then blocks until the receiver connects
runpodctl receive <code> # positional code (no --code flag)
```
Encrypted/incremental/compressed — don't pre-tar. **Key gotchas:** capture the **first line of `send` stdout** (the code) as it streams (background + tee), each `send` mints a **fresh** code, both sides must exit `0`. Full agent flow (pod push via `ssh` + `receive`): [reference/command-reference.md](reference/command-reference.md#file-transfer).
### Utilities
```bash
runpodctl doctor # Diagnose and fix CLI issues
runpodctl update # Update CLI
runpodctl version # Show version
runpodctl completion # Auto-detect shell and install completion
```
## URLs
### Pod URLs
Access exposed ports on your pod:
```
https://<pod-id>-<port>.proxy.runpod.net
```
Example: `https://abc123xyz-8888.proxy.runpod.net`
### Serverless URLs
Prefer `runpodctl serverless run|status|health` (above) — same api, with auth, validation and
bounded polling handled. Use the raw urls for what the commands don't cover: streaming, the
OpenAI-compatible route, or a copy-paste `curl` for a user.
```
https://api.runpod.ai/v2/<endpoint-id>/run # Async request
https://api.runpod.ai/v2/<endpoint-id>/runsync # Sync request
https://api.runpod.ai/v2/<endpoint-id>/health # Health check
https://api.runpod.ai/v2/<endpoint-id>/status/<job-id> # Job status
```
`serverless create`/`get`/`list`/`update` already return `run`/`runsync`/`health` in a
`urls` object — prefer those over hand-assembling, since a non-default
`RUNPOD_INVOKE_URL` changes the base. Only `status/<job-id>` has to be built by hand.
## Source & docs
- CLI source: https://github.com/runpod/runpodctl
- Releases (binaries): https://github.com/runpod/runpodctl/releases
- Docs: https://docs.runpod.io/runpodctl/overview