evals/pod-service-loop.eval.md
# Stand up a service on a pod — the loop's load-bearing details
## Prompt
Stand up an Ollama server on a Runpod pod over SSH and give me the URL. Walk
through the commands.
## Expected behavior
Per `runpod-usage/reference/{pod-workflows.md,on-pod-setup.md}`:
1. **Ports + env at creation** — expose `11434/http` and set `OLLAMA_HOST=0.0.0.0`
at `pod create` time (they can't be added to a running pod without a reset).
2. **Env isn't in the SSH shell** — when starting the service over SSH, pass the
env explicitly (`env OLLAMA_HOST=0.0.0.0 … ollama serve`), because creation
`--env` vars only reach PID 1.
3. **Detach the server** — start it with `setsid … < /dev/null &` so it survives
the SSH channel closing, and return immediately.
4. **Verify from outside** — poll `https://<pod-id>-11434.proxy.runpod.net/api/tags`
until 200 (expect warm-up 502s) before reporting the URL.
5. **Cost guard** — `--terminate-after` (not `--stop-after`).
## Assertions
- Sets the port and env at `pod create` (not after).
- Passes env explicitly on the SSH-launched service command (doesn't rely on `--env` reaching the shell).
- Starts the server detached (`setsid`/`nohup` + `</dev/null`), not a bare `&`.
- Verifies by polling the proxy URL, and uses `--terminate-after` as the cost guard.
evals/verify-with-real-request.eval.md
# Verify readiness with a real request — "up" ≠ "ready"
## Prompt
I started a web server on my Runpod pod and `runpodctl pod get` says it's RUNNING,
but requests to the proxy URL are failing. Is it broken? How should I confirm it's
actually ready?
## Expected behavior
Per `runpod-usage/reference/development-loop.md` (step 6) + `pod-workflows.md`:
1. The agent should explain that **"RUNNING" only means the container exists**, not
that the service serves — and that the proxy commonly returns **502 for ~30–60s**
during warm-up, so failing requests right after boot are expected, not broken.
2. It should confirm readiness by **polling a real request** to the proxy URL
(e.g. `until curl -sf https://<pod-id>-<port>.proxy.runpod.net/<health>; do sleep 5; done`)
with a timeout, not by trusting the status field.
3. It should also check the likely real causes if it stays down: the service isn't
bound to `0.0.0.0`, or it died because it wasn't started detached (`setsid`).
## Assertions
- States that RUNNING/ready is not the same as serving; expects a warm-up 502 window.
- Recommends polling a real request against the proxy URL to confirm readiness.
- Mentions the `0.0.0.0` bind and/or detached-start (`setsid`) as the usual culprits if it stays down.
- Does NOT conclude "it's broken" from the RUNNING status alone.
reference/building-images.md
# Building container images for Runpod
How to think about building an image for Runpod: pick the right base, layer for speed,
decide what to **bake in** vs **mount at runtime**, and match the **image contract** to your
target (pod vs serverless queue vs serverless load-balanced). CLI mechanics (login, tag,
push) live in [companion-clis docker](../../companion-clis/reference/docker.md) and
[docker.md](docker.md); this is the strategy layer.
## Start from an official Runpod base image
**For a GPU workload, build `FROM` an official `runpod/pytorch:<tag>` image.** Two reasons:
- **torch/CUDA already match Runpod hosts**, so you don't fight driver/toolkit mismatches.
- **Runpod pre-caches official base images on its hosts.** The base layers are effectively
already on the machine, so they don't re-download at pull time — you only ship the layers
you add on top. Starting from a random public base throws that away.
**Exceptions (both shown in the golden paths):** a trivial CPU-only workload may use a slim
base (e.g. `python:3.11-slim`) — see [GP23](../../runpod/golden-paths/23-minimal-queue-image/README.md);
and if you build from a **non-Runpod base you must reproduce SSH yourself** for pods (see the
SSH section below and [GP22](../../runpod/golden-paths/22-minimal-pod-image/README.md)).
Then, whatever the base:
- **Pin an exact tag**, e.g. `runpod/pytorch:1.0.2-cu1281-torch280-ubuntu2404` — for
reproducible builds.
- **Build for x86_64:** `docker build --platform=linux/amd64 …` — Runpod hosts are x86_64
(see [docker.md](docker.md)).
## Layer for fast, cacheable pulls
Order layers **least- to most-frequently-changing** so a code edit doesn't invalidate the
heavy dependency layers, and independent layers pull in parallel:
1. base image (`FROM runpod/pytorch:…`)
2. system deps (`apt-get …`)
3. Python deps (`pip install …`)
4. **your code last**
Unchanged layers are reused from cache; only the layers after your edit rebuild/re-pull.
## Dockerfile best practices
- **`.dockerignore`** — exclude `.git`, virtualenvs, datasets, local caches so the build context stays small and pushes fast.
- **Cache the dependency layer** — `COPY requirements.txt` and `pip install` *before* you `COPY` your code, so a code edit doesn't reinstall everything:
```dockerfile
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
```
- **Shrink the image** (smaller = faster pull + cold start) with concrete steps: `apt-get install --no-install-recommends …` then `rm -rf /var/lib/apt/lists/*`; `pip install --no-cache-dir`; use a **multi-stage** build when heavy build tools aren't needed at runtime.
- **BuildKit cache mounts** for fast rebuilds: `RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt`.
- **Never bake secrets** into layers (API keys, tokens) — layers are extractable; pass secrets as runtime env.
- **`ENV PYTHONUNBUFFERED=1`** — so logs stream unbuffered. (Pin the base image tag too — see above.)
## Don't clobber the base image's startup (SSH / web terminal) — **pods**
Official `runpod/pytorch` images ship `CMD ["/start.sh"]`, and **that script is what makes a
pod usable**: it reads `$PUBLIC_KEY` into `~/.ssh/authorized_keys`, runs `ssh-keygen -A`,
starts `sshd`, and brings up the web terminal / Jupyter. It also runs `/pre_start.sh` before
and `/post_start.sh` after, if those exist.
**Rule (pods):** any custom `CMD`/`ENTRYPOINT` **must invoke `/start.sh`** — inherit it, or run
`/start.sh &` before your workload. **Exception:** serverless images are exempt (no SSH).
Why: if a custom `CMD`/`ENTRYPOINT` doesn't chain `/start.sh`, none of that startup runs — you
get **no SSH, no web terminal**, and can be locked out of the pod. For a pod this is the #1
footgun.
Three safe patterns, in order of preference:
1. **Don't override `CMD` at all** (default — use this unless you need your own foreground
process). Add your layers, leave `CMD ["/start.sh"]`. Do per-pod work via the env-driven
hooks the base already runs:
```dockerfile
FROM runpod/pytorch:<tag>
COPY post_start.sh /post_start.sh # base runs this AFTER sshd is up
RUN chmod +x /post_start.sh
# no CMD — inherit the base's /start.sh
```
2. **Override only if you need your own foreground process** (a long-running service as PID 1):
call the base start first, then `exec` your workload:
```dockerfile
COPY run.sh /run.sh
RUN chmod +x /run.sh
CMD ["/run.sh"]
```
```bash
#!/usr/bin/env bash
/start.sh & # SSH + web terminal (base startup), backgrounded
sleep 2
exec python -u my_service.py # your long-running workload in the foreground
```
3. **From a non-Runpod base, reproduce SSH yourself** (only if you can't start from
`runpod/pytorch`). Minimum to not get locked out of a pod:
```dockerfile
RUN apt-get update && apt-get install -y --no-install-recommends openssh-server \
&& rm -rf /var/lib/apt/lists/*
COPY start.sh /start.sh
RUN chmod +x /start.sh
CMD ["/start.sh"]
```
```bash
#!/usr/bin/env bash
mkdir -p ~/.ssh && chmod 700 ~/.ssh
[ -n "$PUBLIC_KEY" ] && echo "$PUBLIC_KEY" >> ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys
ssh-keygen -A # host keys
service ssh start # or: /usr/sbin/sshd -D
exec "$@" # then your workload (or keep sshd in foreground)
```
Reference implementation: `justinwlin/Runpod-GPU-And-Serverless-Base` (a dual pod+serverless
base) and the vendored `start.sh` in
[golden path 09](../../runpod/golden-paths/09-custom-serverless-dev-loop/README.md). Worked
end-to-end in [golden path 22 — minimal pod image](../../runpod/golden-paths/22-minimal-pod-image/README.md).
## Bake in vs mount at runtime (this drives startup speed)
Only the **image** (and whatever is baked into it) lands on the host's **local disk** — fast.
When a **network volume** is attached it takes over the working directory (`/workspace` on a
pod, `/runpod-volume` on serverless); anything written there lives on **networked storage**,
which is slower — **especially for many small files**.
Live proof that this is a real filesystem boundary (baked = `overlay`/local, volume =
`fuse`/MooseFS network mount): [golden path 25 — bake vs mount](../../runpod/golden-paths/25-bake-vs-mount/README.md).
- **Bake into the image:** packages, libraries, and lots of small static files → local, fast.
- **Mount a volume:** large/few files (model weights, datasets), anything that must persist
across pods, or data you stream/live-load.
- **High-throughput / I/O-bound training:**
- **temporary / one-off run** → a **pod using local (non-network) storage** is fastest;
- **persistent, many small files, or I/O-bound** → a **high-performance network volume**.
See [golden path 21 — storage tiers](../../runpod/golden-paths/21-storage-tiers.md).
## Match the image contract to the target
| Target | Needs a handler? | Entry point |
| --- | --- | --- |
| **Pod** | No | your `CMD`/entrypoint — a long-running service; bind `0.0.0.0`, expose ports |
| **Serverless — queue-based** | **Yes** | `runpod.serverless.start({"handler": handler})` |
| **Serverless — load-balanced** | No (different contract) | your own **HTTP server** exposing routes (no queue handler) |
Minimal runnable image per contract (each built + deployed live): pod →
[golden path 22](../../runpod/golden-paths/22-minimal-pod-image/README.md), queue →
[golden path 23](../../runpod/golden-paths/23-minimal-queue-image/README.md), load-balanced →
[golden path 14](../../runpod/golden-paths/14-load-balancing-endpoint.md).
Queue vs load-balanced request/response shapes and when to pick each are covered in
[endpoint-workflows.md](endpoint-workflows.md) and golden paths
[12 (streaming)](../../runpod/golden-paths/12-serverless-streaming.md),
[14 (load-balancing)](../../runpod/golden-paths/14-load-balancing-endpoint.md), and
[17 (WebSocket)](../../runpod/golden-paths/17-serverless-websocket.md). One image can be
**dual-mode** (pod dev + serverless) — see
[golden path 09](../../runpod/golden-paths/09-custom-serverless-dev-loop/README.md).
## Test locally before deploying
- Build for the right platform: `docker build --platform=linux/amd64 …`.
- **Queue handler:** run the container and invoke `handler.py` locally (the dual-mode
`python handler.py` loop in [golden path 09](../../runpod/golden-paths/09-custom-serverless-dev-loop/README.md))
before pushing.
- **Load-balanced:** run the container and hit the HTTP routes locally.
- Then push ([companion-clis docker](../../companion-clis/reference/docker.md)) and deploy
(runpodctl or runpod-mcp).
reference/concepts.md
# Pods vs Serverless
Two ways to run compute on Runpod. Pick based on the shape of the work.
## Pods — interactive / long-lived
A Pod is a GPU or CPU container you rent by the minute and keep running. You get
full control: SSH, a web terminal, JupyterLab, VS Code/Cursor, exposed ports. It
stays up until you stop or delete it, and you pay for every minute it exists
(running or stopped-with-disk).
Use a Pod when:
- You are developing, experimenting, or debugging interactively.
- Work is long-running or stateful (training, fine-tuning, rendering, a notebook).
- You want a persistent environment you shell into and iterate in.
- You need a service up continuously with a stable address.
Not a good fit when traffic is bursty or idle much of the day — you pay for idle time.
Two clouds:
- **Secure Cloud** — T3/T4 data centers, high redundancy. Production and sensitive data.
- **Community Cloud** — vetted peer-to-peer providers, cheaper, variable reliability.
(Runpod is no longer onboarding new Community Cloud hosts; existing capacity remains.)
Limits: no Docker Compose (Runpod runs Docker for you), no UDP (TCP/HTTP only), no Windows.
## Serverless — request/response + autoscale
Serverless runs your container only while it is processing requests. You deploy a
worker image behind an **endpoint** (a URL). Workers spin up on demand, process
jobs, and spin down when idle. You pay for compute time used, with no idle cost
when nothing is running.
Use Serverless when:
- Work is request-shaped: inference, image generation, transcription, batch jobs.
- Traffic is bursty or unpredictable and you want it to scale to zero.
- You want a managed URL, not a machine to babysit.
## Serverless building blocks
- **Endpoint** — the access point (URL) clients send requests to. Holds the scaling
and GPU config.
- **Worker** — a container instance running your image + code. Runpod starts and
stops workers automatically based on load.
- **Handler function** (queue-based) — `def handler(event)` reads `event["input"]`,
processes it, returns a result. Started with `runpod.serverless.start({"handler": handler})`.
- **Job** — one unit of work: the input payload, queued until a worker is free.
## Cold starts and FlashBoot
A **cold start** is the gap between a request arriving at an endpoint with no ready
worker and that worker being warmed up — container start + model load into VRAM +
runtime init. Bigger models = longer cold starts.
Reduce cold starts by:
- **FlashBoot** (on by default) — retains worker state after spin-down so a worker
"revives" faster than a fresh boot. Most effective with steady traffic where
workers cycle between active and idle.
- **Cached models** — schedule workers onto machines with your model files
pre-loaded, cutting model-load time.
- **Active workers** ≥ 1 — keep workers always warm (see below).
## Active vs flex workers, scale-to-zero
- **Active (min) workers** — always-on, kept warm at all times. Setting this to 1+
eliminates cold starts for those slots but bills continuously, even when idle.
Default is **0**.
- **Flex workers** — the elastic pool between active count and **max workers**.
Spun up under load, spun down when idle. **Scale-to-zero** = active workers 0, so
the endpoint drops to zero running workers (and zero cost) when idle, at the price
of a cold start on the next request.
- **Idle timeout** — how long a flex worker stays warm after finishing before it
shuts down (default 5s). Longer = fewer cold starts, more cost.
- **Max workers** — concurrency cap and cost safety limit (default 3). Set ~20%
above expected peak concurrency to absorb spikes.
> **These are the platform defaults** (Console / `runpodctl` / API). The **flash SDK**
> applies its *own* defaults for the same settings — `idle_timeout` **60s** (not 5s),
> `workers` **(0, 1)** i.e. max 1 (not 3), `execution_timeout` unlimited (not 600s). So a
> default value depends on which layer you configured through; don't assume the flash
> number holds for a platform-created endpoint or vice-versa. See
> [`../../flash/reference/api.md`](../../flash/reference/api.md).
Auto-scaling type decides *when* to add workers:
- **Queue delay** — add workers when requests wait longer than a threshold
(default 4s). Good when small delays are acceptable.
- **Request count** — scale on pending + in-progress work
(`ceil((inQueue + inProgress) / scalerValue)`). More aggressive; good for LLMs
and frequent short requests.
## Queue-based vs load-balanced endpoints
Two endpoint types, chosen at creation:
**Queue-based** (traditional)
- Requests go into a queue and are processed in order; execution is guaranteed with
automatic retries.
- Uses a handler function; fixed operations: `/run`, `/runsync`, `/status`, `/stream`,
`/cancel`, `/health`, etc.
- Best for async tasks, batch, long-running jobs. Higher latency (queue + worker).
**Load-balanced**
- Requests route directly to a worker's HTTP server — no queue, no backlog buffering
(overloaded workers drop requests).
- You run any HTTP server (FastAPI, Flask) and define your own URL paths; workers
expose a `/ping` health check.
- Lower latency (single hop). Best for real-time inference, streaming, custom REST APIs.
- No built-in retry.
Analogy from the docs: queue-based is like TCP (guaranteed delivery), load-balanced
is like UDP (fast, no guarantees).
## Templates
A **template** is a saved, pre-configured setup: a Docker image plus its default
config (exposed ports, environment variables, container/volume disk, start command).
Runpod ships official templates (e.g. PyTorch) so you can launch a working
environment without wiring dependencies yourself, and you can save your own custom
templates for repeatable deployments of both Pods and endpoints.
## Where to act
This file is a mental model. To actually create or manage resources use
**runpodctl** / **runpod-mcp** (infra), **flash** (deploy your own code), or the
Runpod console.
reference/development-loop.md
# The Runpod development loop (golden loop)
Every Runpod task an agent runs follows the same spine — proven across the golden
paths (Ollama pod, ComfyUI pod, Whisper endpoint). Learn this loop; it has two
specializations depending on the workload shape.
```
decide shape → prefer prebuilt → plan resources → provision → (set up if scratch)
→ run/deploy → VERIFY with a real request → deliver → cost-guard + teardown
```
## 1. Decide the workload shape
- **A server you open / interactive / long-lived** (Ollama, ComfyUI, Jupyter,
training) → a **pod**. Reached at a proxy URL. Detailed loop: `pod-workflows.md`.
- **A request/response API that should scale to zero** (transcription, inference
endpoint) → a **serverless endpoint**. Invoked via `/run`/`/runsync`. Detailed
loop: `endpoint-workflows.md`.
See `concepts.md` if unsure.
## 2. Prefer a prebuilt / known option before building from scratch
This is the biggest lever for speed and reliability:
- Pod service → look for an **official Runpod template / prebuilt image**
(`runpodctl template search <app>`) — it auto-starts and skips the install
gotchas.
- Serverless → look for a **Hub worker** (`runpodctl hub search <app>`).
- Build **from scratch** (install on a pod, or `flash`, or a custom image) only
when no good prebuilt exists, or you need something **custom or lighter** than
what's shipped.
## 3. Plan resources
GPU/VRAM (`gpu-selection.md`), storage (**default a network volume** —
`storage.md`), and the execution lane (`../../runpod/SKILL.md` router:
runpod-mcp / runpodctl / flash).
## 4. Provision & 5. Set up
Provision through the chosen lane. If from-scratch, do the setup step of the
matching sub-loop (pods: SSH-exec install; serverless: write handler / build image).
Prebuilt options usually skip setup entirely.
## 6. Verify with a real request — "up" ≠ "ready"
The load-bearing step. A pod showing **Running**, or a serverless worker showing
**ready**, does **not** mean it serves. Always confirm from **outside** with a real
call, and expect a warm-up window:
- **Pod:** poll the proxy URL until it answers — expect ~30–60s of **502s** during
boot.
- **Serverless:** send a real input; the **first call cold-starts** (may exceed
`runsync`'s 60s → use `/run` + poll `/status/<id>`). A worker that is `ready` but
leaves jobs `IN_QUEUE` with `inProgress: 0` is a **broken image** — switch, don't
wait.
Only report success once a real request returns the right result.
## 7. Deliver
Return the access URL (pod) or endpoint id + a **working sample call** (serverless),
and note the security posture (proxy URLs and endpoints are public unless you add
auth).
## 8. Escalate on manual steps
If something needs a human — OAuth, a quota/capacity increase, a gated-model
license, a missing credential, a payment issue — **stop and say exactly what's
blocked**. Don't spin or fake progress.
## 9. Cost-guard + teardown
- Pod → `--terminate-after <ts>` at creation (deletes it), not `--stop-after`.
- Serverless → `--workers-min 0` (scale-to-zero, ~$0 idle).
- Delete test resources when done (`runpodctl pod remove` / `serverless delete` /
`flash app delete`; then any network volume).
## Which sub-loop?
| Workload | Sub-loop |
| --- | --- |
| A service you open at a URL (Ollama, ComfyUI, dev box, training) | `pod-workflows.md` |
| A request/response API that scales to zero (Whisper, inference) | `endpoint-workflows.md` |
reference/docker.md
# Building a Docker image Runpod can run
Runpod pulls a container image and runs it on x86_64 Linux GPU/CPU hosts. For
serverless, the image runs a **handler**; for pods, the image runs whatever your
`CMD`/start script does. This file covers the serverless handler contract and the
build/push mechanics that apply to both.
## The serverless handler contract
A queue-based serverless worker is a Python script that starts the Runpod SDK
with a handler function:
```python
# handler.py
import runpod
def handler(job):
job_input = job["input"] # your request payload lives under "input"
prompt = job_input.get("prompt")
# ... do the work ...
return {"result": prompt} # returned value becomes the job output
runpod.serverless.start({"handler": handler}) # required — blocks and serves jobs
```
Request/result shape:
- The platform hands your handler a job dict: `{"id": "<uuid>", "input": { ... }}`.
`id` is Runpod's job id; `input` is exactly what the client sent.
- Whatever the handler **returns** is the job result (must be JSON-serializable).
- Raising an exception marks the job `FAILED` and returns the error details.
Handler variants: return a value (standard), `yield` values (streaming — add
`return_aggregate_stream: True` to expose them via `/run`), or `async def` +
`yield` (async). Concurrent handlers serve multiple requests per worker.
**Load-balanced endpoints do not use a handler.** You expose your own HTTP server
(FastAPI, Flask, vLLM, etc.) and Runpod routes to it. The `runpod.serverless.start`
contract is only for queue-based endpoints.
Best practice: load models and other heavy state **at module level, outside the
handler**, so it initializes once per worker instead of once per request.
## A minimal Dockerfile
```dockerfile
FROM python:3.11.1-slim
WORKDIR /
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY handler.py .
CMD ["python", "-u", "/handler.py"]
```
`requirements.txt` must include the `runpod` SDK plus your libraries:
```
runpod~=1.7.6
torch==2.0.1
transformers==4.30.2
```
> **Version pin gotcha (numpy 2.x):** if you `pip install torch==2.2.x`, also pin
> `numpy<2`. That torch wheel is built against NumPy 1.x; pip otherwise pulls NumPy
> 2.x and the image **builds and starts fine but crashes at inference** with
> `RuntimeError: Numpy is not available`. Testing the container locally (below) catches
> it before deploy (`gotchas.md`).
> **`cryptography` uninstall gotcha (on a `runpod/pytorch` base):** installing the
> `runpod` SDK on an official `runpod/pytorch` image (e.g. behind `runpod-torch-v280`)
> fails with `Cannot uninstall cryptography 41.0.7 … no RECORD file` — the base's copy is
> **Debian-managed**, so pip can't replace it with the newer one `runpod` wants. Fix:
> `pip install --ignore-installed cryptography runpod` (add `--break-system-packages` for
> the base's PEP-668 "externally managed" pip). Install conflict-free deps like
> `faster-whisper` first, on their own. Verified live 2026-07-10 (golden path 09).
For GPU/CUDA workloads, start from a CUDA base and install Python yourself, or
build on a framework image:
```dockerfile
FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04
RUN apt-get update && apt-get install -y python3.11 python3-pip
# or: FROM pytorch/pytorch:2.0.1-cuda11.7-cudnn8-runtime
```
Keep images small (slim bases, multi-stage builds, `.dockerignore`, clean up apt
caches) — smaller images pull and cold-start faster.
## Build for x86_64 — always `--platform=linux/amd64`
Runpod hosts are x86_64. If you build on an ARM machine (Apple Silicon Mac) the
default build produces an arm64 image that Runpod cannot run. **Always pass the
platform flag:**
```bash
docker build --platform=linux/amd64 -t DOCKER_USER/worker-name:v1.0.0 .
```
This is the single most common deployment failure. It applies to serverless
workers, pod images, and template images alike.
## Pin tags — never rely on `latest`
`latest` is mutable: it points to whatever was last pushed without an explicit
tag, and Runpod caches images per host, so a worker can silently keep running a
stale `latest`. Use explicit semantic versions (`v1.0.0`, `v1.0.1`), and for
critical deploys pin the immutable digest (`name:tag@sha256:...`).
```bash
docker build --platform=linux/amd64 -t DOCKER_USER/worker-name:v1.0.0 . # good
docker build -t DOCKER_USER/worker-name:latest . # avoid
```
## Test the container locally before pushing
Provide a `test_input.json` next to the handler and run the image — the SDK
detects it, runs one job, prints the output, and exits:
```json
{ "input": { "prompt": "Hey there!" } }
```
```bash
python handler.py # bare, on host
python handler.py --test_input '{"input":{"prompt":"hi"}}' # inline input
docker run -it DOCKER_USER/worker-name:v1.0.0 # inside the image
docker run --rm --gpus all DOCKER_USER/worker-name:v1.0.0 # with GPU
```
Fix any import/dependency/handler errors here — it is far faster than debugging
after deploy.
## Push, and handle private images
```bash
docker login
docker push DOCKER_USER/worker-name:v1.0.0
```
Runpod pulls public images with no extra config. For **private** registry images
you must register container-registry credentials with Runpod once (Console →
Settings → Container Registry, then select the credential on the template/endpoint).
Runpod supports `docker login`-style credentials. Without this, workers fail to
pull the image.
## Deploy the pushed image as a serverless endpoint
`runpodctl serverless create` takes **`--template-id` or `--hub-id`, not `--image`** —
so deploying your own image is a **two-step**: register a serverless *template* pointing
at the tag, then create the endpoint from that template. Use `--compute-type CPU` for a
CPU-only image to sidestep GPU scarcity, and `--workers-min 0` for scale-to-zero:
```bash
runpodctl template create --name my-tpl --serverless \
--image DOCKER_USER/worker-name:v1.0.0 --container-disk-in-gb 10 # → template id
runpodctl serverless create --template-id <template-id> --name my-ep \
--compute-type CPU --workers-min 0 --workers-max 1 # → endpoint id
```
For a **private** image, attach the registry credential to the template (Console →
Container Registry, or `runpodctl registry create`) so workers can pull it. Full worked
example: golden path 05 (`golden-paths/05-model-to-endpoint-pipeline.md`).
## Payload limits
Handler input/output flows through the job API: `/run` caps at ~10 MB, `/runsync`
at ~20 MB. For anything larger, pass URLs or use a network volume / S3 and return
references instead of inlining bytes.
reference/endpoint-workflows.md
# The serverless endpoint loop
The serverless specialization of the development loop (`development-loop.md`) — for
a **request/response API that scales to zero** (transcription, inference). Unlike a
pod there's no SSH, no exposed ports, no proxy: you deploy a worker and invoke it
over the Runpod job API. Proven on the Whisper golden path (both variants).
## 1. Pick the source (in order of preference)
1. **Hub worker (fastest, least fragile)** — a maintained prebuilt worker.
`runpodctl hub search <app>` → deploy with runpodctl (Hub is runpodctl-only).
Best when a good worker ships what you need.
2. **flash (from scratch, custom/light)** — write an `@Endpoint` handler and
`flash deploy`. Best when you need your own model size / I/O schema / a lighter
image, or no good Hub worker exists.
3. **Custom image + endpoint (last resort)** — write a handler, `docker build
--platform=linux/amd64`, push (private image → registry auth), create the
endpoint (runpodctl/MCP). Only when neither of the above fits.
## 2. Deploy (scale-to-zero)
```bash
# Hub
runpodctl serverless create --hub-id <id> --name <name> --workers-min 0 --workers-max 3
# flash
flash deploy # @Endpoint(workers=(0,3)) in the code
# Custom image — TWO steps: serverless create takes --template-id/--hub-id, NOT an image.
runpodctl template create --name <tpl> --serverless \
--image <you>/<img>:<tag> --container-disk-in-gb 15 \
--env '{"KEY":"VALUE"}' # --env is a JSON object here; → template id
runpodctl serverless create --template-id <template-id> --name <name> \
--gpu-id "NVIDIA GeForce RTX 4090" --workers-min 0 --workers-max 2 # --gpu-id, not --gpu-type
```
`--workers-min 0` = no GPU billing while idle (pay only per request-second). Don't pin
`--data-center-ids` for a custom-image endpoint **unless a network volume forces it** — a
single-DC pin on a scarce GPU leaves workers `throttled` and jobs stuck `IN_QUEUE`
(observed live, 2026-07-10: EU-RO-1 4090 pin → `throttled:1`; unpinned → scheduled at once).
### Picking a Hub worker (this decides success)
Prefer an **actively-maintained** worker on a **broad, high-availability GPU pool**
— don't pin a scarce large tier a small model doesn't need. If deployed workers go
`ready` but jobs sit `IN_QUEUE` with `inProgress: 0`, the image is broken /
mis-dispatching — **switch workers, don't wait it out.** (Confirm it before switching:
`runpodctl serverless health <id>` for the counts (v2.9.0+) and `runpodctl serverless logs
<id> --source system` for the cause (v2.10.0+) — repeated `start container` with no
`container` output is a crash loop, not a capacity problem. MCP `stream-worker-logs` reads
the same logs.)
## 3. Invoke
The raw protocol is below because it is what you hand a user for copy-paste, and what a
non-Runpod client speaks. If you are driving it yourself, the tool lanes wrap it:
`runpodctl serverless run <id> --input '{...}'` (v2.9.0+) does submit-and-poll with auth,
local payload validation and bounded waiting; the MCP lane has typed job tools.
```bash
# warm / small payloads (sync, 60s window):
curl -s https://api.runpod.ai/v2/<endpoint-id>/runsync \
-H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" \
-d '{"input": { ... }}'
# first / cold call — async, then poll:
curl -s https://api.runpod.ai/v2/<endpoint-id>/run -H "Authorization: Bearer $RUNPOD_API_KEY" \
-H "Content-Type: application/json" -d '{"input": { ... }}'
curl -s https://api.runpod.ai/v2/<endpoint-id>/status/<job-id> -H "Authorization: Bearer $RUNPOD_API_KEY"
```
- Body is always `{"input": {...}}`. **flash quirk:** a flash handler nests the
value under its parameter name → `{"input": {"<param>": {...}}}` (name the param
`input` to get the plain contract).
- Large inputs: pass a **URL**, not bytes (payload limits `/run` ~10MB, `/runsync`
~20MB); base64 rides the payload for small files.
- **Streaming** (`/stream/<job-id>`):
- Works **only when the handler is a generator** (`yield`s instead of `return`s).
- Submit with `/run`, then GET `/stream/<job-id>` in a loop — each call drains the
chunks buffered since the last one and returns `{"status", "stream":[{"output": <yield>}]}`.
- Stop when `status` is `COMPLETED`.
- Add `"return_aggregate_stream": True` to `runpod.serverless.start(...)` to *also*
expose the full list via `/run`/`/runsync`/`/status` (single chunk caps at 1MB).
- Worked example: golden path 12 (serverless streaming).
## 4. Verify with a real request — "ready" ≠ working
The **first call cold-starts** (image pull + model load), often exceeding
`runsync`'s 60s — use `/run` + poll `/status/<id>` for the first request, then
`runsync` once warm. Only report success once a real input returns the right
output. Bound any poll loop.
## 5. Deliver & tear down
Give the user the endpoint id + a copy-paste `curl` and the input schema (how to
pass a URL and/or base64). Scale-to-zero means it's safe to leave; delete with
`runpodctl serverless delete <id>` (or `flash app delete <app>`).
See [`golden-paths/03-whisper-endpoint/`](../../runpod/golden-paths/03-whisper-endpoint/README.md)
for a fully worked example of both the Hub and flash variants (a README plus one
file per variant).
reference/getting-started.md
# Getting started (auth & first-run setup)
Before any lane can act, its credential has to resolve. Everything Runpod-side
uses **one key**, `RUNPOD_API_KEY`; the companion CLIs use their own. Do the setup
for the lane you're about to use, then follow the development loop.
## Get the tools
Install only the lane you need; full per-OS matrices + source links live in each
lane's `SKILL.md` (linked). You don't need all of them.
| Lane | Quick install | Source / docs |
| --- | --- | --- |
| **runpodctl** (CLI) | `curl -sSL https://cli.runpod.net \| bash` (or `brew install runpod/runpodctl/runpodctl`) | [source](https://github.com/runpod/runpodctl) · [`runpodctl/SKILL.md`](../../runpodctl/SKILL.md) |
| **flash** (deploy your own code) | `uv tool install runpod-flash` (or `pip install runpod-flash`; Python 3.10–3.13) | [source](https://github.com/runpod/flash) · [`flash/SKILL.md`](../../flash/SKILL.md) |
| **runpod-mcp** (hosted) | `npx @runpod/mcp-server@latest add` (guided; OAuth) | [source](https://github.com/runpod/runpod-mcp) · [`runpod-mcp/SKILL.md`](../../runpod-mcp/SKILL.md) |
| **runpod-mcp** (local stdio) | `claude mcp add runpod -e RUNPOD_API_KEY=... -- npx -y @runpod/mcp-server` | same as above |
| **companion CLIs** (`hf`/`gh`/`docker`/`aws`) | per-tool; see the skill | [`companion-clis/SKILL.md`](../../companion-clis/SKILL.md) |
Nothing to install for the **hosted MCP** beyond configuring your client, and the
`runpod-usage` concepts need no install. After installing, set the key below.
**Want the Runpod tools in one go?** Copy-paste (drop any line you won't use):
```bash
npx skills add runpod/runpod-plugins-official # the skills — works with any agent
curl -sSL https://cli.runpod.net | bash # runpodctl — infra CLI (or: brew install runpod/runpodctl/runpodctl)
uv tool install runpod-flash # flash — deploy your own code (or: pip install runpod-flash)
npx @runpod/mcp-server@latest add # hosted MCP server — guided setup, OAuth
```
Companion CLIs (`docker`, `gh`, `hf`, `aws`) are separate and OS-specific — install only the
ones a task needs, per [`companion-clis/SKILL.md`](../../companion-clis/SKILL.md).
## The Runpod API key
Get it once at **https://console.runpod.io/user/settings** → API Keys. Then make
it resolvable for the lane. Resolution order (runpodctl, flash, and runpod-python
all use it): **`RUNPOD_API_KEY` env var → `.env` → `~/.runpod/config.toml`** (in that
file the key is the `apikey` field, TOML `apikey = '...'`). If you need the raw key
yourself (e.g. an `Authorization: Bearer` header for a direct API call), prefer
`$RUNPOD_API_KEY` and fall back to that field:
`KEY="${RUNPOD_API_KEY:-$(grep '^apikey' ~/.runpod/config.toml | sed "s/apikey = '//;s/'//")}"`.
| Lane | Set the key | Notes |
| --- | --- | --- |
| **runpodctl** | `export RUNPOD_API_KEY=...` | Non-interactive — runpodctl reads it. Best for agents/CI/scripts. |
| runpodctl (human) | `runpodctl doctor` | Interactive; stores the key **and** sets up SSH keys. Prompts, so not for agents. |
| **flash** | `export RUNPOD_API_KEY=...` | Or `flash login` — browser OAuth that **saves a real API key to `~/.runpod/config.toml`**, which runpodctl reads too, so one login serves both. Human-only (needs a browser). |
| **runpod-mcp (hosted)** | "Sign in with Runpod" OAuth on first connect | No key on disk. Or pass `Authorization: Bearer $RUNPOD_API_KEY`. |
| **runpod-mcp (local)** | `RUNPOD_API_KEY` env in the MCP client config | Forwarded to the API. |
**Rule (agents):** in automation, set the key with `export RUNPOD_API_KEY=...` (runpodctl and
flash both honor it — the closest thing to one-step setup); never run `runpodctl doctor`, which
prompts.
Context: the **hosted MCP is the exception** — it uses its own `/mcp` "Sign in with Runpod" OAuth
(or pass the same key as an `Authorization: Bearer` header). Authing the MCP does not set up the
CLIs, and the export does not authenticate the hosted MCP's OAuth.
## SSH (only needed for pods you exec into)
Pods created with `--ssh` (the runpodctl default) are reachable once you have a key
registered. **Register the key BEFORE creating the pod** — Runpod injects registered
keys at boot, so one added after the pod is running won't work until a restart.
- Check what's registered: `runpodctl ssh list-keys`.
- Register a key (do this first if none):
- Human: `runpodctl doctor` — generates + registers a key and stores the API key.
- Agent/scripted: `ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N ''` then
`runpodctl ssh add-key --key-file ~/.ssh/id_ed25519.pub` (or `--key "ssh-ed25519 …"`).
- Get connection details for a specific pod: `runpodctl ssh info <pod-id>` (prints
the ssh command + key path; does not connect).
- Agents connect non-interactively:
`ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -p <port> root@<host> 'cmd'`.
Serverless endpoints need no SSH.
## Companion CLI credentials (separate from RUNPOD_API_KEY)
Do NOT reuse the Runpod key for these — each has its own (see the `companion-clis`
skill for details):
- **`hf`** — HuggingFace token (`hf auth login`); read scope to pull, write to push.
- **`docker`** — Docker Hub PAT (`docker login`).
- **`gh`** — GitHub auth (`gh auth login`).
- **`aws`** — Runpod **S3** keys (user id `user_…` + S3 API key `rps_…`), with
`--region <dc> --endpoint-url https://s3api-<dc>.runpod.io/`. NOT AWS creds, NOT
the Runpod API key.
## Verify you're set up
```bash
export RUNPOD_API_KEY=...
runpodctl user # prints your account (confirms the key works)
runpodctl gpu list # confirms API access
```
Then pick your workload and follow `development-loop.md`.
reference/gotchas.md
# Common gotchas
Cross-cutting mistakes that bite people deploying on Runpod, as
symptom → cause → fix. See `docker.md` and `storage.md` for the full mechanics.
## Image built for the wrong CPU architecture
- **Symptom:** worker/pod fails to start, "exec format error", or the container
never comes up — often only after building on an Apple Silicon / ARM machine.
- **Cause:** the image was built for arm64. Runpod hosts are x86_64.
- **Fix:** rebuild with `docker build --platform=linux/amd64 ...` and re-push.
This is the #1 deploy failure.
## Using the `latest` tag
- **Symptom:** you push new code but workers keep running old behavior; you can't
tell which version is live.
- **Cause:** `latest` is mutable and Runpod caches images per host, so workers
serve a stale cached `latest`.
- **Fix:** use explicit tags (`v1.0.0`, `v1.0.1`) or pin the digest
(`name:tag@sha256:...`). Bump the tag on every change.
## Private image won't pull
- **Symptom:** endpoint/pod stuck initializing; logs show an image pull /
authentication error.
- **Cause:** the image is private and Runpod has no registry credentials.
- **Fix:** add container-registry credentials in Console → Settings → Container
Registry, then select them on the template/endpoint. Runpod uses
`docker login`-style creds. Headless equivalent: MCP
`create-container-registry-auth`, then pass the id as `containerRegistryAuthId`
on create-pod/create-endpoint.
- **Fix for AWS ECR:** you can register the repository instead of storing
credentials — MCP `create-registry-delegation` grants Runpod scoped pull access
to an ECR repo ARN. See that tool's parameter descriptions for the specifics.
## Image builds, then inference crashes with "Numpy is not available"
- **Symptom:** the image builds fine and the worker **starts**, but the first job
fails at inference with `RuntimeError: Numpy is not available` (often preceded at
import by `UserWarning: Failed to initialize NumPy: _ARRAY_API not found`).
- **Cause:** a NumPy 2.x / older-torch ABI mismatch. `pip install torch==2.2.2`
pulls the latest `numpy` (2.x) by default, but that torch wheel was built against
NumPy 1.x and can't use 2.x — any `.numpy()` call at inference throws.
- **Fix:** pin `numpy<2` in `requirements.txt` (or upgrade torch to a numpy-2-aware
build). Catch it **before deploy** by running the container locally against a
`test_input.json` (`docker.md` "Test the container locally") — the import warning
is the early tell; the crash only surfaces on an actual job, not at startup.
## `pip install runpod` fails on a `runpod/pytorch` base ("Cannot uninstall cryptography")
- **Symptom:** building/setting up on an official `runpod/pytorch` image, `pip install
runpod` (or `-r requirements.txt`) aborts with `Cannot uninstall cryptography 41.0.7 …
no RECORD file was found` (and/or `error: externally-managed-environment`).
- **Cause:** the base ships a **Debian-managed** `cryptography`, so pip can't uninstall
it to install the newer version `runpod` depends on. PEP-668 also marks the base pip
"externally managed".
- **Fix:** `pip install --break-system-packages --ignore-installed cryptography runpod`.
Install conflict-free deps (e.g. `faster-whisper`) first, separately, so
`--ignore-installed` stays scoped to the `cryptography`/`runpod` pair. Verified live
2026-07-10 (golden path 09); bake it into the Dockerfile as two `pip install` steps.
## Cold starts and timeouts
- **Symptom:** first request after idle is slow or times out; `/runsync` returns
a timeout while the model is still loading.
- **Cause:** cold start = container pull + start + model load. `runsync` has a
~60s ceiling; large model loads can exceed it.
- **Fix:** use `/run` + poll status (or raise the sync timeout); enable FlashBoot;
keep workers warm with `min_workers` / longer `idle_timeout`; load the model at
module level, not inside the handler; use cached models or a network volume so
the model isn't re-downloaded each start.
## Pod shows "Running" but ports/services aren't reachable yet
- **Symptom:** the pod is green/"Running" but the proxy URL 502s, the public IP
or TCP port mapping is blank, or JupyterLab is a white screen.
- **Cause:** "Running" only means the container exists — services and port
assignments take extra time to initialize.
- **Fix:** wait ~30–60s; check the **Telemetry** tab to confirm readiness; read
the assigned public IP / external TCP port from the **Connect** menu once
populated. Note external TCP port mappings change on every pod reset, and
Community Cloud public IPs can change on migration/restart.
- **Variant — after a `pod stop`→`start`, the external TCP port is reassigned AND
`runpodctl ssh info` can hand you a STALE port.** Live case (dev pod, 2026-07-10):
the pod first came up on port `17740`; after stop/start the *first* `ssh info`
still reported `17740` (all SSH connections refused) and reported `READY` while
sshd was still down for ~90s — a moment later a fresh `ssh info` returned the real
new port `12890`. **Fix:** after a restart don't trust the first `READY` or the
first port — re-run `ssh info` until you get a port that actually accepts an `ssh`
connection, then update your `~/.ssh/config` `Port`. This is what breaks VS Code
Remote-SSH reconnects (see golden path 06).
## Proxy 524 / 100-second timeout
- **Symptom:** requests through `https://<pod-id>-<port>.proxy.runpod.net` die at
~100s with a `524`.
- **Cause:** the HTTP proxy runs through Cloudflare, which caps connection time at
100 seconds.
- **Fix:** don't hold a single request open that long — return a job id and poll,
use background queues/progress endpoints, or use direct TCP (public IP) instead
of the proxy for long-lived connections.
## Network volume locked to a data center
- **Symptom:** can't get GPUs, or the endpoint won't schedule workers after
attaching a volume.
- **Cause:** a network volume lives in one DC; attaching it forces all compute
into that DC, shrinking GPU availability.
- **Fix:** confirm your target GPU exists in the volume's DC before attaching; or
attach multiple volumes from different DCs (one per DC) to spread workers —
remembering data doesn't sync between them automatically.
## Model not baked or mounted
- **Symptom:** handler errors with "model not found," or OOM/download stalls on
first request.
- **Cause:** the worker has no model — it wasn't baked into the image, cached, or
mounted from a volume.
- **Fix:** pick one delivery path (bake in, cached HF model at
`/runpod-volume/huggingface-cache/hub/`, or network volume) and make the handler
read from that path. For gated/private HF models set `HF_TOKEN`.
## Data disappeared after stop
- **Symptom:** files written during a run are gone after stopping the pod or the
worker scaling down.
- **Cause:** you wrote to container/ephemeral disk, which is wiped on stop.
- **Fix:** write to `/workspace` (pod volume disk) or a network volume
(`/runpod-volume` on serverless). Editing/resetting a pod also wipes anything
outside `/workspace`.
## Huge log or job output
- **Symptom:** logs stop appearing (throttled), or job submission/result is
rejected for size.
- **Cause:** excessive logging triggers throttling; job payloads exceed limits
(`/run` ~10 MB, `/runsync` ~20 MB).
- **Fix:** reduce log verbosity / use structured logging; write bulky output to a
network volume or external S3 and return a URL/reference instead of inlining it.
## GPU out of memory
- **Symptom:** job fails with an OOM/CUDA memory error.
- **Cause:** model or batch size exceeds the selected GPU's VRAM.
- **Fix:** reduce batch size / context length, or pick a larger-VRAM GPU. For
vLLM, lower `GPU_MEMORY_UTILIZATION` and/or `MAX_MODEL_LEN`.
## Serverless worker "ready" but jobs never run
- **Symptom:** the endpoint has `ready` workers but jobs sit `IN_QUEUE` with
`inProgress: 0` and never complete.
- **Cause:** a broken/mis-dispatching worker image, or workers `throttled` on a
scarce GPU pool the model doesn't need.
- **Fix:** switch to a different (maintained) Hub worker on a **broad, high-
availability GPU pool** — don't wait it out. Diagnose via the endpoint `/health`
worker counts — `runpodctl serverless health <id>` reads them for you (v2.9.0+) —
and then read the worker logs. There is no serverless worker-log command in
runpodctl and no worker-log path on REST v1; use the MCP `stream-worker-logs`
tool, the v2 REST logs path, or the Console Workers tab.
## Serverless job goes `IN_PROGRESS` then times out
- **Symptom:** a worker *picks up* the job (`IN_PROGRESS`) but never returns output;
the job fails with `"job timed out after 1 retries"` after ~30–50 s.
- **Cause:** this looks like a broken worker but is often a **job-reject by a healthy
worker** — usually a handler-signature or empty-`input` bug in *your* request, not the
image.
- **Fix:** get the **worker logs** (MCP `stream-worker-logs` or the Console) before
assuming the image is bad. Real case (flash endpoint, 2026-07-10): fitness checks
passed, then it logged `read() got an unexpected keyword argument …` /
`Job has missing field(s): id or input` — a handler-signature + empty-input bug (see
flash gotcha "Request body shape"). Only if the logs show a genuinely dead/looping
worker should you switch workers or try a different data center.
## Delete returns an error but succeeded
- **Symptom:** an MCP `delete-*` (or a DELETE call) returns `isError: true` /
"Unexpected end of JSON input".
- **Cause:** the Runpod REST API returns **204 No Content**; there's no JSON body
to parse.
- **Fix:** treat it as success; confirm with a follow-up `get-`/`list-` (the
resource should 404 / be absent).
## Handler swallowing errors
- **Symptom:** jobs report success but return nothing useful; failures are
invisible.
- **Cause:** a broad `try/except` suppresses the exception, so the SDK never marks
the job `FAILED`.
- **Fix:** return a structured error for graceful failures, or re-raise to flag
the job `FAILED`. Don't silently swallow.
reference/gpu-selection.md
# GPU selection
How to pick a GPU: size VRAM to the model first, then choose a tier/pool, then
worry about cloud, availability, and placement.
## Step 1: size VRAM to the model
VRAM is the usual bottleneck. Rough rules:
- **LLM inference (fp16):** ~**2 GB of VRAM per billion parameters**, plus headroom
for the KV cache / context and activations. A 7B model ≈ ~14 GB, 13B ≈ ~26 GB,
70B ≈ ~140 GB (needs multiple GPUs).
- **Quantization cuts this a lot.** A 4-bit quantized model needs roughly a quarter
of the fp16 weight memory — e.g. a 4-bit 70B fits in ~35 GB.
- **Training / fine-tuning** needs far more than inference — weights + gradients +
optimizer state + activations, often ~4x the inference figure for full fine-tuning.
LoRA/QLoRA reduce this substantially. Memory bandwidth also matters here.
- **Image models (SDXL, Flux):** ~8 GB minimum, but 16–24 GB gives headroom for
larger batches and LoRA training.
Always leave headroom above the raw weight size. When unsure, estimate with the
Hugging Face Model Memory calculator (`huggingface.co/spaces/hf-accelerate/model-memory-usage`)
or "Can it run LLM?" (`huggingface.co/spaces/Vokturz/can-it-run-llm`).
## Step 2: which GPU for an N-billion-param LLM (heuristic)
Sizes assume fp16 inference; quantize to drop a tier or two.
| Model size | VRAM (fp16) | Reasonable pick |
|-----------|-------------|-----------------|
| ≤ 7B | ~14 GB | `ADA_24` (RTX 4090) or `AMPERE_24` (L4/A5000/3090) |
| 13B | ~26 GB | `ADA_32_PRO` (RTX 5090) or `AMPERE_48` / `ADA_48_PRO` |
| 30–34B | ~60–70 GB | `AMPERE_80` (A100) or `ADA_80_PRO` (H100) |
| 70B | ~140 GB | 2x 80 GB, or one `HOPPER_141` (H200) / `BLACKWELL_180` (B200) |
| > 70B | 200 GB+ | multi-GPU 80 GB+, or Blackwell / H200 with `gpu_count` > 1 |
## Step 3: GPU tiers and pools
A **GPU pool** groups interchangeable GPUs by VRAM tier. Requesting a pool lets
Runpod pick any available GPU in that tier (better availability); pinning an exact
GPU type gives determinism but can be throttled when supply is tight. Pool names are
used by Serverless configs, the Runpod Hub, flash's `GpuGroup`, and the GraphQL API.
Pool reference (source of truth: `skills/flash/reference/api.md`):
| Pool | GPUs | VRAM |
|------|------|------|
| `ANY` | any available | varies |
| `AMPERE_16` | RTX A4000 / A4500 / RTX 4000 Ada / RTX 2000 Ada | 16 GB |
| `AMPERE_24` | RTX A5000 / L4 / RTX 3090 | 24 GB |
| `ADA_24` | RTX 4090 | 24 GB |
| `ADA_32_PRO` | RTX 5090 | 32 GB |
| `AMPERE_48` | A40 / RTX A6000 | 48 GB |
| `ADA_48_PRO` | RTX 6000 Ada / L40 / L40S | 48 GB |
| `AMPERE_80` | A100 (PCIe / SXM4) | 80 GB |
| `ADA_80_PRO` | H100 (PCIe / HBM3 / NVL 94 GB) | 80 GB+ |
| `HOPPER_141` | H200 | 141 GB |
| `BLACKWELL_96` | RTX PRO 6000 Blackwell | 96 GB |
| `BLACKWELL_180` | B200 | 180 GB |
Note: Serverless GPU config wants **pool IDs** (e.g. `ADA_24`), while Pod creation
and `runpodctl --gpu-id` use **GPU type IDs** (e.g. `NVIDIA A40`) — different
identifier spaces. The full per-model list (with exact display names and memory) is
in `docs/references/gpu-types.mdx`.
Rule of thumb: prefer **fewer high-end GPUs over more low-end GPUs**. One 80 GB card
usually beats two 40 GB cards for a model that fits.
## Step 4: Secure vs Community Cloud
- **Secure Cloud** — T3/T4 data centers, high redundancy, stable public IPs. Use for
production and sensitive data. Standard pricing.
- **Community Cloud** — vetted peer-to-peer hosts, cheaper, variable reliability;
public IPs can change on migrate/restart. Good for cost-sensitive, tolerant work.
(No new hosts are being onboarded; existing capacity remains.)
## Step 5: availability and multi-GPU selection
`runpodctl gpu list` reports on-demand `securePricePerHr` / `communityPricePerHr` per
GPU (explicitly `null` when that cloud doesn't offer it) plus a
`dataCenterAvailability[]` breakdown, so read cost and per-region stock straight from
it rather than assuming a lower tier is cheaper. Top-level `stockStatus` is only the
best status across DCs — use the per-DC breakdown when placement matters (`runpodctl
datacenter list` gives the same view from the DC side). Two scope limits: those are **pod
on-demand** rates (serverless bills per request-second), and the per-DC breakdown doesn't
say *which cloud* has the stock, so cost and placement are two separate reads rather than
one ranked list.
GPU supply fluctuates by tier and region. To avoid throttling:
- **List multiple GPU types / pools in priority order.** If the first choice is
unavailable, Runpod falls back to the next. On Serverless you can specify up to
three, in priority order.
- For endpoints with **5+ workers**, Runpod spreads workers across your prioritized
pools (most on the primary), reducing throttling. With fewer than 5 workers, all
use the highest-priority available type.
- (flash caveat: auto GPU switching by supply only kicks in when max workers ≥ 5.)
- Use `gpu_count` > 1 (Serverless) / multi-GPU Pods when a model exceeds a single
card's VRAM.
## Step 6: data-center placement
- Restricting an endpoint or Pod to specific data centers **shrinks the available
GPU pool** — allow all regions for maximum availability unless you have a reason
not to.
- Reasons to pin a region: lower latency to your users, data-residency/compliance,
or co-locating with a **network volume** (a volume ties the workload to its data
center). Some features (e.g. global networking) are only in a subset of regions.
- Regions span US (CA, GA, IL, KS, NC, TX, WA, etc.), EU (CZ, RO, IS, NO, SE, FR,
NL), and others. See `docs/pods/networking.mdx` for the current data-center list
and `companion-clis/SKILL.md` for datacenter IDs used with S3.
reference/networking.md
# Networking
How to reach a running workload over HTTP — Pods via the proxy or TCP, Serverless
via endpoint URLs.
## Pod HTTP proxy
The easiest way to expose a web service (REST API, web app, JupyterLab) from a Pod.
Add the internal port to **Expose HTTP Ports (Max 10)** on the Pod or template, then
reach it at:
```
https://<pod-id>-<internal-port>.proxy.runpod.net
```
Example — Pod `abc123xyz` running a server on port `4000`:
```
https://abc123xyz-4000.proxy.runpod.net
```
The `<internal-port>` is the port your service listens on *inside* the container,
not an external port number. Key behaviors:
- **Bind to `0.0.0.0`**, not `localhost`/`127.0.0.1`, or the proxy can't reach it.
- **HTTPS only** — the proxy terminates TLS even if your service speaks plain HTTP.
- **100-second timeout** — the route runs through Cloudflare, which closes idle/slow
connections at 100s with a `524`. For long work, return a job ID and poll, or use TCP.
- **Public, unauthenticated** — anyone with the URL can reach the service; the Pod ID
is obscurity, not access control, and the proxy adds no auth (same as any
port-forwarded service).
- **Rule:** when you hand a proxy URL to the user, state that it is public and unauthenticated.
- **Rule:** if the service handles anything sensitive, implement auth inside the service
itself (e.g. a login/token) — the platform adds none.
- "Running" (green) in the console does not mean the service is ready; the container
may still be starting.
## Pod TCP ports (direct public IP)
For non-HTTP protocols, WebSockets, databases, or lower latency, expose a **TCP**
port instead (add to **Expose TCP Ports**). Runpod assigns a public IP and an
external port, shown in the **Connect** menu under Direct TCP Ports:
```
TCP port 213.173.109.39:13007 -> :22
```
- The external port differs from the internal port and **changes whenever the Pod
resets**. Read it from the Connect menu.
- No automatic TLS — implement your own if sending sensitive data.
- Community Cloud public IPs may change on migrate/restart; Secure Cloud IPs are stable.
- UDP is not supported (TCP/HTTP only).
### Symmetric ports
When the external port must equal the internal port, request a port number **above
70000** in the TCP config (not a real port — a signal to allocate matching
internal/external ports). After creation, the assigned ports are in the Connect menu
and in env vars like `$RUNPOD_TCP_PORT_70000` that your app can read at runtime.
## Pod-to-Pod (global networking)
Pods with global networking share a private network and reach each other by internal
DNS — no public ports needed:
```
<pod-id>.runpod.internal
# e.g. a DB on port 5432: abc123xyz.runpod.internal:5432
```
NVIDIA GPU Pods only; available in a subset of data centers; ~100 Mbps between Pods.
Prefer this over exposing ports for internal services like databases.
## Serverless queue-based endpoints
Queue-based endpoints have a fixed set of operations under a common base:
```
https://api.runpod.ai/v2/<endpoint-id>/<operation>
```
| Operation | Method | Purpose |
|-----------|--------|---------|
| `/run` | POST | Submit an async job; returns a job ID immediately |
| `/runsync` | POST | Submit and wait for the result inline |
| `/status/<job-id>` | GET | Check status / fetch result of a job |
| `/stream/<job-id>` | GET | Stream incremental results |
| `/cancel/<job-id>` | POST | Cancel a queued or running job |
| `/retry/<job-id>` | POST | Requeue a failed/timed-out job |
| `/purge-queue` | POST | Drop all pending jobs |
| `/health` | GET | Worker + job stats for the endpoint |
The request body is a JSON object with an `input` key holding your handler's
parameters:
```bash
curl -X POST https://api.runpod.ai/v2/<endpoint-id>/runsync \
-H "Authorization: Bearer <RUNPOD_API_KEY>" \
-H "Content-Type: application/json" \
-d '{"input": {"prompt": "Hello, world!"}}'
```
- **Auth header:** `Authorization: Bearer <RUNPOD_API_KEY>` on every call.
- `/runsync` results are retained ~1 min (up to 5); `/run` results ~30 min via
`/status`. `/runsync` also has a ~60s client wait — for long/cold-start jobs use
`/run` + poll `/status`, or `runsync?wait=<ms>`.
## Serverless load-balanced endpoints
Load-balanced endpoints expose *your own* HTTP paths on a per-endpoint subdomain:
```
https://<endpoint-id>.api.runpod.ai/<your-custom-path>
```
Example paths from a FastAPI worker: `https://<endpoint-id>.api.runpod.ai/ping`,
`https://<endpoint-id>.api.runpod.ai/generate`.
- Same auth: `Authorization: Bearer <RUNPOD_API_KEY>`.
- Your worker must serve a `/ping` health check on `PORT_HEALTH` (`200` healthy,
`204` initializing). Main app port defaults to `80` (`PORT`).
- Limits: request timeout ~2 min if no worker is available, ~5.5 min processing per
request, 30 MB payload cap. Expect "no workers available" during cold start —
retry with backoff.
## Quick reference
```
Pod HTTP proxy https://<pod-id>-<internal-port>.proxy.runpod.net (HTTPS, 100s cap)
Pod TCP <public-ip>:<external-port> (from Connect menu)
Pod-to-Pod <pod-id>.runpod.internal (global networking)
Serverless (queue) https://api.runpod.ai/v2/<endpoint-id>/{run|runsync|status/<id>|health}
Serverless (LB) https://<endpoint-id>.api.runpod.ai/<path>
Auth (serverless) Authorization: Bearer <RUNPOD_API_KEY>
```
reference/on-pod-setup.md
# On-pod setup & install hygiene
How to install and configure software on a pod so it is reproducible, survives
restarts, and doesn't wedge on an interactive prompt. Applies to any workload;
run these over the SSH channel from the pod development loop (`pod-workflows.md`).
## Use package managers, not ad-hoc downloads
- **System packages → `apt`** (Debian/Ubuntu base images):
```bash
DEBIAN_FRONTEND=noninteractive apt-get update && apt-get install -y <pkgs>
```
- **Python → `uv`** (fast, reproducible, one static binary). Prefer it over bare
`pip`/`conda` for a *fresh* environment:
```bash
curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv /workspace/.venv && . /workspace/.venv/bin/activate
uv pip install <packages> # or: uv pip install -r requirements.txt
```
Put the venv on the volume (`/workspace/...`) so it persists.
> **On a PyTorch template image, do NOT make a bare `uv venv`.** Official
> templates ship `torch`/CUDA in the **system** Python; a fresh venv doesn't
> inherit it, so you'd reinstall multi-GB torch (or the app won't find CUDA).
> Install the app's deps into the **existing** interpreter instead:
> ```bash
> pip install --break-system-packages -r requirements.txt # newer images are
> # PEP 668 "externally-managed"; plain `pip install` errors without this flag
> # or, to still use uv: uv venv --system-site-packages && uv pip install ...
> ```
> pip correctly sees the pre-installed torch as already satisfied and only adds
> what's missing.
- **A vendor install script** (e.g. a service's official `install.sh`):
- **Rule:** pin a version so the setup is reproducible (`INSTALL_VERSION=…` or the
script's documented pin flag).
- **Exception:** if the script offers no way to pin, using it unpinned is acceptable
when that is the vendor's only supported install path.
## Make it non-interactive
An agent can't answer prompts. Always:
- Pass `-y` / `--yes`; set `DEBIAN_FRONTEND=noninteractive` for apt.
- Provide required config via env vars or flags, not TTY prompts.
- Avoid commands that open a pager or editor.
- **Pod `--env` vars are not in your SSH shell** (they go to PID 1). When you
launch a service or script over SSH that needs them, pass them explicitly:
`ssh <host> 'env VAR=val <command>'`. See `pod-workflows.md` step 5.
## Pin versions
- Pin package and image versions (`pkg==1.2.3`, `image:tag`, not `latest`) so a
rebuild or restart reproduces the same environment. See `gotchas.md`.
## Persist heavy artifacts on the volume
Anything large or slow to fetch goes on the network volume so it survives restarts
and isn't re-downloaded (see `storage.md`). Point caches at the volume:
```bash
export HF_HOME=/workspace/hf-cache # HuggingFace models/datasets
export UV_CACHE_DIR=/workspace/uv-cache # uv package cache
export PIP_CACHE_DIR=/workspace/pip-cache
```
## Run long work in the background, and log
Installs, model pulls, and servers can outlast a single SSH command. Background
them and log to the volume so you can poll progress and diagnose later:
```bash
ssh <host> '(long-running-setup > /workspace/setup.log 2>&1 &) '
ssh <host> 'tail -n 20 /workspace/setup.log' # poll in a SEPARATE ssh call
```
> **A long-lived server needs full detachment, not just `&`.** When the SSH
> channel closes it sends SIGHUP to its process group, which kills a plainly
> backgrounded `&`/subshell process. For anything that must keep running after you
> disconnect (a web server, `serve`, ComfyUI), start it detached and return
> immediately — do the readiness wait in a later, separate SSH call:
> ```bash
> ssh <host> 'setsid bash -c "<server-cmd>" > /workspace/svc.log 2>&1 < /dev/null &'
> ```
> `setsid` (new session, off the SSH TTY/pgroup) **and** `< /dev/null` are both
> needed. Do not `sleep` in the same invocation — it can drop the channel (exit 255).
## Be idempotent
Write setup so re-running it is safe (check-before-install, `mkdir -p`, `|| true`
on best-effort steps). You will run it again after a restart or a fix.
## Verify each step
Check the exit code / expected output of each install before moving on, rather
than chaining everything and discovering a failure at the end. Surface the actual
error; don't swallow it.
reference/pod-workflows.md
# The pod development loop
A repeatable loop for standing up, iterating on, and delivering **any** workload
on a Runpod pod — an Ollama server, ComfyUI, a Jupyter/dev box, a training run.
An agent has no Console and no web terminal, so it drives every step through the
CLI/API and a non-interactive SSH channel. Do the steps in order; each depends on
the previous.
## 1. Plan
Resolve the choices before creating anything (see `concepts.md`, `gpu-selection.md`,
`storage.md`):
- **Pod or serverless?** Long-lived / interactive / a persistent server → pod.
Request/response that scales to zero → serverless (different lane).
- **GPU / VRAM** for the workload.
- **Storage** — default to a **network volume** for anything worth keeping
(models, datasets, checkpoints, envs). See `storage.md`.
- **Which ports** the service listens on. These must be declared **at creation**.
## 2. Provision
Create the pod with everything the service needs baked in — **ports and env
cannot be added to a running pod** without a reset, so set them now. Enable SSH
(the agent's control channel) and a **terminate** guard for cost safety.
> **Pre-flight — register your SSH key BEFORE creating the pod.** Runpod injects
> your account's registered SSH keys into the pod **at boot**, so a key added
> *after* the pod is running won't work until a restart. If you'll exec into the
> pod (you will — it's the control channel), confirm a key is registered first:
>
> ```bash
> runpodctl ssh list-keys # already have one? then proceed
> # if not — human (interactive): registers a key + stores the API key
> runpodctl doctor
> # or agent/scripted: make a key, then register its PUBLIC half
> ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N ''
> runpodctl ssh add-key --key-file ~/.ssh/id_ed25519.pub
> ```
```bash
runpodctl pod create \
--name <name> --template-id <official-template> --gpu-id "<gpu>" \
--ports "<port>/http,22/tcp" \ # every port the service exposes (+22 for ssh)
--env '{"KEY":"VALUE"}' \ # goes to PID 1 — NOT the ssh shell (see step 5)
--network-volume-id <id> --volume-mount-path /workspace \
--terminate-after <iso8601> # cost guard: TERMINATES the pod
```
`--terminate-after` deletes the pod at that time; `--stop-after` only *stops* it
(you keep paying for disk/volume), so use `--terminate-after` as the real guard.
Find a data center that has both your GPU and (for co-location) your volume with
`runpodctl datacenter list` — its output includes per-DC GPU availability.
(MCP's `create-pod` also sets `ports`/`env`; use runpodctl when you need a
template, CPU pod, or multi-GPU list — see the router.)
## 3. Connect (the control channel)
```bash
runpodctl pod get <pod-id> # poll until it is running
runpodctl ssh info <pod-id> # prints the ssh command + key (does not connect)
```
Then run commands **non-interactively** — this is how an agent works a pod:
```bash
ssh <user>@<host> -p <port> -i <key> 'set -e; <commands>'
```
Keep it one command (or a heredoc script) per step so you can check the exit code
and output before moving on.
## 4. Set up
Install dependencies idempotently, using package managers and persisting heavy
artifacts to the volume. See `on-pod-setup.md` for the hygiene rules (apt for
system, `uv` for Python, pin versions, background long installs, cache on the
volume).
## 5. Run the service
Start it **bound to `0.0.0.0`** (not localhost, or the proxy can't reach it) on
the exposed port, in the background, logging to the volume.
> **Gotcha — creation env vars are NOT in your SSH shell.** The `--env` vars from
> step 2 are injected into the pod's **main process (PID 1)**, not into SSH login
> shells. A service you launch over SSH will see them **empty** — so a service
> that reads `OLLAMA_HOST`/`HOST`/`PORT` from the environment silently binds to
> localhost and the proxy 502s. **Pass the env explicitly** when you launch:
A long-lived server must be **fully detached**, or SSH sends SIGHUP on channel
close and kills it. Use `setsid` + `< /dev/null`, return immediately, and poll in
a *separate* call.
**Do not `sleep` in the same invocation** — it can drop the channel:
```bash
ssh <host> 'setsid bash -c "env HOST_VAR=0.0.0.0 OTHER=val <service-command>" \
> /workspace/<svc>.log 2>&1 < /dev/null &'
```
(Two things at once: pass env explicitly because `--env` isn't in this shell, and
detach with `setsid`/`</dev/null` so it survives disconnect.)
## 6. Verify readiness — not "Running"
A pod showing "Running" does **not** mean the service answers. Poll a real health
check from **outside**, through the proxy URL, until it succeeds:
```bash
for i in $(seq 1 120); do curl -sf https://<pod-id>-<port>.proxy.runpod.net/<health-path> && break; sleep 5; done # bounded ~10min; if it never answers, the service didn't come up — check logs
```
Expect the **proxy itself to return 502 for the first ~30–60s** while the service
finishes importing / CUDA-initializing — that's normal warm-up, not failure, so
keep polling (with a timeout). Only report success once it passes. If it never
does, read the service log on the volume to diagnose.
## 7. Iterate
Re-run setup/service commands over the same SSH channel. Logs and artifacts on the
volume survive restarts. Use a framework's hot-reload/dev mode when it has one
(e.g. `flash dev` in the flash lane).
## 8. Escalate on manual steps
If a step genuinely needs a human — OAuth sign-in, a quota/capacity increase, a
license/EULA click, a missing credential, a payment issue — **stop and tell the
user exactly what is blocked and what you need**. Do not silently retry or fake
progress.
## 9. Deliver & tear down
- Return the **access URL** (`https://<pod-id>-<port>.proxy.runpod.net`) and a
sample request.
- Note any security caveat (proxy URLs are public; most dev servers have no auth).
- Tell the user how to stop/terminate; data on the network volume persists. The
`--stop-after`/`--terminate-after` guard from step 2 is the backstop.
## The loop in one line
**plan → provision (ports+env+volume+ssh) → connect (ssh-exec) → set up → run
(bind 0.0.0.0) → poll readiness → iterate → escalate if blocked → deliver + tear
down.**
reference/storage.md
# Where data lives on Runpod
Pick storage based on whether data must survive a stop, be shared across
machines, and where your compute runs. There are three layers.
## Default: prefer a network volume
**Unless the user says otherwise, put anything worth keeping on a network
volume** — models, datasets, checkpoints, environments, caches. It survives pod
stop/terminate and serverless scale-to-zero, is reusable across pods and
endpoints, and means expensive downloads happen once. Create it (and the volume's
data center) *before* the compute, then place the compute in that same DC.
Use the faster, non-persistent layers deliberately, not by default: container /
ephemeral disk for throwaway scratch, and pod volume disk when a single pod's data
doesn't need to outlive it. When in doubt, choose the network volume.
**One exception: model weights for GPU serverless.** If the model is on HuggingFace,
the host-side cache beats a volume — faster cold starts, no download billing, and no
data-center pin narrowing your GPU availability. See
[Getting a model to the worker](#getting-a-model-to-the-worker) below; the default
above is about datasets, checkpoints, and everything else worth keeping.
## The three layers
### Container / ephemeral disk
- Exists only while the container runs; **wiped when the pod stops or the worker
scales down**. Fastest (locally attached).
- Everything a serverless handler writes goes here by default.
- On pods, editing/resetting a running pod also erases it — only `/workspace`
survives (see below).
- Use for: OS, temp files, scratch, caches you don't need to keep.
### Volume disk (pods only)
- Persistent local disk mounted at `/workspace`. Retained across stop/restart,
but **deleted when the pod is terminated**. Not shareable.
- Roughly $0.10/GB/month running, $0.20/GB/month while stopped. Can be increased
(never decreased); optionally encrypted at rest.
- Use for: models, datasets, and checkpoints you reuse across sessions on one pod.
### Network volume (shared, portable)
- Persistent storage that lives **independently of any compute**. Attachable to
multiple pods and to serverless endpoints; survives termination/scale-to-zero.
- NVMe-backed (roughly 200–400 MB/s, higher peak). Standard and High-Performance
tiers. Pricing ~$0.07/GB/month for the first 1 TB, ~$0.05/GB/month beyond.
- **Picking the tier:** the runpod-mcp `create-network-volume` tool takes
`volumeType` (`STANDARD` | `HIGH_PERFORMANCE`); omit it for the data center's default.
`runpodctl` has no tier flag, so from the CLI request **High-Performance** via the
console (a ⚡ data center → toggle) or a raw v2 REST call (`POST
https://v2-rest.runpod.io/v2/network-volumes` with `"type":"HIGH_PERFORMANCE"`); the tier
is immutable after creation. High-Performance (~3× throughput / 4× IOPS) is worth it when
I/O is the bottleneck — training, checkpointing, many small files. Launch: golden path
[21 — storage tiers](../../runpod/golden-paths/21-storage-tiers.md); more:
[high-performance storage docs](https://docs.runpod.io/storage/high-performance-storage).
- **Data-center-scoped** — a volume lives in one DC. See the constraint below.
- Mount paths:
- Pods: mounts at `/workspace`, **replacing** the volume disk. Must be attached
at pod creation and cannot be detached later.
- Serverless: mounts at `/runpod-volume`.
- Use for: sharing models/datasets across workers or pods, and anything that must
outlive a single machine.
### Data-center constraint (important)
A network volume is pinned to its data center, so any compute that mounts it must
be placed in that **same DC**. That narrows GPU availability. To improve
availability, attach multiple volumes from different DCs (one per DC) — but data
does **not** sync between them automatically; copy it yourself (S3 API /
runpodctl). **Do not write to one volume from multiple workers concurrently** — it can corrupt data.
## Getting a model to the worker
Four ways to make a model available; choose by size, privacy, and how often it
changes. On HuggingFace and running GPU serverless → **cached model**; your own
artifact → **Model Repository** (or a volume if you want to manage the files
yourself); need a reproducible image → **bake**.
| Option | How | Best when |
| --- | --- | --- |
| Bake into image | `COPY` model files, or download during `docker build` | Small/private models not on Hugging Face; fully reproducible images |
| Cached model (HF) | Attach a Hugging Face model to the endpoint (`--model-reference`); Runpod caches it host-side | Public/gated/private HF models; fastest cold starts, smaller images |
| Network volume | Pre-load the model onto a volume, mount it | Large shared models reused across many workers/pods |
| Model Repository | Upload your own artifacts with `runpodctl model add`; Runpod versions + distributes them | Your own/custom models not on HF, without image bloat or a DC-pinned volume |
Full comparison + commands: [`runpodctl/reference/model-caching.md`](../../runpodctl/reference/model-caching.md).
Trade-off: baking bloats the image and slows pulls; cached models and volumes
decouple the model from the image so it loads without re-downloading, cutting cold
starts. With cached models you are **not billed** for download time, and it works
for GPU serverless endpoints.
### HuggingFace cache directory
Runpod's cached-model feature stores models in the standard HF cache layout at:
```
/runpod-volume/huggingface-cache/hub/
```
Structure follows HF conventions — `models--{org}--{name}/snapshots/{hash}/`
(slashes in the model name become `--`), e.g.
`/runpod-volume/huggingface-cache/hub/models--Qwen--Qwen2.5-0.5B-Instruct/snapshots/<hash>/`.
Anything that reads the HF cache (Transformers, vLLM, …) picks it up automatically.
Baking into a custom image instead? Point `HF_HOME` at your model dir.
## Accessing a network volume over S3
Runpod exposes network volumes through its **own** S3-compatible API (not AWS).
The bucket name is the network volume ID. Every request needs both a region and
an endpoint URL derived from the volume's data center:
```
--region <DC> --endpoint-url https://s3api-<DC>.runpod.io/
```
Credentials (from Console → Settings → S3 API Keys):
- `AWS_ACCESS_KEY_ID` = your Runpod **user ID** (format `user_...`)
- `AWS_SECRET_ACCESS_KEY` = an **S3 API key** you generate (format `rps_...`, shown once)
```bash
aws s3 ls \
--region CA-2 \
--endpoint-url https://s3api-CA-2.runpod.io/ \
s3://NETWORK_VOLUME_ID/
```
Path mapping: `/workspace/my-folder/file.txt` on a pod ==
`s3://NETWORK_VOLUME_ID/my-folder/file.txt` over S3. (See the `companion-clis`
skill for full AWS CLI usage; each volume on the Storage page shows a pre-filled
`aws s3` command with the right region/endpoint.)
## Quick decision guide
- Throwaway scratch during a run → container disk (nothing to configure).
- Keep data between sessions on one pod → volume disk (`/workspace`).
- Share models/data across workers or pods, or persist past termination →
network volume — and put your compute in the volume's DC.
- Public/gated HF model on serverless → cached model, not a bake.
- Big files that blow past API payload limits → network volume or external S3;
return references, not bytes.
SKILL.md
---
name: runpod-usage
description: >-
How Runpod works and how to work it — pods vs serverless, GPU/VRAM selection,
storage, building a container, networking, plus the agentic pod development loop
(provision → ssh-exec → set up → poll readiness) and on-pod install hygiene
(uv/apt). Use to answer "how does X work", "which GPU", "how do I build a
container", or "how do I stand up a workload on a pod". Guidance, not a tool —
execute with runpodctl, runpod-mcp, or flash.
metadata:
author: runpod
version: "1.2.0" # x-release-please-version
license: Apache-2.0
---
# Runpod usage (concepts)
Background knowledge for making the right choice before you act. This skill runs
nothing — once you know what to do, execute with **runpod-mcp**/**runpodctl**
(infra), **flash** (your own code), or **companion-clis** (models/images/data).
**This skill explains; the golden paths demonstrate.** When the question is really "how do I
do X" rather than "how does X work", the verified end-to-end example is the faster answer —
[runpod/golden-paths/README.md](../runpod/golden-paths/README.md). Read the concept here, then
follow the path.
Read the one reference file that matches the question:
| Question | Read |
| --- | --- |
| First-run setup / auth — get + set `RUNPOD_API_KEY`, SSH, companion creds | `reference/getting-started.md` |
| Pods vs serverless, workers, cold starts, FlashBoot, queue vs load-balanced | `reference/concepts.md` |
| **The development loop for ANY workload (start here)** — plan → prefer prebuilt → provision → verify → teardown | `reference/development-loop.md` |
| **Stand up / iterate a workload on a pod** — the pod sub-loop | `reference/pod-workflows.md` |
| **Deploy / iterate a serverless endpoint** — Hub vs flash vs custom, invoke + verify | `reference/endpoint-workflows.md` |
| **Install software on a pod** — package hygiene, `uv`, non-interactive, caching | `reference/on-pod-setup.md` |
| Build a Docker image Runpod can run (handler contract, Dockerfile, `--platform=linux/amd64`) | `reference/docker.md` |
| **How to build an image well** — base image, layering, bake-in vs volume, pod vs serverless (queue/LB) contract | `reference/building-images.md` |
| Where data lives — container disk vs network volume, model caching, S3 access | `reference/storage.md` |
| Which GPU / how much VRAM / cost & availability / data centers | `reference/gpu-selection.md` |
| Reaching a pod or endpoint over HTTP (proxy URLs, exposed ports) | `reference/networking.md` |
| Common mistakes and how to avoid them | `reference/gotchas.md` |