assets/preflight.sh
#!/usr/bin/env bash
# Fast, automatable subset of the G1-G10 checks in SKILL.md.
# Output contract: every result line starts with its G-number.
# G1/G9 emit PASS/FAIL/WARN verdicts; G3/G4 emit INFO lines
# (raw readings needing judgment — see SKILL.md G3/G4). G7 emits
# PASS/WARN for hardware capability only — it does NOT verify
# the kernel target arch (sm_121a), which stays a manual step,
# see gotcha-checks.md G7.
# Non-automatable gotchas (G2 flash-attn presence/Unsloth
# override check, G6 process survey, G8 upstream-issue lookup,
# G10 config review) are not covered here — see
# references/gotcha-checks.md for those.
# Usage: bash preflight.sh
set -uo pipefail
echo "== G1: CUDA 12/13 ABI =="
g1_cuda_ver=$(python3 -c "import torch; print(torch.version.cuda or '')" 2>/dev/null)
if [ -z "$g1_cuda_ver" ]; then
echo "G1 SKIP: torch not importable"
elif [[ "$g1_cuda_ver" == 13* ]]; then
echo "G1 PASS: torch built against CUDA $g1_cuda_ver"
else
echo "G1 FAIL: torch built against CUDA $g1_cuda_ver (need 13.x — see gotcha G1)"
fi
echo "== G3: UMA headroom (raw reading) =="
free -g | awk 'NR==2 {print "G3 INFO: free/used (GB):", $4, $3}'
echo "== G4: thermal snapshot (raw reading) =="
nvidia-smi --query-gpu=temperature.gpu,power.draw --format=csv,noheader 2>/dev/null \
| sed 's/^/G4 INFO: /' || echo "G4 SKIP: nvidia-smi not available"
echo "== G7: SM121 capability (kernel target arch NOT checked here) =="
python3 -c "
import torch
cap = torch.cuda.get_device_capability()
print('G7 PASS: SM121 hardware capability confirmed (partial — verify the kernel target arch (sm_121a) manually, see gotcha-checks.md G7)' if cap == (12, 1) else f'G7 WARN: unexpected capability {cap}')
" 2>/dev/null || echo "G7 SKIP: torch not importable"
echo "== G9: container vs bare pip =="
if [ -f /.dockerenv ] || [ -f /run/.containerenv ]; then
echo "G9 PASS: running inside a container (Docker/Podman marker file present)"
elif grep -qE '(docker|containerd|kubepods)' /proc/1/cgroup 2>/dev/null; then
echo "G9 PASS: running inside a container (cgroup marker present)"
elif [ -r /proc/1/cgroup ]; then
echo "G9 UNKNOWN: no container marker matched — this does not prove a bare host (cgroup v2 and some runtimes hide the marker); verify manually before assuming bare-host, prefer an NGC or Unsloth container if unsure"
else
echo "G9 SKIP: /proc/1/cgroup not readable"
fi
echo "Full details and remaining gotchas: references/gotcha-checks.md"
references/gotcha-checks.md
Last verified: 2026-07-14 — refresh when CUDA, PyTorch, or the
DGX Spark stack ships a new major version.
Runnable check commands for each gotcha in `SKILL.md`, keyed by
G-number. Run the check before assuming the gotcha is the cause;
each command is safe to run read-only unless noted.
## G1: CUDA 12/13 ABI Mismatch
```bash
python3 -c "import torch; print(torch.version.cuda)"
python -c "import ctypes; ctypes.CDLL('libcudart.so.13')"
ldconfig -p | grep libcudart
```
`torch.version.cuda` is the authoritative signal: it should
report `13.x`. Don't rely on `pip show torch | grep cu130` — NGC
container builds (e.g. `nvcr.io/nvidia/pytorch:25.11-py3`) build
torch internally against CUDA 13 with no `+cu130` wheel tag, so
the absence of a `cu130` tag in `pip show` output is NOT a
failure by itself on those containers. The `ctypes` load
confirms `libcudart.so.13` is actually present on the system; if
it raises `OSError`, the driver/runtime install is the problem,
not the wheel. `ldconfig -p` lists every CUDA runtime version
currently registered — a `libcudart.so.12` entry alongside
`libcudart.so.13` is a common leftover from an earlier install
and a likely ABI-mismatch source.
## G2: flash-attn — Presence Check and the Unsloth Override
```bash
python -c "import flash_attn; print(flash_attn.__version__)" 2>&1 | tail -5
python -c "import torch; print(torch.backends.cuda.flash_sdp_enabled())"
```
Don't assume this import fails — it depends on the environment.
On a **bare-pip** install it's expected to fail (no aarch64/
sm_121 wheel exists), confirming no training code path silently
depends on flash-attn. On an **NGC PyTorch container**
(confirmed on `nvcr.io/nvidia/pytorch:25.09-py3`), flash-attn
2.7.4.post1 ships pre-bundled and a live `flash_attn_func`
kernel call on GB10 (capability `(12, 1)`) executes successfully
with correct output shape — the "no sm_121 kernels" framing only
applies to building it yourself, not to what these containers
already carry.
**The actual trap on a container with flash-attn present:**
Unsloth's import-time patch banner reports `FA2 = True` and
auto-prefers flash-attn over SDPA — including when the caller
explicitly requested `attn_implementation="sdpa"`. Unsloth's
loader (`unsloth/models/llama.py`, 2026.7.2) calls its internal
`resolve_attention_implementation(...)` without forwarding the
caller's `attn_implementation` as that function's
`requested_attn_implementation` parameter, then pops the kwarg
outright with a `# No need since we auto call it` comment —
silently discarding whatever was requested. Confirmed via a live
load: `attn_implementation="sdpa"` passed explicitly still
resolved to `model.config._attn_implementation ==
"flash_attention_2"`.
**The only working override** on this Unsloth version, for any
model routed through Unsloth's llama-architecture loader
(`unsloth/models/llama.py` — this covers more than one base
model family; see `finetuning-method-selection`'s model catalog
for which) when flash-attn is importable, is a monkeypatch
before calling `from_pretrained`:
```python
import unsloth.models._utils as _unsloth_utils
_unsloth_utils.HAS_FLASH_ATTENTION = False
```
This forces `resolve_attention_implementation`'s auto-resolution
down its `elif supports_sdpa:` branch instead of the flash-attn
branch — confirmed working (`config._attn_implementation ==
"sdpa"` after the monkeypatch). On plain TRL/PEFT (bypassing
Unsloth entirely — see `lora-qlora-recipes`'
`references/unsloth-trl-mapping.md` escape hatch),
`attn_implementation="sdpa"` passed to
`AutoModelForCausalLM.from_pretrained` **is** honored correctly;
this gap is Unsloth-specific, not a general TRL/transformers
issue.
## G3: UMA OOM Below 128GB
```bash
free -g
cat /proc/meminfo | grep -i huge
```
Read real memory pressure from `free -g`, not `nvidia-smi` —
unified memory means CUDA allocations and host RAM share one
pool, and `nvidia-smi` only reports the CUDA side. On some
driver/setup combinations, `nvidia-smi --query-gpu=memory.used,
memory.total --format=csv` doesn't just undercount — it returns
`[N/A], [N/A]` for the whole-GPU memory query outright. A script
that greps for a numeric value there gets nothing, not a
misleading number; don't build a headroom check on that query on
this hardware. If `free -g` shows most of the 128GB consumed
while a load is failing, reclaim it:
```bash
sync
echo 3 > /proc/sys/vm/drop_caches
```
This needs root and flushes the page cache system-wide — every
process on the box loses cached file reads, not just the
training job. Run it between runs when memory looks pinned by
stale mmap'd pages, not as a routine step during training.
## G4: Thermal Throttling
```bash
nvidia-smi --query-gpu=temperature.gpu,power.draw --format=csv -l 5
```
Sampling loop (`-l 5` = every 5 seconds); let it run for at
least 10–15 minutes on a representative workload before judging.
Rated power is 240W; if draw plateaus around 100W while
temperature keeps climbing or has already plateaued high, the
box is throttling. A rising temperature with power still near
peak is not yet a throttle event — keep watching. Interrupt with
Ctrl-C when done; this does not need root.
## G5: Bandwidth Ceiling
**Intrusive, unlike the other checks in this file — do not run
against an active workload.** It allocates ~4GB on a UMA system
where GPU and host memory share one pool, and repeatedly clones
that tensor. Run only on an idle host; a box with another job
already using most of its 128GB headroom can OOM that job.
```bash
python -c "
import torch, time
x = torch.randn(1_000_000_000, device='cuda', dtype=torch.float32)
torch.cuda.synchronize()
t0 = time.time()
for _ in range(20):
y = x.clone()
torch.cuda.synchronize()
dt = time.time() - t0
gbps = (x.numel() * 4 * 2 * 20) / dt / 1e9
print(f'{gbps:.1f} GB/s')
del x, y
torch.cuda.empty_cache()
"
```
Expect roughly 180–192 GB/s, not the 273 GB/s spec figure. If a
throughput plan assumed the spec number, revise it against this
measured range before committing to a schedule.
## G6: Global UMA Resource Contention
```bash
nvidia-smi
ps aux | grep -E "vllm|ollama|python.*train" | grep -v grep
```
List every GPU-resident process before starting a long run.
`nvidia-smi` shows per-process memory; the `ps` filter catches
inference servers (vLLM, Ollama) that may not show up clearly in
`nvidia-smi` output on unified memory. The one-heavy-job rule
applies to **uncapped or near-capacity** workloads — stop
unrelated servers before a run that needs the full 128GB pool,
or that runs alongside another process without an explicit
memory cap. It does not apply to small, memory-capped
coexistence: a <4GB LoRA fine-tune has been observed running
cleanly alongside a vLLM server started with
`--gpu-memory-utilization 0.5` (or lower) on the same box — check
the other process's own memory cap, not just its presence,
before deciding whether it needs to be stopped.
## G7: NVFP4 Slower Than FP8 on SM121
```bash
python -c "import torch; print(torch.cuda.get_device_capability())"
```
Expect `(12, 1)` on GB10 — this confirms SM121. SM121 lacks a
native `cvt.e2m1x2` conversion path, so NVFP4 kernels not
compiled for the `sm_121a` target fall back to a slower path,
roughly 32% behind FP8. Check the kernel build's target arch
(often `TORCH_CUDA_ARCH_LIST` or a similar build flag) before
assuming NVFP4 is the faster choice on this hardware.
## G8: Stale Official Playbooks
```bash
gh issue list --repo NVIDIA/dgx-spark-playbooks --state open --limit 20
```
Requires the `gh` CLI authenticated, or substitute a browser
visit to the same URL. Scan open issues for the specific
playbook and command being followed before trusting it verbatim
for a long or expensive run.
## G9: Container-First, Not Bare Pip
```bash
if [ -f /.dockerenv ] || [ -f /run/.containerenv ]; then
echo "in container (marker file)"
elif grep -qE '(docker|containerd|kubepods)' /proc/1/cgroup 2>/dev/null; then
echo "in container (cgroup marker)"
else
echo "unknown — no container marker matched, this does not prove a bare host"
fi
pip list 2>/dev/null | grep -E "^(torch|triton|xformers|transformers) "
```
The first block checks Docker's `/.dockerenv` and Podman's
`/run/.containerenv` marker files, then falls back to a
cgroup-string check — a single `grep docker /proc/1/cgroup` alone
is not reliable: cgroup v2 layouts and some runtimes/namespaces
hide the runtime name, so a failed match is "unknown," never proof
of a bare host. The second command lists the versions actually
installed — compare against the pinned set in an NGC or Unsloth
image if running bare pip, to catch drift early rather than at
import time.
## G10: Dual-Spark Is DDP/FSDP Only
```bash
python -c "
import os
print('WORLD_SIZE:', os.environ.get('WORLD_SIZE'))
print('parallelism strategy check: confirm config uses DDP or FSDP, not TP/tensor_parallel')
"
rg -l --iglob '*.yaml' --iglob '*.yml' -e 'tensor_parallel|tp_size|tensor-parallel' . \
|| grep -rlE "tensor_parallel|tp_size|tensor-parallel" --include='*.yaml' --include='*.yml' .
```
Search recursively, not just the current directory — a
non-recursive `*.yaml *.yml` glob misses nested configs, and a
redirected/suppressed error there reads as "no TP configuration"
when it may just be "wrong directory." If the workload's config
path is already known, pass it explicitly instead of searching.
Any match on a two-Spark job is a configuration error — switch to
DDP or FSDP before launching; TP is not viable across the
ConnectX-7 link on this hardware.
SKILL.md
---
name: spark-training-gotchas
description: Preflight and diagnose the ten known failure modes for ML training on NVIDIA DGX Spark. Use when a training run on DGX Spark fails to start, OOMs below the 128GB limit, slows down mid-run, or before any multi-hour training job on GB10.
---
# Spark Training Gotchas
DGX Spark's GB10 chip (Grace Blackwell, SM121, 128GB unified
memory, aarch64) has ten recurring failure modes across
launch, memory, thermals, bandwidth, and precision. Each is
named G1–G10 so it can be checked by number — the numbering
is load-bearing for tooling that runs these checks. Read this
before a long run, not after hour six.
## When to Use This Skill
- A training run fails to start, with an import error or a
segfault that doesn't point at the real cause.
- A run OOMs while `nvidia-smi` still shows headroom.
- Throughput degrades partway through a run that started fine.
- Before any multi-hour or multi-epoch job on GB10.
- Wiring two Sparks together, before picking a parallelism
strategy.
- Choosing between FP8 and NVFP4 for a Spark-hosted run.
## Common Issues Quick Reference
| # | Symptom | Fix |
|---|---|---|
| G1 | undefined symbol / segfault | cu130 wheel or container |
| G2 | flash-attn wrong backend used | skip pip build; monkeypatch on NGC |
| G3 | OOM despite headroom | drop page cache |
| G4 | throughput drop / reboot | expect ~100W sustained cap |
| G5 | memory-bound step slow | budget 180–192 GB/s |
| G6 | cache evicted mid-run | one GPU server at a time |
| G7 | NVFP4 slower than FP8 | stay FP8 unless `sm_121a` |
| G8 | playbook fails outright | check upstream issues |
| G9 | env breaks after install | use a container |
| G10 | 2-Spark TP hangs | DDP/FSDP only, never TP |
## The Ten Gotchas
### G1: CUDA 12/13 ABI Mismatch
- **SYMPTOM:** `ImportError: undefined symbol` naming a CUDA
function, or a segfault on the first `.cuda()` call.
- **CAUSE:** most PyPI wheels link `libcudart.so.12`; Spark
ships CUDA 13. pip never checks CUDA ABI, so it surfaces
only at import or first kernel launch.
- **CHECK:** `references/gotcha-checks.md` G1 — the wheel's
CUDA build tag.
- **FIX:** reinstall from `download.pytorch.org/whl/cu130` or
use a matched container.
### G2: flash-attn — Skip the pip Build, Watch Unsloth's Auto-Detect
- **SYMPTOM:** `pip install flash-attn` still fails/hangs.
Unsloth may also silently train flash-attn over an
explicitly requested SDPA.
- **CAUSE:** no aarch64/sm_121 wheel for bare pip — but NGC
containers ship a working SM121 flash-attn, and Unsloth
auto-prefers it, dropping `attn_implementation="sdpa"`.
- **CHECK:** `references/gotcha-checks.md` G2 — is flash-attn
already present and working.
- **FIX:** bare pip — skip flash-attn, use SDPA (unchanged). On
NGC — the only reliable override is the monkeypatch in
`references/gotcha-checks.md` G2.
### G3: UMA OOM Below 128GB
- **SYMPTOM:** OOM during model load/training while
`nvidia-smi` still reports free memory under the 128GB cap
— or, on some setups, `[N/A]` outright instead of a number.
- **CAUSE:** mmap and the CUDA allocator double-count pages
during safetensors load; QLoRA can OOM *earlier* than bf16
since dequantization adds transient allocs.
- **CHECK:** `references/gotcha-checks.md` G3 — read `free -g`
and `/proc/meminfo`, not `nvidia-smi`.
- **FIX:** drop the page cache with
`sync; echo 3 > /proc/sys/vm/drop_caches` — needs root, a
between-run reset, not a mid-training step.
### G4: Thermal Throttling
- **SYMPTOM:** throughput drops partway through a multi-hour
run, or the box spontaneously reboots under sustained load.
- **CAUSE:** sustained power draw caps around 100W versus the
240W rated figure; long runs push into that ceiling and
throttle or, sometimes, reboot.
- **CHECK:** `references/gotcha-checks.md` G4 — sample
`nvidia-smi --query-gpu=temperature.gpu,power.draw`.
- **FIX:** if power plateaus under 240W while temperature
climbs, treat throttling as the cause; improve cooling or
cap run length.
### G5: Bandwidth Ceiling
- **SYMPTOM:** memory-bound workloads, decode-heavy RL loops
especially, plateau well below expected throughput.
- **CAUSE:** 273 GB/s is a spec ceiling, not sustained;
measured bandwidth runs 180–192 GB/s.
- **CHECK:** `references/gotcha-checks.md` G5 — observed step
time vs. the measured range, not spec.
- **FIX:** budget throughput from 180–192 GB/s; revise a plan
built on the 273 GB/s figure.
### G6: Global UMA Resource Contention
- **SYMPTOM:** a process's KV cache/weights get evicted
mid-run silently, no OOM in its own logs.
- **CAUSE:** unified memory is
one global pool; an uncapped
or near-capacity process
competes with anything else
and can evict it. A small,
bounded workload doesn't — a
<4GB LoRA coexists fine
alongside vLLM capped at
`gpu-memory-utilization<=0.5`.
- **CHECK:** `references/gotcha-checks.md`
G6 — other GPU-resident
processes and whether
capped.
- **FIX:** the one-heavy-job
rule applies to **uncapped or
near-capacity** workloads —
cap or stop unrelated servers
first. A small, capped
workload need not
stop.
### G7: NVFP4 Slower Than FP8 on SM121
- **SYMPTOM:** switching an inference workload from FP8 to
NVFP4 on Spark makes it slower, not faster.
- **CAUSE:** SM121 lacks `cvt.e2m1x2` unless kernels target
`sm_121a`; NVFP4 runs ~32% slower without it.
- **CHECK:** `references/gotcha-checks.md` G7 — capability
reports `(12, 1)`; does the build target `sm_121a`?
- **FIX:** stay on FP8 unless the build targets `sm_121a`.
### G8: Stale Official Playbooks
- **SYMPTOM:** following an official DGX Spark playbook still
fails, with no local misconfiguration explaining it.
- **CAUSE:** official playbooks have shipped broken before;
the stack moves faster than the docs.
- **CHECK:** `references/gotcha-checks.md` G8 — the playbook
repo's recent issues.
- **FIX:** check `github.com/NVIDIA/dgx-spark-playbooks` issues
before trusting a recipe for an expensive run.
### G9: Container-First, Not Bare Pip
- **SYMPTOM:** a bare-pip environment that worked yesterday
breaks after an unrelated `pip install`, or two "identical"
environments behave differently.
- **CAUSE:** bare pip lets Triton, xformers, and transformers
drift independently; nothing pins them to GB10's SM121
target.
- **CHECK:** `references/gotcha-checks.md`
G9 — container or bare pip?
- **FIX:** prefer an NGC container (see `spark-environment-setup`
for tag guidance) or Unsloth's container. If bare pip is
unavoidable, follow the NVIDIA install order, including
`--no-deps` on Unsloth.
### G10: Dual-Spark Is DDP/FSDP Only
- **SYMPTOM:** a tensor-parallel launch across two Sparks
hangs, runs far slower than single-Spark, or errors out.
- **CAUSE:** ConnectX-7 is fast enough for gradient/parameter
sync (DDP, FSDP) but too thin for TP's fine-grained traffic.
- **CHECK:** `references/gotcha-checks.md` G10 — the
configured parallelism strategy.
- **FIX:** on a two-Spark setup, choose DDP or FSDP, never
tensor parallelism — TP is single-node only here.
## Fast Triage
The cheapest checks to run before anything else:
```bash
python3 -c "import torch; print(torch.version.cuda)" # expect 13.x (G1); NGC builds have no +cu130 tag — that's not a failure
```
```python
import torch; print(torch.cuda.get_device_capability()) # expect (12, 1) (G7)
```
```bash
{ [ -f /.dockerenv -o -f /run/.containerenv ] || grep -qE 'docker|containerd' /proc/1/cgroup; } 2>/dev/null && echo container || echo unknown # G9
```
`assets/preflight.sh` runs G1, G3, G4, G7, G9 and produces one
output line per gotcha in a fixed format: G-number first, then
PASS/FAIL/WARN where automatable, SKIP when unavailable, or
`INFO:` for a raw reading (G3, G4). Full commands:
`references/gotcha-checks.md`. See also
`spark-environment-setup` for the environment assumed working.