references/agent-loop-patterns.md
# Agent Loop Patterns
Loaded during Steps 6–7 of the token-optimization workflow: model
routing and loop-overhead reduction.
## Why loops dominate cost
In any harness with a tool-use loop, **every iteration re-transmits the
full conversation context**. If the system prompt + tool defs + history
is 30k tokens, a 10-iteration loop costs 300k input tokens regardless of
how small each step is. Cutting iterations is therefore multiplicative
in a way that cutting per-call output is not.
The right mental model: each iteration is "one shot at making
progress". Maximize the work-per-iteration, not the iterations-per-task.
## Model routing decision table
Match the model to the task; the largest model on every call is the
single most common cause of overspend.
| Tier | Tasks | Anthropic model class | Why |
|---|---|---|---|
| Smallest | Status checks, ping/health, classification, single-fact lookup, heartbeat-style background pings | Haiku-class | 10–20× cheaper than mid; latency wins |
| Small | File reads, lints, formatting fixes, autocomplete, schema validation, simple search | Haiku-class | Quality is sufficient |
| Mid | Code generation, single-file debug, test writing, explain-this-code, moderate refactor | Sonnet-class | Sweet spot for most coding work |
| Top | Architecture, multi-step planning, deep refactor across many files, very long context, hard reasoning | Opus-class | Reserve for problems that genuinely need it |
### Routing signals
Inputs to the routing decision:
- **Keyword match** in the user request (status/ping → smallest;
architect/design/plan → top).
- **Complexity score** (rough 0–10): single fact = 0–2, multi-file
reasoning = 6–8, system design = 9–10.
- **Context size**: prompts > 100k input tokens may justify the top
tier purely because the cheaper model would degrade.
- **Tool-call count expected**: many small tool calls → small model.
### Fallback chain
When the chosen tier returns 429/503 or response time blows past a
ceiling, downgrade one step: top → mid → small → smallest. Log the
reason. Surface it to the user when the downgrade would change quality
materially.
## Sub-agent delegation as context isolation
Sub-agents exist to **isolate context**, not to simulate org charts.
The pattern:
1. Main loop has a 50k-token context.
2. Main loop dispatches a sub-agent with a narrow prompt ("find all
call sites of X and report").
3. Sub-agent reads files, runs greps — *its own* context fills with
raw output, but that context is discarded when it returns.
4. Sub-agent returns a 500-token synthesis to the main loop.
The main loop never pays for the raw search output. This is the single
biggest reason to use sub-agents on large codebases: not parallelism,
but context-cost isolation.
When NOT to delegate:
- The task needs the full context the parent already has.
- The result is itself huge (sub-agent returns 30k tokens — savings
evaporate).
- The orchestration overhead exceeds the work done.
## Parallel tool calls
When two tool calls have **no data dependency between them**, issue
both in the *same* assistant turn. The host runs them concurrently and
returns both results in one observation, saving an entire iteration of
context re-transmission.
Examples of parallelizable calls:
- `Read(file_a)` + `Read(file_b)` when both are needed for the next step.
- `Bash(git status)` + `Bash(git diff)` + `Bash(git log)`.
- `grep("pattern", path_a)` + `grep("pattern", path_b)`.
Examples of NOT parallelizable:
- Read a file, then edit it (edit depends on the read).
- Run a test, then read its log (log depends on the test having run).
- Any sequence where the second call's input is the first call's output.
In the Anthropic SDK, this is supported natively via multiple
`tool_use` blocks in a single assistant message. Most modern hosts
(Claude Code, Codex, Cursor) expose it through their tool-use protocol.
## Iteration caps
Set a soft ceiling on retry/iteration loops on a single failing operation, with a clean exit — leave room for legitimately multi-step plans:
```
max_iterations: 5
on_limit_reached: summarize_progress_and_stop
```
Without a cap, a stuck agent burns tokens indefinitely. With a cap, the
worst case is bounded and the user gets a useful summary instead of a
silent timeout.
For genuinely long workflows (e.g., "scan 200 files"), batch in fewer,
larger iterations rather than one-per-item:
| Pattern | Iterations | Context overhead |
|---|---|---|
| One file per iteration × 200 | 200 | catastrophic |
| Ten files per iteration × 20 | 20 | 10× cheaper |
| Sub-agent batches 50 files × 4 | 4 main + 4 sub-agent contexts | cheapest |
## Background and scheduled jobs
If a task takes minutes (a long build, a deploy, a remote training
run), don't keep the model in a polling loop — every poll re-sends the
full context.
- Use the host's background-process primitive (Claude Code's
`run_in_background`, Codex equivalents). The model checks in only
when notified.
- Schedule a wake-up rather than sleeping with poll loops.
- For genuinely fire-and-forget tasks, exit the loop entirely and let
the user re-engage when the task completes.
## Extended-thinking budgets
Models with extended thinking (e.g., Anthropic `thinking` block) charge
for thinking tokens. Set an explicit budget:
| Task class | Suggested `budget_tokens` |
|---|---|
| Routine code edit, lookup, formatting | 1,024 or off |
| Standard debug, single-file refactor | 2,048 – 4,096 |
| Complex architecture, multi-step planning | 8,192 – 16,384 |
| Genuinely hard reasoning (proof, deep analysis) | 32,000+ |
Defaults are usually generous; explicit budgets prevent runaway
thinking on simple tasks.
## Streaming and stop sequences
- **Streaming** does not reduce tokens but improves perceived latency
and lets you cancel early when the output goes off-track. Cancel
early to save output tokens.
- **Stop sequences**: when the useful output ends with a known
delimiter (e.g., a closing tag, a sentinel string), set it as a stop
sequence. Generation halts immediately.
## Claude Code-specific operational levers
When the host is Claude Code specifically, several settings move the
needle without any prompt-engineering work:
### `~/.claude/settings.json`
```json
{
"model": "sonnet",
"env": {
"MAX_THINKING_TOKENS": "10000",
"CLAUDE_CODE_SUBAGENT_MODEL": "haiku"
}
}
```
| Setting | Default | Recommended | Effect |
|---|---|---|---|
| `model` | varies | `sonnet` | Sonnet handles most coding work; switch to Opus only for complex reasoning. |
| `MAX_THINKING_TOKENS` | high (~32k) | `10000` | Caps extended-thinking budget; hidden output cost drops sharply. Set to `0` for trivial sessions. |
| `CLAUDE_CODE_SUBAGENT_MODEL` | inherits main | `haiku` | Sub-agent (Task) runs on Haiku — exploration, file reads, summarization at a fraction of the cost. |
### Slash commands during a session
| Command | When to use |
|---|---|
| `/clear` | Between unrelated tasks — stale context is paid every subsequent turn |
| `/compact` | At logical task boundaries (after exploration, after a milestone, before switching focus) |
| `/cost` | Check current session spend |
| `/model sonnet \| opus \| haiku` | Switch mid-session to match the next task's complexity |
**When to `/compact`**: after exploration before implementation, after a
milestone, after debugging before new work, before a major context
shift. **When NOT to**: mid-implementation of related changes, during
active debugging, during a multi-file refactor.
### Extended-thinking toggles
- **Alt+T** (Linux/Windows) or **Option+T** (macOS): toggle thinking on/off
- **Ctrl+O**: show thinking output (verbose mode)
Turning thinking off entirely for routine work is often the single
biggest win on simple sessions.
### MCP server hygiene
Each enabled MCP server injects its tool definitions on every turn.
Common rule of thumb: keep fewer than ~10 enabled per project.
- Run `/mcp` to see active servers and their context footprint.
- Disable unused servers (e.g., a default `memory` server you don't
actually consume) via `/mcp`. Settings-file edits don't unload an
already-running server in the live session.
- Prefer CLI tools (`gh`, `aws`, `kubectl`) over MCP equivalents when
the CLI exists — the CLI doesn't pay per-turn schema overhead.
### Agent Teams (experimental)
Agent Teams spawns multiple independent context windows in parallel —
each teammate pays its own context cost. Use only when parallelism
clearly adds value (multi-module work, parallel reviews). For
sequential work, sub-agents (Task tool) are more token-efficient
because they isolate context without duplicating it.
### Auto-compaction overrides
Some Claude Code builds expose `CLAUDE_AUTOCOMPACT_PCT_OVERRIDE`.
Community reports indicate it can only *lower* the compaction
threshold, so values below default tend to make compaction fire
*earlier* than expected. If behavior surprises you, remove the
override and rely on manual `/compact` at planned breakpoints.
## Batch API
For non-interactive workloads (evaluation runs, bulk data processing),
the **batch API** typically provides a ~50% discount with a 24-hour
SLA. If real-time isn't required, batch the work — the discount is
larger than any prompt-engineering optimization can match.
references/context-management.md
# Context Management
Loaded during Steps 3–5 of the token-optimization workflow: slimming
always-loaded files, compressing conversation history, and masking
verbose tool observations.
## Slimming always-loaded instruction files
Files like `CLAUDE.md`, `AGENTS.md`, `GEMINI.md`, `.cursorrules`, and
project memory indexes are **multiplied by every turn**. A 5,000-token
instruction file across a 100-message session is 500,000 input tokens of
overhead — likely larger than the actual work.
### Audit checklist
Walk through the file with these questions:
1. **Is this a default behavior?** "Be helpful", "write clean code",
"follow best practices" — strike them; they're already true.
2. **Is this redundant with another bullet?** Merge synonyms into one.
3. **Is this an example?** If yes, move it to a `references/<topic>.md`
loaded only when the relevant skill activates.
4. **Is this referenced anywhere?** If a section hasn't been used in two
weeks, archive it.
5. **Is this a paragraph that could be a bullet?** Convert.
6. **Is this an enumeration of frameworks/tools?** Replace with a single
adaptive rule ("adapts to the project's stack").
### Target budgets
| File | Target | Rationale |
|---|---|---|
| Top instruction file (CLAUDE.md / AGENTS.md / etc.) | < 1,500 tokens | Loaded every turn |
| Sub-agent / persona file | < 800 tokens | Loaded every turn when active |
| Memory index | < 3,000 tokens | One-line pointers per entry, no inline content |
| Per-skill SKILL.md body | < 5,000 tokens | Move detail to references/ |
### Anti-patterns to remove
- "You are an expert in [long list]." → adapt to the project's stack.
- Three rules saying the same thing in different words. → one rule.
- Worked examples inline. → move to `references/`.
- Catch-all "if in doubt, ask the user" reminders. → already a default.
## Compressing conversation history
When the conversation-history bucket dominates measurement, apply tiered
compression:
| Age | Treatment |
|---|---|
| Last 3–5 turns | Verbatim — the model needs the immediate context |
| Turns 6–10 | Decisions and conclusions only; drop intermediate reasoning |
| Older | One-line summary per turn |
| Failed tool calls past the fix | Drop the failure, keep the working result |
| File reads superseded by edits | Drop the old read, keep the latest state |
| Long Bash output | Keep first/last 5 lines + exit code |
### Structured-summary template
When generating a compressed snapshot, use this shape so nothing
load-bearing is silently dropped:
```markdown
# Session Snapshot — <timestamp>
> Original ~N tokens → compressed ~M tokens (saved X%)
## Primary objective
<the goal that drives all turns>
## Key decisions
- <decision 1, with the constraint that motivated it>
- <decision 2>
## Active artifacts
- <file path>: <one-line summary of current state>
- <file path>: <one-line summary>
## Pending work
- <todo 1>
- <todo 2>
## Recent context (verbatim, last 3–5 turns)
<last few turns, full text>
```
### Hard rules during compression
- **Never prune the system prompt or safety constraints.** Loss-of-context
jailbreaks are a real failure mode.
- **Pin the primary objective at the top of the new prompt** so the model
can't drift after compression.
- **Preserve the artifact trail.** If a file was read, then edited, then
re-read, the latest state is canonical, but record that an edit
happened — otherwise the model may re-read again to "verify".
- **Don't compress within the active turn.** Compress between turns, not
mid-task.
## Masking verbose tool observations
Tool output is the largest unbounded token source. Once a result has
been *consumed* (the model used it to make a decision), replace the raw
output with a short reference the model can re-fetch on demand:
| Tool | Raw output | Masked reference |
|---|---|---|
| File read | full file content | `[read src/foo.ts: 312 lines, exports: a,b,c — re-read with Read]` |
| grep / search | every match line | `[grep "X": 7 hits in 4 files — re-run if needed]` |
| Bash command | full stdout | `[bash <cmd>: exit=0, head/tail kept — re-run if needed]` |
| Web fetch | full HTML | `[fetched <url>: title, key headings — re-fetch if needed]` |
| Database query | full result set | `[query: 1428 rows, schema: …, sample: …]` |
The model still has the *address* (path, URL, command) to re-fetch — it
just doesn't carry the bytes through every subsequent turn.
## Image and screenshot tokens
Vision input is unusually expensive per "token" because images are
encoded into hundreds-to-thousands of tokens depending on resolution.
| Resolution | Approx. tokens (Claude vision) | Use case |
|---|---|---|
| 3840 × 2160 (4K) | ~1,600 | Almost never necessary |
| 1920 × 1080 | ~900 | Detail-rich diagrams |
| 1280 × 720 | ~500 | UI element detection, OCR — sweet spot |
| 1024 × 768 | ~400 | General OCR |
| 800 × 600 | ~300 | Minimum viable for OCR |
| 640 × 480 | ~200 | Layout-only checks |
Resize before sending. For browser-automation workflows (Playwright,
Puppeteer, MCP browser tools), set the viewport to 1280×720 by default
and only escalate when the task fails at that resolution.
## File-system as cold context
For long-horizon work, the file system is the cheapest possible "memory":
- Plans and decisions: write to `plan.md` or a checkpoint file. The
model re-reads only the relevant section on demand.
- Sub-agent communication: shared files instead of message-passing
through context.
- Scratch pads for tool output: dump big results to a temp file and pass
the *path* between turns.
This is the **filesystem-as-memory** pattern — strictly cheaper than
keeping the same content in the conversation history, because each turn
only pays for what it reads.
references/measurement-and-budgets.md
# Measurement and Budgets
Loaded during Step 1 of the token-optimization workflow.
## Heuristic estimation
When no tokenizer is available, these rules of thumb are accurate to
roughly ±15%:
| Content type | Rule | Notes |
|---|---|---|
| English prose | `tokens ≈ words × 1.3` | Most reliable for natural language |
| Code (any language) | `tokens ≈ chars / 4` | Includes whitespace and punctuation |
| Mixed code + prose | `tokens ≈ chars / 4` | Use the code rule for safety margin |
| CJK / multi-byte text | `tokens ≈ chars / 2` | Tokenizers treat each character as ~1–2 tokens |
| JSON / structured | `tokens ≈ chars / 3.5` | Brace and quote density inflates count |
For the cost figure, multiply by the per-million-token price of the
model in question. Input and output prices differ — output is typically
3–5× the input price, which is why output trimming (Step 8) often
dominates the savings on chatty agents.
## Exact counts
When precision matters (e.g., guarding against context-window overflow
on a long prompt), call the host provider's count-tokens facility before
the real generation:
- **Anthropic SDK**: `client.messages.count_tokens(...)` returns the
exact input-token count without consuming generation budget.
- **OpenAI / compatible**: `tiktoken` library locally; `cl100k_base` /
`o200k_base` encodings depending on model.
- **Google Gemini SDK**: `model.count_tokens(...)` on the request.
These calls are typically free or trivially cheap and are the right tool
when you need to *guarantee* fitting within a window.
## Output-size estimation by complexity
When you need to predict the response window before answering, classify
the prompt and apply a multiplier to the input token count:
| Complexity | Input → output multiplier | Example prompts |
|---|---|---|
| Simple | 3× – 8× | "What is X?", yes/no, single-fact lookup |
| Medium | 8× – 20× | "How does X work?", short walk-through |
| Medium-high | 10× – 25× | Code generation with provided context |
| Complex | 15× – 40× | Multi-part analysis, architecture, comparison |
| Creative | 10× – 30× | Stories, essays, narrative writing |
The product `input_tokens × multiplier` is the *full window*; clamp it
to the model's configured `max_tokens` ceiling. Pick a depth tier
(below) to land inside the window.
## Depth tiers
When the user asks for control over response length, present four tiers
based on the predicted window:
| Tier | Position in window | Target shape |
|---|---|---|
| Essential (25%) | `min + (max-min) × 0.25` | 2–4 sentences; direct answer only |
| Moderate (50%) | `min + (max-min) × 0.50` | 1–3 paragraphs; answer + 1 example |
| Detailed (75%) | `min + (max-min) × 0.75` | Structured response; pros/cons |
| Exhaustive (100%) | `max` | No restriction |
If the user has already chosen a tier earlier in the session, hold it
silently for follow-ups; don't re-ask each turn.
## Bucketed measurement template
When diagnosing where tokens are going, fill in this table for the
current request:
```
Bucket | Est. tokens | % of total | Optimization layer
---------------------|-------------|------------|--------------------
System / role | | | Step 3 (slim)
Tool definitions | | | Step 3 (drop unused tools)
Conversation history | | | Step 4 (compress)
Tool results | | | Step 5 (mask)
Current turn | | | Step 8 (output trim)
Total | | |
```
The dominant bucket dictates which step in the main workflow to focus
on. If two buckets are within 10% of each other, address them in
parallel.
## Pricing-aware reporting
When reporting savings to a user, frame the delta in three ways:
1. **Per-call**: "−40% input, −15% output → ~$0.0028 saved per call"
2. **Per-day** (if usage is known): "≈ $4.20/day at current pace"
3. **Per-month** (extrapolate × 30 only if usage is steady): with the
caveat that real usage varies and the figure is an estimate
Avoid quoting absolute monthly numbers without noting the assumption —
pricing changes, usage patterns shift, and small estimation errors
amplify across millions of tokens.
references/output-and-verification.md
# Output and Verification
Loaded during Steps 8–9 of the token-optimization workflow: trimming generated output and verifying that the optimization actually moved the dominant bucket. The SKILL.md body has the headline bullets; this file expands the templates, cookbooks, and troubleshooting matrices.
---
## Step 8 — Trim the output
Output tokens are typically priced 3–5× input tokens (provider-dependent). Trimming output usually pays back faster than trimming input. Tactics in priority order:
### 8.1 Depth tiers (offer before answering)
Default behavior most agents accidentally adopt: maximum verbosity for every question. Counter with an explicit depth picker for any answer ≥ 200 words. Phrase the picker once, then default to the smallest tier the user implied.
**Template:**
```
Depth options:
essential — 1–3 sentences answering the literal question.
moderate — single paragraph (4–8 sentences) with reasoning.
detailed — multi-paragraph (~300–500 words) covering edge cases.
exhaustive — full coverage including alternatives, gotchas, references.
Default: essential. Reply "moderate" / "detailed" / "exhaustive" to expand.
```
**Heuristic for the default tier** (pick the smallest that fits):
| User signal | Default |
|---|---|
| "what is X?" / "how do I X?" / one-line query | essential |
| "explain X" / "walk me through X" | moderate |
| "compare X and Y" / "design X" | detailed |
| "I'm building X and need to choose between …" | exhaustive |
When in doubt, drop a tier. User can always ask for more.
### 8.2 Cut filler, keep orientation
Strip pure ceremony from every response, but keep one-line orientations that help the reader (e.g., "Updated X to Y because Z"). Common offenders worth stripping:
| Bridge phrase | What to replace it with |
|---|---|
| "Sure! Here is …" | (delete; start at the answer) |
| "I've updated the file." (status only, no why) | (delete; if the *why* matters, use a one-line "Updated X to Y because Z" instead) |
| "Let me know if you need anything else!" | (delete) |
| "I hope this helps!" | (delete) |
| "Great question." / "Excellent point." | (delete) |
| "First, let me explain …" | (start with the explanation) |
| "Before answering, I should note …" | (lead with the answer; caveats follow inline) |
Each bridge phrase = 5–15 wasted output tokens, multiplied by every response. Over a long agent loop this compounds.
### 8.3 Structured output for machine consumption
When the consumer is another tool / parser, JSON or a tool-use schema beats prose every time:
- 30–50% shorter than equivalent prose at the same information density.
- No filler tokens (no "the result is …" wrappers).
- Provider tool-use APIs (Anthropic `tool_use`, OpenAI function-calling) often skip the prose portion entirely on the wire.
**When to use:**
- Any agent-to-agent handoff.
- Any parseable output (lists of files, structured diffs, configuration).
- Any output that will be re-fed to a model in a subsequent turn.
**When NOT to use:**
- Human-facing explanations (prose is more readable).
- Anything where reasoning visibility matters (debugging, code review).
### 8.4 Stop sequences
When a known terminator delimits the useful output, set it as a stop sequence on the API call. The model stops generating immediately on the terminator and you don't pay for the trailing reasoning.
**Common stop sequences by use case:**
| Use case | Stop sequence |
|---|---|
| Code completion (single function) | `\n}\n` or `\nend\n` |
| List-building (until end marker) | `\n---\n` or `\nDONE\n` |
| Q&A with explicit terminator | `\n\nQ:` or `\n[END]` |
| Tool-call extraction | `\n</tool>` |
| Single-line classification | `\n` (literal newline) |
Stop sequences are MOST effective with extended-thinking models — they prevent the model from elaborating after producing the answer.
### 8.5 Cap thinking budgets
For models with extended thinking (Claude Opus / Sonnet 4.x with thinking enabled, GPT-o1, etc.), set an explicit thinking budget. Reserve large budgets for problems that genuinely benefit; cap routine tasks tightly.
**Calibration table:**
| Task type | Thinking budget |
|---|---|
| Classification, status check, single-file edit | 1k tokens |
| Routine debugging, test generation | 2k–4k tokens |
| Multi-file refactor, architecture sketch | 8k–16k tokens |
| Hard math, deep reasoning, novel design | 32k–64k tokens (the model's max) |
Misuse to avoid: setting "max thinking" by default. The model spends the full budget on every task, including trivial ones; you pay for thinking tokens that produce no value.
---
## Step 9 — Verify the win
Optimization without measurement is hope. Re-run the Step 1 measurement after applying tactics; report a structured delta.
### 9.1 Re-measurement template
```
Optimization target: <bucket name from Step 1, e.g., "system prompt">
Before:
input tokens: N1 (or estimated: ~N1)
output tokens: M1 (or estimated: ~M1)
est cost: $C1 (per call)
After:
input tokens: N2
output tokens: M2
est cost: $C2
Delta:
input: -X% (saved N1 - N2 tokens)
output: -Y% (saved M1 - M2 tokens)
cost: -$Z (per call; -$Z * estimated daily calls = -$ZZ/day)
```
CSV-friendly variant for tracking over multiple optimization rounds:
```
round,bucket,before_in,before_out,after_in,after_out,delta_pct,delta_$_per_call
1,sys_prompt,12000,800,4500,800,-58,-0.18
2,tool_defs,4500,800,2800,800,-29,-0.05
3,thinking,2800,800,2800,400,-30_output,-0.06
```
### 9.2 Negligible-delta troubleshooting
If the delta is < 5% on the targeted bucket, the dominant bucket was something else. Re-measure all buckets fresh; the actual cost driver is hiding somewhere you didn't look.
**Common causes when "obvious" optimizations don't move the needle:**
| Symptom | Likely actual culprit |
|---|---|
| Trimmed system prompt by 50%, total tokens dropped 5% | Conversation history or tool definitions are the real bulk |
| Added prompt caching, no cost change visible | Cache breakpoint is invalidated every call (timestamp / random ID at front?) |
| Switched to cheaper model, no cost saving | Loop count went up because cheaper model needed more turns |
| Reduced output tokens, no cost saving | Output was already minor; input is the dominant bucket |
| Compaction added, marginal gain | Conversation already short relative to system prompt |
**Recovery procedure:** redo Step 1 with fresh measurements (don't trust the prior bucket sizes — they may have shifted). Pick a different dominant bucket. Apply Steps 2–8 tactics targeted at that bucket.
### 9.3 Tokens-per-task vs tokens-per-call
Always verify against the **task** target, not the **call** target. A 30% per-call saving that quintuples the loop count is a 350% increase in tokens-per-task.
**Verification method:**
1. Pick a representative end-to-end task (not a single API call).
2. Run it before optimization; log total tokens across all calls.
3. Run the same task after optimization; log total again.
4. Compare totals, not per-call numbers.
If the per-task total didn't move (or got worse), the optimization was a per-call win that hurt the loop. Roll back.
### 9.4 When to call the optimization "done"
Stop when any of these conditions hits:
- The dominant bucket is now within 20% of the next-largest bucket (no single dominant target).
- Each subsequent tactic yields < 5% improvement.
- The optimization itself starts costing more in operator time than the savings buy back per month.
- A fresh measurement shows the workload pattern has shifted (e.g., user behavior changed) — re-baseline before continuing.
references/prompt-caching.md
# Prompt Caching
Loaded during Step 2 of the token-optimization workflow. Provider-specific
mechanics; the *principle* (stable prefix → cheap reuse) is universal.
## The mental model
Cached prompt prefixes work like a content-addressed key/value store: the
provider hashes the cached portion of your prompt, and on the next request
that begins with the *byte-identical* prefix it serves the KV state from
cache instead of re-encoding it. You pay a small write surcharge once and
a heavy read discount thereafter, for a TTL-bounded window.
Any change inside the cached region — even a whitespace tweak, a swapped
tool ordering, or a date string — invalidates that prefix and all longer
prefixes built on it.
## What to cache
Best candidates are **stable**, **large**, and **reused** content:
- The system prompt and any always-loaded instructions
- Tool definitions (schemas tend to be hundreds of tokens each)
- Long reference documents the agent re-reads across turns
- Few-shot examples that don't change per request
- Memory / convention files injected on every turn
Bad candidates:
- The current user message (changes every turn — put it after the cache)
- The latest tool result (changes every turn)
- Anything containing a timestamp, random ID, or per-request variable
## Where to place the cache breakpoint
Order the prompt in **decreasing stability**, then mark the breakpoint at
the *end* of the largest stable block:
```
[ system prompt ]
[ tool definitions ]
[ long reference docs ] ← cache breakpoint here
[ conversation history ]
[ current user message ]
```
In the Anthropic SDK, set `cache_control: {type: "ephemeral"}` on the
last content block of the stable section. Multiple breakpoints are
allowed (newer blocks fall through to older cached prefixes), but each
breakpoint adds bookkeeping — start with one.
## TTL and minimum size
Provider-specific; verify against current docs before committing:
- Anthropic ephemeral cache: 5-minute TTL by default; 1-hour cache also
available with different pricing. Minimum block size applies (rough
order: ~1024 tokens for Sonnet/Opus, ~2048 for Haiku — confirm before
relying).
- The cache is per-organization, per-model, per-region. Switching models
means re-warming.
- A read on a near-expired cache entry usually refreshes it for the same
TTL window; a long idle gap drops the entry.
## Cache hit verification
After enabling caching, look for a hit-rate signal in the API response:
- Anthropic: `usage.cache_read_input_tokens` and
`usage.cache_creation_input_tokens` on the response. The first should
grow on subsequent calls; the second should be near zero except on
cache writes.
- A hit rate below ~50% on a workload that should reuse prefixes means
something is invalidating the cache. Common culprits below.
## Common cache-invalidation traps
1. **Timestamps or request IDs at the top.** Move them to the user
message at the end.
2. **Tool definition reordering.** SDKs sometimes serialize tool lists
in different orders — pin the order explicitly.
3. **Whitespace drift.** Auto-formatters that touch the system prompt
between calls invalidate the cache. Pin the exact bytes.
4. **Conditional content.** A flag that "only adds X when Y" inside the
cached region produces two different prefixes; lift the conditional
*outside* the cache or move it to the end.
5. **Memory file edits.** If the host injects a `MEMORY.md` that you
edited mid-session, the cache breaks. Either freeze memory for the
duration of a session or keep memory after the breakpoint.
6. **Multi-region failover.** A failover to a different region warms a
cold cache.
## When NOT to cache
- One-shot prompts with no plausible reuse.
- Prefixes shorter than the provider's minimum cache block — the write
surcharge isn't recouped.
- Highly variable system prompts that change per user / per task.
- When the cache write cost is paid more often than the read benefit —
audit hit rate before scaling caching out widely.
## Other provider equivalents
- **OpenAI**: implicit prompt caching on long stable prefixes; no manual
breakpoint, but ordering still matters (stable prefix first).
- **Google Gemini**: explicit `cachedContent` / context caching with TTL.
- **Cursor / Claude Code / Codex**: typically pass the underlying SDK's
caching through; check the host's documentation for any wrapper that
controls breakpoint placement.
The host-agnostic rule holds across all of these: **stable first, mutable
last, byte-identical between calls.**
SKILL.md
---
name: token-optimization
description: >
Use when you need to reduce token usage, lower API cost, fit work within a
context window, or speed up an expensive/slow agent loop. Triggers on
phrases like "optimize tokens", "reduce token usage", "save on cost",
"fit in context", "context too long", "running out of tokens", "prompt
caching", "compact the context", "agent is too expensive", "shrink the
prompt", "cheaper model", "tokens-per-task". Provides a layered tactic
catalog — measurement (heuristic counting and budget allocation),
prompt-side levers (caching, system-prompt diet, tool-definition pruning),
context management (compression, observation masking, file-system
offload), agent-loop patterns (parallel tool calls, batch operations,
model routing by complexity), and output-side controls (depth tiers, stop
sequences, structured output). Project-agnostic and provider-agnostic;
works across Claude Code, Codex CLI, Cursor, Gemini CLI, and Copilot.
extensions:
claude: {}
copilot: {}
cursor: {}
gemini: {}
codex: {}
version: "1.8.3"
forge:
status: reviewed
forged: 2026-05-07
reviewed: 2026-05-11
---
# Token Optimization
## Overview
Reducing token usage is rarely a single trick — it's a stack of small wins
applied at the right layer. This skill is a triage map: it diagnoses *where*
the tokens are going (system prompt, conversation history, tool results,
agent loops, output verbosity), then points to the matching tactic. The
correct optimization target is **tokens-per-task**, not tokens-per-request:
a one-shot 12k-token prompt that solves the problem beats a 2k-token loop
that takes 30 turns. Apply tactics in order: measure first, then cache,
then compress, then route, then trim output.
## When to activate
- ✅ User says they're hitting context limits, billing surprises, or slow loops
- ✅ User asks how to use prompt caching, compaction, or extended thinking budgets
- ✅ User wants to choose a cheaper model or split work across models
- ✅ Agent itself notices a long-running session crossing roughly 70% context-window utilization (rule-of-thumb trigger; use exact counts when available, heuristic estimate otherwise — trigger earlier on long-running sessions where headroom matters more, later on short tasks)
- ✅ Reviewing a system prompt, CLAUDE.md, or memory file that "feels heavy"
**Do NOT activate when:**
- The topic is a specific tokenizer library (use a `tokenization-*` skill)
- "Token" refers to auth/JWT/payment tokens, or UI design tokens
- The user only wants pricing math (point at provider pricing pages)
## Workflow
Apply the tactics in the order below. Each layer is cheap to try and
independent of the next, so stop as soon as the cost/context goal is met.
### Step 1 — Measure where the tokens go
Before optimizing, locate the cost. For a single request, mentally split
the prompt into five buckets and estimate each:
| Bucket | What it contains | Typical % of total |
|---|---|---|
| System / role | static system prompt, CLAUDE.md, AGENTS.md, memory files | 5 – 40% |
| Tool definitions | tool schemas injected per turn | 5 – 30% |
| Conversation history | prior user/assistant turns | 10 – 50% |
| Tool results | file reads, search output, command stdout | 20 – 70% |
| Current user message + output | this turn's request and reply | 5 – 20% |
Heuristic estimation (no real tokenizer needed):
- English prose: `tokens ≈ words × 1.3`
- Code or mixed: `tokens ≈ chars / 4`
- CJK / multi-byte: `tokens ≈ chars / 2`
Accuracy is roughly ±15%. For exact counts in a Claude / Anthropic SDK
context, use the SDK's `count_tokens` endpoint before the real call. For
detailed budget math (output multipliers, depth tiers), see
[`references/measurement-and-budgets.md`](references/measurement-and-budgets.md).
Output: state which 1–2 buckets dominate. The bucket dictates which step
below to focus on.
### Step 2 — Make the static prefix cacheable
The single biggest win for repeated calls is **prompt caching**: a stable
prefix (system prompt + tool defs + long reference docs) is hashed once
and reused at a heavy discount on subsequent requests within the cache
TTL. To exploit it:
- Place everything stable at the **front** of the prompt; place changing
content (the user's current question, the latest tool result) at the
**end**. Cache lookups match by exact prefix.
- Mark the cache breakpoint at the end of the largest stable block. In
the Anthropic SDK, set `cache_control: {type: "ephemeral"}` on the last
content block of that prefix.
- Keep the prefix byte-identical between calls. Reordering tool defs,
inserting timestamps, or shuffling memory files invalidates the cache.
- Group reads in a stable order so file-content blocks reused across
turns hit the same cache entry.
**When the prefix legitimately needs to change**, never edit an earlier
message in place. Append a new message carrying the updated state
instead — every prior call's cached prefix stays valid.
| Change | Wrong (cache-breaking) | Right (cache-preserving) |
|---|---|---|
| Sandbox / permission mode | edit the earlier permission message | append a new `role=developer` message with the new mode |
| Working directory | edit the earlier environment message | append a new `role=user` message announcing the cwd change |
| Tool list addition | reorder or rewrite earlier tool defs | append at the end; new tools start caching from the next call |
| MCP `tools/list_changed` | hot-reload mid-conversation | defer refresh to a conversation boundary, or accept the miss |
Editing in place rewrites bytes the cache already hashed — every subsequent
call misses. Appending costs only the new message's tokens. (Role names like
`role=developer` / `role=user` above are Anthropic-flavored — use the
equivalent role bucket on other hosts.)
For provider-specific cache mechanics (TTL, minimum block sizes, billing
behavior), see [`references/prompt-caching.md`](references/prompt-caching.md).
### Step 3 — Slim the always-loaded context
Files like `CLAUDE.md`, `AGENTS.md`, `GEMINI.md`, `.cursorrules`, and
memory indexes are injected on **every** message. A 5,000-token memory
file across 100 messages is 500,000 input tokens of pure overhead.
Targets and trims:
| File | Target budget | Common bloat to remove |
|---|---|---|
| Top-level instruction file (CLAUDE.md / AGENTS.md) | < 1,500 tokens | restating defaults, long framework lists, paragraph examples |
| Memory index | < 3,000 tokens | inline content; keep one-line pointers per entry |
| Per-skill instructions | < 5,000 tokens | duplicated explanations across skills |
Slimming rules:
1. **Cut defaults.** "You are a helpful assistant", "write clean code" —
the model already does these.
2. **Merge overlapping bullets.** "Be concise" + "don't be verbose" +
"keep responses short" → one bullet.
3. **Move examples to references.** Examples bloat the always-loaded
payload; put them in on-demand `references/<topic>.md` files.
4. **Bullets, not paragraphs.** Shorter and parsed more reliably.
5. **Drop unused agents/skills.** If a memory entry hasn't been
referenced in two weeks, it's a candidate for removal.
6. **Audit MCP-style tool servers.** Each enabled server injects its tool
defs into every turn — ten servers with 5–10 tool defs each is easily
5,000+ tokens of per-turn overhead. Audit the enabled list against actual
use — disable servers that haven't been called in the last few sessions
before reaching for harder optimizations. A working ceiling of around 10
enabled per project keeps overhead in check, but capability trumps the
count: keep what the task needs. Prefer a CLI binary when an equivalent
exists (e.g., `gh` over a GitHub MCP server).
### Step 4 — Compress conversation history
Long sessions accumulate stale turns. Apply a tiered compression policy
when the history bucket dominates Step 1's accounting:
- Last 3–5 turns: keep verbatim.
- Turns 6–10: keep decisions and conclusions, drop intermediate reasoning.
- Older than that: one-line summary per turn — but keep verbatim any turn
whose specific reasoning is still load-bearing for the current task.
- Failed tool calls past the fix: drop the failure noise; keep the fix.
- Large file reads superseded by edits: keep only the final state.
Pin the **primary objective** at the top of any compressed prompt so the
model never loses the goal during compression. Never prune system
instructions or safety headers. For the structured-summary template and
artifact-trail rules, see
[`references/context-management.md`](references/context-management.md).
**Anti-anxiety calibration.** Frontier models with context-window awareness
sometimes *underestimate* remaining context and start rushing — premature
summaries, shortcut decisions, "let me finalize" mid-task. Counter it with:
- Aim to stay well below the window cap on long sessions (e.g., budget ~200k
of a 1M window for active context). The headroom is a soft buffer for the
model's confidence, not a hard ceiling — extend it when the task genuinely
needs more.
- Include an explicit budget statement near the start of the compressed
snapshot (not in the cacheable static prefix — that would break Step 2's
discipline): "Context budget: ~180k tokens remaining. Continue working at
full depth."
- If you spot mid-task wrap-up signals (sudden summary, "let me finalize")
while real work clearly remains, re-state the budget rather than letting
the model self-direct. If the wrap-up is correct — task done or context
genuinely tight — let it finalize.
This is model-side behavior, not host-side — applies on any host running
a context-aware model.
### Step 5 — Mask verbose tool observations
Tool output is the largest unbounded source of tokens. Replace verbose
results with **observation references** the model can re-fetch on demand:
- Successful file read → `[read src/foo.ts: 312 lines, exports: a,b,c]`
- Successful search → `[grep "X": 7 hits in 4 files: …]`
- Long command stdout → keep first/last 5 lines + exit code
- Screenshots / images → resize to the lowest resolution that still works
for the task. 1280×720 is usually the sweet spot for UI screenshots;
bump up to 1920×1080 or higher for dense text / OCR work where
legibility is load-bearing — full 4K is ~5× more tokens than needed
for typical UI work.
This is *masking*, not *deleting*: the underlying file/command is still
addressable, the model just doesn't carry the bytes through every turn.
### Step 6 — Route to the right model
Match model to task complexity. Reaching for the largest model on every
call is the most common single source of overspend.
| Task | Suggested tier |
|---|---|
| Status checks, pings, classification, simple lookups | smallest/cheapest tier (e.g., Haiku-class) |
| File reads, autocomplete, lints, formatting, single-file edits | small tier |
| Multi-file edits, debugging, test generation | mid tier (e.g., Sonnet-class) |
| Architecture, multi-step planning, deep refactors, very long context | top tier (e.g., Opus-class) |
Keep a fallback chain (top → mid → small) for rate-limit or latency
backoff. When sub-agents are available, **delegate** narrow searches and
file scans to a cheaper-model sub-agent and return only the synthesis to
the main loop — the main loop never pays for the raw search output.
### Step 7 — Cut agent-loop overhead
Each loop iteration re-transmits the full conversation context. Reducing
iterations is multiplicative.
- **Batch tool calls.** Issue independent tool calls in **parallel** in
one assistant turn, not sequentially across turns.
- **Combine small tasks.** Process N files in one iteration instead of
spawning N iterations.
- **Set iteration caps.** Cap retries on a single failing operation
(e.g., ~5) with a clean "summarize and stop" exit, while leaving room
for legitimately multi-step plans to run their length.
- **Avoid speculative reads.** Read what the task needs, not "everything
related". Use `glob`/`grep` to narrow before opening files.
- **Background long jobs.** Don't keep the model in a polling loop on a
multi-minute task — schedule a wake-up or use the host's background
primitive.
For platform-specific patterns (parallel tool calls in the Anthropic SDK,
sub-agent delegation in Claude Code, Task tools in Codex), see
[`references/agent-loop-patterns.md`](references/agent-loop-patterns.md).
### Step 8 — Trim the output
Output tokens are typically priced higher than input. Tactics:
- **Depth tiers.** Offer "essential / moderate / detailed / exhaustive"
before answering long-form questions; default to the smallest tier the
user clearly implied. When intent is ambiguous, err one tier higher
rather than under-answering.
- **Cut filler, keep orientation.** Skip pure throat-clearing ("Sure, here
is …", "Great question!"). A one-line orientation ("Updated X to Y
because Z") is fine and often helpful — the goal is to cut filler, not
strip context.
- **Structured output for machine consumption.** JSON / tool-use schemas
are usually shorter than equivalent prose and don't need filler.
- **Stop sequences.** When a known terminator delimits the useful output,
set it; the model stops generating once reached.
- **Cap thinking.** For models with extended thinking, set an explicit
thinking budget (e.g., 1k–4k tokens for routine tasks; reserve larger
budgets for genuinely hard problems).
For depth-tier templates, bridge-phrase strip list, stop-sequence
cookbook, and thinking-budget calibration tables, see
[`references/output-and-verification.md`](references/output-and-verification.md).
### Step 9 — Verify the win
Before declaring success, re-measure with the same buckets from Step 1
and report a delta:
```
Before: input=N1, output=M1, est cost C1
After: input=N2, output=M2, est cost C2
Delta: -X% input, -Y% output, -$Z per call
```
If the delta is negligible, the dominant bucket was different from what
you optimized — go back to Step 1 with fresh measurements.
For the full re-measurement template (incl. CSV variant for tracking
multiple optimization rounds), the negligible-delta troubleshooting
matrix, tokens-per-task vs tokens-per-call verification method, and
"when to call optimization done" exit conditions, see
[`references/output-and-verification.md`](references/output-and-verification.md).
**Expected ranges.** Use these as calibration points, not guarantees:
- Compaction: 50–70% reduction at <5% quality loss.
- Masking: 60–80% reduction in masked observations.
- Prefix cache hit rate: ≥ 70% on stable workloads.
If your delta is at the upper end of these ranges, you're in the
diminishing-returns zone for that tactic on typical workloads — further wins
come from layering, not from squeezing one lever harder.
## Cross-agent invocation
The tactics above are agent-agnostic; the *commands* to apply them differ per
host runtime. Use this table to translate a tactic into the host you're in.
| Tactic | Claude Code | Codex CLI | Cursor | Gemini CLI | Copilot CLI |
|---|---|---|---|---|---|
| Reset context (Step 4) | `/clear` | `/clear` | New chat | `/clear` | New session |
| Manual compaction (Step 4) | `/compact` | manual summary only | n/a | n/a | n/a |
| Auto-compaction threshold (Step 4) | `CLAUDE_AUTOCOMPACT_PCT_OVERRIDE=<pct>` (env) | n/a | n/a | n/a | n/a |
| Switch model (Step 6) | `/model sonnet\|opus\|haiku` | `--model <id>` flag | model picker | `/model <id>` | `--model <id>` |
| Subagent / delegated model (Step 6) | `CLAUDE_CODE_SUBAGENT_MODEL=<id>` (env) | per-Task model arg | n/a | sub-agent config | n/a |
| Cap thinking budget (Step 8) | `MAX_THINKING_TOKENS=<n>` (env) | reasoning-effort flag | n/a | thinking config | n/a |
| Token spend check (Steps 1, 9) | `/cost` | session footer | status bar | `/stats` | `gh copilot usage` |
| Prefix cache (Step 2) | automatic + `cache_control` SDK flag | automatic on exact prefix | automatic | implicit context cache | via provider |
Notes:
- Env vars apply to the host runtime's settings file (e.g.,
`~/.claude/settings.json`'s `env` block for Claude Code). Other hosts have
their own settings location.
- `n/a` means the feature is not exposed on that host's CLI surface today —
the underlying tactic may still apply via direct provider SDK use.
- MCP-style tool servers count against tool-definition tokens on every host
that supports them. Keep enabled count modest (around 10 per project is a
reasonable working ceiling); see Step 3 for the broader instruction-file diet.
## Rules
**Hard rules (never violate):**
- Never prune system instructions, safety headers, or compliance text
during compression — context-loss jailbreaks are a real risk.
- Never invalidate a stable prefix unintentionally (e.g., by injecting
a timestamp or random ID at the front). If a value must vary per call,
put it at the end.
- Tokens-per-task is the optimization target, not tokens-per-request.
Don't celebrate a 30% per-call saving that quintuples the loop count.
- Don't remove an instruction just because the model "should know it" if
doing so changes user-visible behavior. Verify with a real example.
**Preferences (override-able):**
- Apply tactics in the listed order; stop as soon as the goal is met.
- Prefer reference-fetch (file paths the model can re-read) over
inline-dumping large content.
- For repeated calls, optimizing the cacheable prefix usually beats any
other single change.
- When unsure between two tactics, pick the one that makes the prompt
shorter at the *start* (caching benefits) over the *end*.
## Output
This skill produces a diagnosis and a prioritized tactic list — not a
mechanical rewrite. The expected hand-off is:
1. A bucketed measurement of current token usage (Step 1).
2. The 2–3 tactics from Steps 2–8 most likely to move the dominant bucket.
3. Concrete proposed edits (system-prompt diff, cache-breakpoint
placement, model-routing rule, etc.) for the user to accept or reject.
4. A re-measurement after changes are applied (Step 9).
The abstract consumer is **whichever agent runtime is hosting this
session** (Claude Code, Codex, Cursor, Gemini CLI, Copilot). Tactics that
reference provider-specific features (cache_control, thinking budgets,
sub-agents) carry their own provider tag in the reference files.
## Progressive disclosure
Heavy detail lives in subfolders, loaded only on demand:
- [`references/measurement-and-budgets.md`](references/measurement-and-budgets.md)
— load during Step 1. Heuristic-vs-exact estimation, complexity-based
output multipliers, depth-tier math.
- [`references/prompt-caching.md`](references/prompt-caching.md)
— load during Step 2. Cache-breakpoint placement, TTL, minimum block
sizes, invalidation triggers, debugging cache misses.
- [`references/context-management.md`](references/context-management.md)
— load during Steps 3–5. Slimming rules with examples, structured
summary template for compaction, observation-masking patterns, image
resolution guidance.
- [`references/agent-loop-patterns.md`](references/agent-loop-patterns.md)
— load during Steps 6–7. Model-routing decision table, parallel
tool-call pattern, sub-agent delegation, iteration caps, background
job patterns.
- [`references/output-and-verification.md`](references/output-and-verification.md)
— load during Steps 8–9. Depth-tier picker template + default-tier
heuristic, bridge-phrase strip list, structured-output rules of
thumb, stop-sequence cookbook by use case, thinking-budget
calibration table, re-measurement template (incl. CSV variant),
negligible-delta troubleshooting matrix, "when to call
optimization done" exit conditions.
No `assets/` or `scripts/` ship in v1. Token estimation is heuristic and
described inline; if a project later wants exact counts, call the host
provider's `count_tokens` endpoint directly.
## Body budget
Soft targets — not hard caps. Grow the body when the topic genuinely
warrants it; move overflow to `references/<topic>-extras.md` rather
than truncating signal.
- `description`: agentskills.io spec caps this at 1,024 chars; that one
is an external hard limit, not a soft target.
- Body: aim for ~500 lines / ~5,000 tokens as a comfortable read.
- Total file size: stay well below ~30,000 chars so the body still
parses cleanly when re-rendered as a Copilot custom agent.