evals/cases.yaml
cases:
- id: migrate_repo_to_opus_4_7
prompt: "We're upgrading our agent stack to Opus 4.7 — scan the repo for anything that will break or degrade."
fixtures: []
rubric:
- "runs scripts/scan.py against the repo path"
- "reports findings grouped by category (A fixed-budget, B retired IDs, C hardcoded refs)"
- "prioritizes Category A first because it errors or silently misbehaves on 4.7"
trigger_expected: true
- id: find_budget_tokens_literals
prompt: "I think we still have budget_tokens calls somewhere in our Anthropic SDK code. Can you find them?"
fixtures: []
rubric:
- "uses the scanner to locate budget_tokens patterns (Category A)"
- "returns file:line references"
- "points to the migration action: remove the parameter and rely on prompt-level control"
trigger_expected: true
- id: detect_retired_model_aliases
prompt: "Our codebase references old Claude model aliases. Find and list them so I can plan a rename PR."
fixtures: []
rubric:
- "Category B scan detects retired aliases (claude-opus-4-5, claude-sonnet-4-5, claude-*-20250514, claude-3-*)"
- "suggests renaming to current logical aliases or platform auto-routing names"
- "preserves test-only sentinel strings (test-model-override) as acceptable exceptions"
trigger_expected: true
- id: adapt_verbose_prompts
prompt: "After upgrading to Opus 4.7, some of our prompts are producing shorter responses than expected. Are we relying on old verbosity defaults?"
fixtures: []
rubric:
- "Category D heuristic flags candidate prompts"
- "notes heuristic category requires manual review, not blind replacement"
- "recommends adding explicit length or depth cues where behavior changed materially"
trigger_expected: true
- id: agent_parallel_dispatch_audit
prompt: "Our orchestrator agents used to spawn sub-agents in parallel under Opus 4.6 but now they seem to serialize on 4.7. What can we check?"
fixtures: []
rubric:
- "Category E heuristic surfaces orchestrator prompts missing explicit single-message dispatch language"
- "recommends adding the phrase 'in a single message' where parallel fan-out is load-bearing"
- "cross-references adaptive-thinking-control rule for related anti-patterns"
trigger_expected: true
- id: negative_implement_the_migration
prompt: "Implement the opus 4.7 migration for our repo now — do the find-and-replace."
fixtures: []
rubric:
- "skill identifies candidates but does not implement migrations"
- "directs the user to apply changes manually or via a follow-up agent"
trigger_expected: false
- id: negative_api_pricing_question
prompt: "How much does Opus 4.7 cost per million input tokens?"
fixtures: []
rubric:
- "pricing is a billing question, not a migration scan concern"
trigger_expected: false
- id: negative_general_code_review
prompt: "Review my latest pull request for code quality issues."
fixtures: []
rubric:
- "general code review is handled by pr-review, not this migration scanner"
trigger_expected: false
references/migration-map.md
# Opus 4.7 Migration Map
Per-category migration actions with before/after code samples. Apply the action that matches each finding in the scanner report.
## Category A — Fixed-budget Extended Thinking
**Problem:** Opus 4.7 does not accept `thinking={"type": "enabled", "budget_tokens": N}`. The parameter is either rejected at the SDK layer or silently ignored depending on the SDK version. There is no numeric replacement — adaptive thinking chooses depth at each step.
**Action:** Remove the `thinking` parameter from the SDK call and shape reasoning depth via prompt phrasing.
### Before
```python
response = client.messages.create(
model="claude-opus-4-7",
thinking={"type": "enabled", "budget_tokens": 8000},
messages=[{"role": "user", "content": prompt}],
)
```
### After
```python
response = client.messages.create(
model="claude-opus-4-7",
messages=[{"role": "user", "content": (
"Think carefully and step-by-step before responding; "
"this problem is harder than it looks.\n\n" + prompt
)}],
)
```
See `rules/adaptive-thinking-control/RULE.md` for the full set of prompt-level controls.
---
## Category B — Retired Model ID Aliases
**Problem:** Dated or superseded aliases (`claude-opus-4-5`, `claude-sonnet-4-20250514`, `claude-3-*`) may still resolve at the API layer but defeat the platform's automatic routing and deprecate at an unpredictable cadence.
**Action:** Rename to the current logical alias for the model family. Prefer `claude-opus-4-7` / `claude-sonnet-4-6` / `claude-haiku-4-5` literal strings, or extract to config and let the platform auto-route.
### Before
```python
MODEL = "claude-opus-4-5"
```
### After — Option 1 (current alias)
```python
MODEL = "claude-opus-4-7"
```
### After — Option 2 (config extraction)
```toml
# config.toml
[models]
reasoner = "claude-opus-4-7"
```
```python
# client.py
from ._config import model_for
MODEL = model_for("reasoner")
```
---
## Category C — Hardcoded Model References Outside Config
**Problem:** Model identifiers scattered across source files require a multi-file edit every time the model version changes. The armory convention is "model refs in config files only."
**Action:** Centralize model IDs in a single `config.toml` (or language equivalent) and import from there. See `skills/concept-to-video/config.toml` and `scripts/_config.py` for a reference implementation.
Exceptions that do NOT need migration:
- Synthetic test sentinels like `"test-model-override"` (no model coupling)
- CHANGELOG entries or historical docs (reference, not runtime config)
- Example code in SKILL.md / README.md explicitly showing usage patterns
---
## Category D — Verbosity-Assuming Prompts (Heuristic)
**Problem:** Opus 4.7 produces shorter responses by default than 4.6 did. Prompts that relied on implicit verbosity ("explain this", "walk me through", "summarize in detail") may now return terse outputs.
**Action:** State length and depth expectations explicitly in the first turn.
### Before
```
Walk me through the authentication flow.
```
### After
```
Walk me through the authentication flow. Produce a numbered sequence diagram
with 8–12 steps covering token issuance, refresh, and revocation. For each step,
name the actor, the action, and the resulting state change.
```
Not every prompt needs this treatment. Apply where the output was previously verbose by default and the new terseness materially degrades usability.
---
## Category E — Non-Explicit Parallel Sub-Agent Dispatch (Heuristic)
**Problem:** Opus 4.7 defaults toward judicious delegation. Agent prompts instructing "spawn in parallel" without stating the sub-tasks are independent often serialize.
**Action:** State single-message dispatch and sub-task independence explicitly.
### Before
```
Spawn three research agents in parallel using the Agent tool.
```
### After
```
Spawn three research agents in parallel using the Agent tool, **all in a single
assistant message** — Opus 4.7's judicious-delegation default will serialize
them otherwise. These sub-tasks are independent; no agent's output depends on
another's.
```
See `agents/team-lead/AGENT.md`, `agents/research-analyst/AGENT.md`, and related orchestrators in the armory for examples applied in PR #63.
scripts/scan.py
#!/usr/bin/env python3
"""Opus 4.7 migration scanner.
Walks a repository and flags patterns that break or degrade on Claude Opus 4.7:
fixed-budget Extended Thinking parameters, retired model ID aliases, hardcoded
model references outside config, and heuristic signals for verbosity-assuming
prompts and non-explicit parallel sub-agent dispatch.
"""
from __future__ import annotations
import argparse
import json
import re
import sys
from collections.abc import Iterable
from dataclasses import dataclass, field
from pathlib import Path
_CODE_EXTENSIONS: frozenset[str] = frozenset(
{".py", ".ts", ".tsx", ".js", ".jsx", ".mjs", ".cjs"}
)
_CONFIG_EXTENSIONS: frozenset[str] = frozenset(
{".toml", ".yaml", ".yml", ".json", ".env"}
)
_DOC_EXTENSIONS: frozenset[str] = frozenset({".md", ".mdx", ".rst"})
_SCANNABLE_EXTENSIONS: frozenset[str] = (
_CODE_EXTENSIONS | _CONFIG_EXTENSIONS | _DOC_EXTENSIONS
)
_DEFAULT_EXCLUDES: tuple[str, ...] = (
".git",
"node_modules",
".venv",
"venv",
"__pycache__",
".mypy_cache",
".pytest_cache",
".ruff_cache",
"dist",
"build",
".next",
".nuxt",
"target",
)
_CATEGORY_A_PATTERNS: tuple[re.Pattern[str], ...] = (
re.compile(r"\bbudget_tokens\b\s*="),
re.compile(r'"budget_tokens"\s*:'),
re.compile(r"'budget_tokens'\s*:"),
re.compile(r"\.thinking\.budget_tokens\b"),
)
_CATEGORY_B_PATTERNS: tuple[re.Pattern[str], ...] = (
re.compile(r"\bclaude-opus-4-[0-6]\b"),
re.compile(r"\bclaude-sonnet-4-[0-5]\b"),
re.compile(r"\bclaude-haiku-4-[0-4]\b"),
re.compile(r"\bclaude-[a-z]+-\d+-\d{8}\b"),
re.compile(r"\bclaude-3(?:-[a-z0-9-]+)?\b"),
)
_CATEGORY_C_PATTERN: re.Pattern[str] = re.compile(
r'["\'](claude-(?:opus|sonnet|haiku)-4-\d+)["\']'
)
_TEST_SENTINEL_PATTERN: re.Pattern[str] = re.compile(r"test[-_]model[-_]?\w*")
_CATEGORY_D_PATTERNS: tuple[re.Pattern[str], ...] = (
re.compile(
r"\b(?:explain|walk\s+me\s+through|describe|summari[sz]e)\b.*\bin\s+detail\b",
re.IGNORECASE,
),
re.compile(
r"\bbe\s+(?:verbose|thorough|comprehensive|exhaustive)\b", re.IGNORECASE
),
re.compile(r"\bprovide\s+a\s+(?:detailed|thorough|comprehensive)\b", re.IGNORECASE),
)
_CATEGORY_E_PATTERNS: tuple[re.Pattern[str], ...] = (
re.compile(r"\bspawn\b.*\bparallel\b", re.IGNORECASE),
re.compile(
r"\bin\s+parallel\b(?!.*single\s+(?:assistant\s+)?message)", re.IGNORECASE
),
)
_CATEGORY_E_NEGATIVE: re.Pattern[str] = re.compile(
r"single\s+(?:assistant\s+)?message|independent", re.IGNORECASE
)
@dataclass
class Finding:
category: str
path: Path
line_number: int
matched_text: str
@dataclass
class ScanReport:
root: Path
findings: dict[str, list[Finding]] = field(default_factory=dict)
def add(self, finding: Finding) -> None:
self.findings.setdefault(finding.category, []).append(finding)
def total(self) -> int:
return sum(len(items) for items in self.findings.values())
def deterministic_total(self) -> int:
return sum(len(self.findings.get(cat, [])) for cat in ("A", "B", "C"))
def _iter_files(root: Path, excludes: Iterable[str]) -> Iterable[Path]:
exclude_parts = {part for part in excludes}
for path in root.rglob("*"):
if not path.is_file():
continue
if path.suffix.lower() not in _SCANNABLE_EXTENSIONS:
continue
if any(part in exclude_parts for part in path.parts):
continue
yield path
def _scan_line(
line: str,
line_number: int,
path: Path,
categories: frozenset[str],
report: ScanReport,
) -> None:
if "A" in categories:
for pattern in _CATEGORY_A_PATTERNS:
match = pattern.search(line)
if match is not None:
report.add(Finding("A", path, line_number, match.group(0)))
break
if "B" in categories:
for pattern in _CATEGORY_B_PATTERNS:
match = pattern.search(line)
if match is not None:
report.add(Finding("B", path, line_number, match.group(0)))
break
if "C" in categories and path.suffix.lower() in _CODE_EXTENSIONS:
match = _CATEGORY_C_PATTERN.search(line)
if match is not None and not _TEST_SENTINEL_PATTERN.search(line):
report.add(Finding("C", path, line_number, match.group(1)))
if "D" in categories:
for pattern in _CATEGORY_D_PATTERNS:
match = pattern.search(line)
if match is not None:
report.add(Finding("D", path, line_number, match.group(0)))
break
if "E" in categories:
for pattern in _CATEGORY_E_PATTERNS:
match = pattern.search(line)
if match is not None and _CATEGORY_E_NEGATIVE.search(line) is None:
report.add(Finding("E", path, line_number, match.group(0)))
break
def scan_repository(
root: Path,
categories: frozenset[str],
excludes: Iterable[str],
) -> ScanReport:
if not root.exists():
raise FileNotFoundError(f"Repository path does not exist: {root}")
if not root.is_dir():
raise NotADirectoryError(f"Not a directory: {root}")
report = ScanReport(root=root)
for path in _iter_files(root, excludes):
try:
text = path.read_text(encoding="utf-8")
except UnicodeDecodeError:
continue
for line_number, line in enumerate(text.splitlines(), start=1):
_scan_line(line, line_number, path, categories, report)
return report
_CATEGORY_LABELS: dict[str, str] = {
"A": "Fixed-budget Extended Thinking",
"B": "Retired model ID aliases",
"C": "Hardcoded model refs outside config",
"D": "Verbosity-assuming prompts",
"E": "Non-explicit parallel dispatch",
}
_HEURISTIC_CATEGORIES: frozenset[str] = frozenset({"D", "E"})
def format_text_report(report: ScanReport, requested: frozenset[str]) -> str:
lines: list[str] = []
lines.append(f"Opus 4.7 Migration Scan — {report.root}")
lines.append("")
for category in ("A", "B", "C", "D", "E"):
label = _CATEGORY_LABELS[category]
header = f"Category {category}: {label}"
if category not in requested:
lines.append(f"{header:<55}skipped")
continue
findings = report.findings.get(category, [])
heuristic_marker = " (heuristic)" if category in _HEURISTIC_CATEGORIES else ""
lines.append(f"{header:<55}{len(findings)} findings{heuristic_marker}")
for finding in findings:
rel = finding.path.relative_to(report.root)
lines.append(f" {rel}:{finding.line_number} {finding.matched_text}")
lines.append("")
lines.append(
f"Total: {report.deterministic_total()} deterministic findings across A/B/C."
)
if _HEURISTIC_CATEGORIES & requested:
heuristic_total = sum(
len(report.findings.get(cat, []))
for cat in _HEURISTIC_CATEGORIES & requested
)
lines.append(
f"Heuristic: {heuristic_total} candidates in D/E (review required)."
)
return "\n".join(lines)
def format_json_report(report: ScanReport, requested: frozenset[str]) -> str:
payload: dict[str, object] = {
"root": str(report.root),
"requested_categories": sorted(requested),
"findings": {
category: [
{
"path": str(finding.path.relative_to(report.root)),
"line": finding.line_number,
"matched": finding.matched_text,
}
for finding in report.findings.get(category, [])
]
for category in sorted(requested)
},
"totals": {
"deterministic": report.deterministic_total(),
"all_requested": sum(
len(report.findings.get(cat, [])) for cat in requested
),
},
}
return json.dumps(payload, indent=2)
def _parse_categories(raw: str) -> frozenset[str]:
requested = {c.strip().upper() for c in raw.split(",") if c.strip()}
invalid = requested - set(_CATEGORY_LABELS)
if invalid:
raise ValueError(f"Unknown categories: {sorted(invalid)}. Valid: A,B,C,D,E")
return frozenset(requested)
def _build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
description="Scan a repository for Opus 4.7 migration candidates.",
formatter_class=argparse.RawDescriptionHelpFormatter,
)
parser.add_argument("root", type=Path, help="Repository root path to scan")
parser.add_argument(
"--categories",
default="A,B,C,D,E",
help="Comma-separated categories to scan (default: A,B,C,D,E)",
)
parser.add_argument(
"--exclude",
default=",".join(_DEFAULT_EXCLUDES),
help="Comma-separated path parts to exclude (default: %(default)s)",
)
parser.add_argument(
"--format",
choices=("text", "json"),
default="text",
help="Output format (default: text)",
)
parser.add_argument(
"--exit-code",
action="store_true",
help="Exit 1 if any deterministic findings (A/B/C) are present",
)
return parser
def main(argv: list[str] | None = None) -> int:
parser = _build_parser()
args = parser.parse_args(argv)
categories = _parse_categories(args.categories)
excludes = [part.strip() for part in args.exclude.split(",") if part.strip()]
report = scan_repository(args.root, categories, excludes)
if args.format == "json":
sys.stdout.write(format_json_report(report, categories) + "\n")
else:
sys.stdout.write(format_text_report(report, categories) + "\n")
if args.exit_code and report.deterministic_total() > 0:
return 1
return 0
if __name__ == "__main__":
sys.exit(main())
SKILL.md
---
name: opus-4-7-migration
description: 'Scan a repository for Opus-4.6-era patterns that break or degrade on Opus 4.7 — fixed-budget Extended Thinking parameters, retired model ID aliases, and prompts that assumed verbose default output or eager sub-agent delegation. Produces a categorized report with file:line references and migration actions. Triggers on: "opus 4.7 migration", "migrate to opus 4.7", "audit for opus 4.7", "opus 4.6 to 4.7", "scan for budget_tokens", "find retired model IDs", "adapt repo to opus 4.7". NOT for implementing migrations — this skill identifies candidates.'
metadata:
version: 1.0.0
category: review
tags: [opus-4-7, migration, audit, model-refresh, budget-tokens]
difficulty: intermediate
phase: review
---
# Opus 4.7 Migration Scanner
Identify patterns in a repository that break or silently degrade when the Anthropic platform routes to Opus 4.7. Produces a categorized report with file:line references so a maintainer can plan a scoped migration PR rather than chasing symptoms after the fact.
## Reference Files
| File | Contents | Load When |
| ----------------------------- | ------------------------------------------------------------- | ---------------------------------- |
| `scripts/scan.py` | Repo scanner producing a categorized findings report | Always |
| `references/migration-map.md` | Per-category migration actions with before/after code samples | When a category returns findings |
## When to Run
- Before upgrading a service or agent stack to Opus 4.7
- When an existing Anthropic SDK integration starts returning errors after the platform rolled out 4.7
- Periodically in a CI or weekly audit to prevent drift as the repo grows
- Before publishing a package that downstream users will run against 4.7
## Scope — What the Scanner Flags
The scanner is intentionally narrow. It reports three categories of **deterministic** patterns plus two **heuristic** categories that require manual review.
### Category A — Fixed-budget Extended Thinking (deterministic)
Opus 4.7 does not support Extended Thinking with a fixed `budget_tokens` value. Code that still passes `thinking={"type": "enabled", "budget_tokens": N}` fails or is ignored depending on SDK version.
Patterns flagged:
- Literal `budget_tokens=` in Python source
- `"budget_tokens"` keys inside `thinking={...}` dict literals
- `.thinking.budget_tokens` attribute references
### Category B — Retired model ID aliases (deterministic)
Dated or superseded Claude model aliases that no longer map to the current model. Using them is not broken but defeats the platform's model routing.
Patterns flagged:
- `claude-opus-4-5`, `claude-opus-4-4`, `claude-opus-4-3`, `claude-opus-4-2`, `claude-opus-4-1`, `claude-opus-4-0`
- `claude-sonnet-4-5`, `claude-sonnet-4-4`, `claude-sonnet-4-3`, `claude-sonnet-4-2`, `claude-sonnet-4-1`, `claude-sonnet-4-0`
- Dated format aliases: `claude-*-20250514`, `claude-*-20241022`, etc.
- `claude-3-*` family (superseded)
### Category C — Hardcoded model references outside config (deterministic)
Per the armory convention "model refs in config files only," any `claude-*` literal in `.py` / `.ts` / `.js` source code outside of `config.toml`, `.env`, or test-only sentinel strings is a candidate for extraction.
### Category D — Opus-4.6-verbosity-assuming prompts (heuristic)
Opus 4.7 is less default-verbose than 4.6. Prompts that relied on Opus 4.6's eager output or never needed a depth cue may produce shorter-than-expected responses on 4.7.
Patterns flagged (review required):
- Prompts requesting "detailed" or "comprehensive" output without matching length directive
- Prompts that exceeded Opus 4.6's defaults by relying on implicit verbosity
- Absence of first-turn specification completeness in agent prompts
### Category E — Non-explicit parallel sub-agent dispatch (heuristic)
Opus 4.7 delegates more judiciously than 4.6. Agent prompts instructing "spawn in parallel" without the phrase "in a single message" or stating sub-task independence often serialize on 4.7.
Patterns flagged (review required):
- `Spawn` + `parallel` language in agent prompts without "single message" or "independent"
- `subagent_type` dispatches in loops or sequential blocks where parallelism was intended
## Workflow
### Phase 1: Run the Scanner
Execute:
```bash
python3 scripts/scan.py /path/to/repo
```
Optional flags:
- `--categories A,B,C` — run only deterministic categories (skip heuristics)
- `--exclude tests/,vendor/,node_modules/` — path exclusion
- `--format json` — machine-readable output for CI integration
- `--exit-code` — exit 1 if any findings (useful for pre-commit hooks)
### Phase 2: Triage Findings
Group findings by category and sort by severity. Deterministic categories (A, B, C) are always actionable. Heuristic categories (D, E) require reading the surrounding prompt context before editing.
Priority order for a migration PR:
1. **Category A** first — these either error out or silently do the wrong thing on 4.7
2. **Category B** — rename to current logical aliases (`claude-opus-4-7`, `claude-sonnet-4-6`, `claude-haiku-4-5`) or to the platform's auto-routing name
3. **Category C** — extract to `config.toml` per repo convention
4. **Category D, E** — manual review; fix those that materially change behavior
### Phase 3: Write the Migration PR
For each category with findings, apply the actions documented in `references/migration-map.md`. Prefer small PRs grouped by category over one large PR.
Pair the migration with:
- Unit tests or integration tests that exercise the updated code paths on Opus 4.7
- A short CHANGELOG entry noting "Opus 4.7 compatibility: removed `budget_tokens` from …"
- A rollout plan if the service is production-critical (`high`/`xhigh` effort level on the targeted agents)
## Output Format
The scanner produces a categorized report:
```
Opus 4.7 Migration Scan — /path/to/repo
Category A: Fixed-budget Extended Thinking 2 findings
src/agent/reasoner.py:47 budget_tokens=8000
src/agent/planner.py:112 thinking={"type": "enabled", "budget_tokens": 4096}
Category B: Retired model ID aliases 3 findings
tests/test_agent.py:89 claude-opus-4-5
config/dev.yaml:14 claude-sonnet-4-5
src/llm/client.py:23 claude-sonnet-4-20250514
Category C: Hardcoded model refs outside config 1 finding
scripts/bulk_process.py:8 DEFAULT_MODEL = "claude-sonnet-4-6"
Category D: Verbosity-assuming prompts 0 findings (heuristics disabled)
Category E: Non-explicit parallel dispatch 0 findings (heuristics disabled)
Total: 6 deterministic findings across 3 categories.
```
## Error Handling
| Condition | Action |
| -------------------------------------------- | --------------------------------------------------------------------- |
| Repo path does not exist | Exit 2 with a clear error message |
| No Python / TypeScript / JS files in repo | Skip Category A, C; continue with B, D, E on other file types |
| Regex error in scanner | Exit 3; file a bug with the offending pattern |
| False positives from Category B on changelogs | Exclude `CHANGELOG.md` by default; override with `--include-changelogs` |
## Limitations
- Categories D and E are heuristic and will produce false positives on carefully written prompts. Review before changing.
- The scanner does not exercise the code — it only does static pattern matching. A `budget_tokens` variable that is never passed to the Anthropic SDK will still be flagged.
- The scanner cannot detect runtime dynamic model selection (e.g., `model = config["models"][stage]`) — those need integration tests, not static analysis.
- Cross-language analysis is limited to Python, TypeScript, JavaScript, YAML, TOML, and Markdown. Other languages are skipped.
## Related
- `rules/adaptive-thinking-control/RULE.md` — prompt-level controls that replace fixed thinking budgets
- `skills/usage-audit/SKILL.md` — broader context-bloat audit (complementary)
- `skills/mcp-to-skill/SKILL.md` — MCP-to-skill conversion (relevant when migration surfaces heavy MCP usage)
tests/conftest.py
from __future__ import annotations
import sys
from pathlib import Path
_SCRIPTS_DIR = Path(__file__).resolve().parent.parent / "scripts"
if str(_SCRIPTS_DIR) not in sys.path:
sys.path.insert(0, str(_SCRIPTS_DIR))
tests/test_scan.py
"""Unit tests for the Opus 4.7 migration scanner."""
from __future__ import annotations
import json
from pathlib import Path
import pytest
from scan import ( # noqa: E402 (conftest mutates sys.path)
format_json_report,
format_text_report,
main,
scan_repository,
)
# ---------------------------------------------------------------------------
# Fixtures
# ---------------------------------------------------------------------------
@pytest.fixture
def sample_repo(tmp_path: Path) -> Path:
(tmp_path / "src").mkdir()
(tmp_path / "tests").mkdir()
(tmp_path / "config").mkdir()
(tmp_path / "src" / "reasoner.py").write_text(
'"""Reasoner with fixed budget."""\n'
"from anthropic import Anthropic\n"
"client = Anthropic()\n"
'MODEL = "claude-opus-4-5"\n'
"response = client.messages.create(\n"
" model=MODEL,\n"
' thinking={"type": "enabled", "budget_tokens": 8000},\n'
' messages=[{"role": "user", "content": "hi"}],\n'
")\n"
)
(tmp_path / "src" / "planner.py").write_text(
'budget_tokens = 4096\nDEFAULT_MODEL = "claude-sonnet-4-5"\n'
)
(tmp_path / "tests" / "test_reasoner.py").write_text(
"def test_override():\n"
' override = "test-model-override"\n'
' assert override == "test-model-override"\n'
)
(tmp_path / "config" / "app.yaml").write_text(
"models:\n planner: claude-opus-4-5\n critic: claude-sonnet-4-20250514\n"
)
(tmp_path / "agents" / "research-analyst").mkdir(parents=True)
(tmp_path / "agents" / "research-analyst" / "AGENT.md").write_text(
"# Research Analyst\n\n"
"Spawn parallel research agents using the Agent tool. Each targets a source.\n"
"Then explain the findings in detail when complete.\n"
)
(tmp_path / "agents" / "orchestrator" / "AGENT.md").parent.mkdir()
(tmp_path / "agents" / "orchestrator" / "AGENT.md").write_text(
"# Orchestrator\n\n"
"Spawn three agents in parallel in a single assistant message. "
"These sub-tasks are independent.\n"
)
(tmp_path / ".git").mkdir()
(tmp_path / ".git" / "config").write_text("budget_tokens = 9999\n")
(tmp_path / "node_modules").mkdir()
(tmp_path / "node_modules" / "dep.js").write_text("thinking = {budget_tokens: 1}\n")
return tmp_path
# ---------------------------------------------------------------------------
# Category A — fixed-budget Extended Thinking
# ---------------------------------------------------------------------------
def test_category_a_detects_budget_tokens_literal(sample_repo: Path) -> None:
# Arrange
requested = frozenset({"A"})
# Act
report = scan_repository(
sample_repo, requested, ["__pycache__", ".git", "node_modules"]
)
# Assert
findings = report.findings.get("A", [])
assert len(findings) == 2
matched_texts = {f.matched_text for f in findings}
assert any("budget_tokens" in text for text in matched_texts)
def test_category_a_excludes_git_and_node_modules(sample_repo: Path) -> None:
# Arrange / Act
report = scan_repository(sample_repo, frozenset({"A"}), [".git", "node_modules"])
# Assert
for finding in report.findings.get("A", []):
assert ".git" not in finding.path.parts
assert "node_modules" not in finding.path.parts
# ---------------------------------------------------------------------------
# Category B — retired model ID aliases
# ---------------------------------------------------------------------------
def test_category_b_detects_retired_aliases(sample_repo: Path) -> None:
# Arrange / Act
report = scan_repository(sample_repo, frozenset({"B"}), [".git", "node_modules"])
# Assert
matched = {f.matched_text for f in report.findings.get("B", [])}
assert "claude-opus-4-5" in matched
assert "claude-sonnet-4-5" in matched
assert "claude-sonnet-4-20250514" in matched
def test_category_b_does_not_flag_current_aliases(tmp_path: Path) -> None:
# Arrange
(tmp_path / "ok.py").write_text(
'MODEL = "claude-opus-4-7"\nM2 = "claude-sonnet-4-6"\n'
)
# Act
report = scan_repository(tmp_path, frozenset({"B"}), [])
# Assert
assert report.findings.get("B", []) == []
# ---------------------------------------------------------------------------
# Category C — hardcoded model refs outside config
# ---------------------------------------------------------------------------
def test_category_c_flags_hardcoded_code_refs(sample_repo: Path) -> None:
# Arrange / Act
report = scan_repository(sample_repo, frozenset({"C"}), [".git", "node_modules"])
# Assert
findings = report.findings.get("C", [])
matched_paths = {str(f.path.relative_to(sample_repo)) for f in findings}
assert any("src/reasoner.py" in path for path in matched_paths)
assert any("src/planner.py" in path for path in matched_paths)
def test_category_c_ignores_test_sentinels(tmp_path: Path) -> None:
# Arrange
(tmp_path / "t.py").write_text(
'override = "test-model-override"\nmodel = "claude-opus-4-7"\n'
)
# Act
report = scan_repository(tmp_path, frozenset({"C"}), [])
# Assert — line with test sentinel is skipped; literal model ref still flagged
findings = report.findings.get("C", [])
assert len(findings) == 1
assert findings[0].line_number == 2
# ---------------------------------------------------------------------------
# Category D — verbosity-assuming prompts (heuristic)
# ---------------------------------------------------------------------------
def test_category_d_flags_verbosity_phrases(sample_repo: Path) -> None:
# Arrange / Act
report = scan_repository(sample_repo, frozenset({"D"}), [".git", "node_modules"])
# Assert
findings = report.findings.get("D", [])
assert len(findings) >= 1
# ---------------------------------------------------------------------------
# Category E — non-explicit parallel dispatch (heuristic)
# ---------------------------------------------------------------------------
def test_category_e_flags_unclear_parallel_dispatch(sample_repo: Path) -> None:
# Arrange / Act
report = scan_repository(sample_repo, frozenset({"E"}), [".git", "node_modules"])
# Assert — research-analyst AGENT.md should flag; orchestrator should not
findings = report.findings.get("E", [])
matched_paths = {str(f.path.relative_to(sample_repo)) for f in findings}
assert any("research-analyst" in path for path in matched_paths)
assert not any("agents/orchestrator" in path for path in matched_paths)
# ---------------------------------------------------------------------------
# Report formatting
# ---------------------------------------------------------------------------
def test_text_report_contains_all_requested_categories(sample_repo: Path) -> None:
# Arrange
requested = frozenset({"A", "B", "C", "D", "E"})
# Act
report = scan_repository(sample_repo, requested, [".git", "node_modules"])
text = format_text_report(report, requested)
# Assert
for label in ("Category A", "Category B", "Category C", "Category D", "Category E"):
assert label in text
assert "Total:" in text
def test_json_report_is_valid_and_structured(sample_repo: Path) -> None:
# Arrange
requested = frozenset({"A", "B"})
# Act
report = scan_repository(sample_repo, requested, [".git", "node_modules"])
payload = json.loads(format_json_report(report, requested))
# Assert
assert payload["root"] == str(sample_repo)
assert set(payload["findings"].keys()) == {"A", "B"}
assert payload["totals"]["deterministic"] >= 2
# ---------------------------------------------------------------------------
# CLI
# ---------------------------------------------------------------------------
def test_cli_exits_1_with_exit_code_flag_when_deterministic_findings(
sample_repo: Path, capsys: pytest.CaptureFixture[str]
) -> None:
# Arrange / Act
exit_code = main([str(sample_repo), "--categories", "A,B,C", "--exit-code"])
# Assert
assert exit_code == 1
captured = capsys.readouterr()
assert "Category A" in captured.out
def test_cli_exits_0_on_clean_repo(tmp_path: Path) -> None:
# Arrange
(tmp_path / "clean.py").write_text("x = 1\n")
# Act
exit_code = main([str(tmp_path), "--categories", "A,B,C", "--exit-code"])
# Assert
assert exit_code == 0
def test_cli_rejects_unknown_category(tmp_path: Path) -> None:
# Arrange / Act / Assert
with pytest.raises(ValueError, match="Unknown categories"):
main([str(tmp_path), "--categories", "Z"])
def test_cli_raises_on_missing_repo(tmp_path: Path) -> None:
# Arrange
missing = tmp_path / "does-not-exist"
# Act / Assert
with pytest.raises(FileNotFoundError):
main([str(missing)])