ce-retune

어떤 스킬인가요?

Retune a skill corpus for a new model, measurement-first: mine the run archive for a baseline, establish a noise floor, audit the corpus adversarially, then cut in measured passes until a pre-registered bar clears. Requires a benchmark harness that can A/B two builds of the corpus; refuses without one.

설치 방법

프로젝트 폴더에서 아래 명령어를 실행하고, 설치할 에이전트를 선택하세요.

설치 명령을 직접 실행해야 적용됩니다. 지원 에이전트와 필요한 권한·라이선스는 제작자의 안내를 확인하세요.

스킬 원문 파일

policy:
  allow_implicit_invocation: false

# Phase 1: mine the archive

Zero model cost. Everything here reads files that already exist on disk. Do not run the harness in this phase. Phase 2 is where the first paid runs happen (`references/noise-floor.md`).

An archive is usually an order of magnitude larger than the experiment you can afford this week. In the engagement this method came from, 745 archived runs were on disk and 458 carried usable traces. That is more evidence than any deliberate sample, sitting unread.

## What the archive must contain

Per run, in whatever form the harness stores it (JSONL transcript, structured log, a directory per run): the ordered tool-call trace, token counts, timestamps, and the final assistant message. If it stores only a pass/fail verdict, the archive is not minable. Say so and go to Phase 2.

Find the archive by asking the harness where it writes run output, or by locating the directory it appends to after a run. Do not assume a schema. Read one run end to end and derive the field paths before writing any extractor.

## Build the phase-marker map first

`phase_trace`, `process_followed`, and the terminal-phase cross-tab are all functions of one artifact that does not exist yet: a table mapping each phase of the corpus's own workflow to the observable side effect that proves it ran. Derive it before writing the extractor, by reading the corpus's spine. Do not infer phases from the logs; that is circular.

| Phase name, in spine order | Marker in the trace |
|---|---|
| `<phase>` | the reference file it must read, the helper it must dispatch, the artifact path it must write, or the sentinel it must emit |

Rules: one marker per phase minimum, each a **tool call or a file that exists**, never a phrase in the assistant's prose. A phase with no observable marker cannot be scored. Record it as unobservable rather than assuming it ran, and carry that list forward. It is the same list Phase 2 needs for the probe's unentered phases and Phase 5 needs for the unmeasured paths. Keep the map in the same directory as the extractor. Every later re-run and every re-scored table depends on it.

## Extract one row per run

Write a script, not a per-run read. Reading 458 transcripts by hand is the failure mode this phase is supposed to avoid.

| Field | How to derive it | Why it pays for itself |
|---|---|---|
| `run_id`, `model`, `settings` | harness metadata | the stratification keys; without them the whole table is one undifferentiated blob |
| `session_id`, and the child id on every dispatch | whatever the trace calls the conversation and the spawned agent | the only way to tell a real runtime boundary from a fictional one. `references/halt-taxonomy.md` opens on this comparison; extract it here or that diagnostic has no data |
| `phase_trace` | ordered list of which corpus phases ran | tells you **where** it stopped, not just that it did. This is the field that localizes the defect |
| `marker_present` | boolean; the corpus's own terminal marker in the final message | the corpus's own claim of completion, separable from whether it is true |
| `output_tokens` | harness token accounting | the variance channel; spread here is a stronger signal than the mean |
| `wall_clock_min` | last timestamp minus first | useless alone, required as a denominator |
| `helper_dispatches` | count of subagent/helper dispatches | how far the run got, and how it got there |
| `max_parallel_dispatch` | most dispatched in a single message | separates fan-out from serial plodding; a corpus that mandates parallelism and never gets it is a finding |
| `final_message` | verbatim, untruncated | halt language lives here and nowhere else. Do not summarize it during extraction |

Derive `phase_trace` from **artifacts, not claims**. A phase ran if the trace shows its side effect: the reference file it must read, the helper it must dispatch, the file it must write. A run asserting "Phase 3 complete" in prose is not evidence. The defect class you are hunting is exactly runs that report success without doing the work.

## Derived metrics

**Tokens per minute is the metric that separates working from stalling.** `output_tokens / wall_clock_min`. Compute it for every run and look for bimodality before looking at anything else.

In the engagement, healthy cells ran 10-12k tokens/min; every failing cell sat at 1.5-3.8k. A run burning hours at 2k tok/min is not thinking hard. It is waiting, because the prose told it a second party would respond. Duration alone cannot show this. A long successful run and a long stalled run look identical until you divide.

Treat those bands as the shape to look for, not as thresholds to import. Find your own corpus's two clusters, then read the gap.

**The two booleans the spine requires, derived independently:**

```
task_done       = the run's expected deliverable exists (artifact on disk, commit, PR, file changed)
process_followed = every required phase appears in phase_trace, in spine order
```

Never let one imply the other, and never let `marker_present` stand in for either. `task_done AND NOT process_followed` is `done-no-workflow` below: the run did the job inline and skipped the corpus. Collapsed into one success column, it reads as a win and hides the most expensive defect in the set.

## Outcome taxonomy

Apply in order. First match wins. `broken` is first so that a harness fault can never be scored as a model failure.

| # | Outcome | Test |
|---|---|---|
| 1 | `broken` | empty transcript, error exit, or `output_tokens` implausibly low for the task (set the floor from the smallest run that did real work, then check the excluded set by hand) |
| 2 | `complete` | `marker_present AND task_done AND process_followed` |
| 3 | `done-but-thin` | `marker_present AND task_done`, closing phases absent from `phase_trace` |
| 4 | `done-no-workflow` | `marker_present AND task_done AND NOT process_followed`; the spine barely ran |
| 5 | `bail-with-handoff` | no marker; `final_message` matches hand-off language (below) |
| 6 | `ran-but-no-marker` | substantial trace, no marker, no hand-off language. Ambiguous; keep the bucket rather than forcing it |
| 7 | `bail-early` | trace ends inside the first phase or two |

Report the `broken` count as its own number every time you report a rate. In the engagement 20% of usable-looking runs were broken, and excluding them falsified the first headline finding. That finding was an apparently clean monotonic relationship between reasoning effort and completion, which was really the broken runs clustering in one cell (see Confounds).

## Halt language in the final message

Match against `final_message`. Starter set, case-insensitive:

```
\b(let me know|just let me know)\b
\b(would you like|shall I|should I)\b.*\?
\b(ready (to|for)|standing by|awaiting)\b
\b(waiting (for|on)) (your|the)\b
\b(next steps? (for|would be))\b
\bI('ll| will) (wait|pause|hold)\b
\bonce you (confirm|approve|reply|respond)\b
\b(remaining|the rest|still needs? to be) (work|done|completed)\b
```

These are **evidence that the run stopped, not proof of why.** A match tells you to open that transcript and find which phase boundary produced it. A workflow that is *supposed* to stop and ask will match every one of these and be behaving correctly. Phase 4 sorts stops by who is actually on the other side. Extend the set with the phrasings your own corpus's prose invites. Grep the corpus for its own hand-off wording and add those.

## Confounds that fool careful analysts

Each of these was believed by someone competent before being ruled out. Run the rule-out before reporting the effect.

| Apparent finding | Why it is suspect | Rule it out by |
|---|---|---|
| "The harness is timing out" | the cheapest explanation, and it makes the corpus innocent | comparing durations. Timed-out runs are **longer** than successes. In the engagement failures averaged 48 min against 88 for successes, so failures were shorter and the timeout hypothesis died. Failing *early* is a behavior, not a limit |
| "More helper dispatches cause success" | `helper_dispatches` is a **collider**: it measures how far the run got. A run that stopped in phase 2 cannot dispatch phase 5's helpers | never reading it as a cause. Use it as a position estimate. If you want the causal claim, it needs a build that changes dispatch behavior and a Phase 2 comparison |
| A clean dose-response across a setting | `broken` runs cluster by cell (one config, one date range, one machine), manufacturing the gradient | re-running the cross-tab with `broken` excluded. If the effect vanishes, it was never there. Also check whether exclusions land evenly across arms |
| "This model is worse at the task" | model identity and workload are often confounded in an archive nobody designed | checking whether the arms ran the *same* task list. Unequal task mixes make any per-model rate meaningless |

## Stratify before believing anything

This is the highest-leverage work in the phase. A single pooled completion rate hides everything worth knowing.

Cross-tabulate `outcome` against, at minimum:

1. **model**, the identity axis
2. **settings**: reasoning effort, temperature, context budget, whatever the harness varies
3. **terminal phase**, the last entry in `phase_trace`, which is where the run died

Read the third one against the first two. In the engagement, model identity explained roughly 3x the variance that reasoning-effort settings did, and the failing models died at **one specific phase boundary** 38-58% of the time while succeeding models died there 0-8%. That single cross-tab localized the defect without reading a line of corpus prose, and it turned Phase 3's audit from a 31-skill sweep into a targeted read.

Watch cell counts. A 3-of-4 cell is not a rate. Label small cells rather than ranking them. If every cell is small, the stratification's output is a hypothesis list, which is still the correct output of this phase.

## Hand off

Carry forward: the `broken`-excluded baseline rate for `complete`, the observed `output_tokens` range, the tokens/min clusters, and the ranked list of phase boundaries where runs die. Phase 2 needs the baseline rate to compute a bar (`references/noise-floor.md`). Phase 3 needs the phase list to know which files to read first.

## The limit

Archive mining is observational. Nobody assigned the conditions, the arms are not balanced, and the confounds above are the ones you thought to check. It produces a baseline rate and a ranked hypothesis list. It cannot attribute a cause, and a rate computed from it is not a bar until Phase 2 shows how wide the noise is. Do not credit a change against an archive number alone.

# Phase 3: Adversarial Corpus Audit

Produces one finding set per corpus unit, each finding carrying a defender ruling. That ruling set is the input to Phase 4's passes. Nothing here edits a file.

The audit is not evidence. It ranks candidates; only the harness says whether a cut helped. Do not let a large finding count imply a large effect. A corpus's run-to-run spread on fixed inputs (`references/noise-floor.md`) is the same whether the audit found 6 findings or 600.

## Dispatch shape

**The two waves must run as separate dispatched agents.** A defender's ruling is only worth recording when it can genuinely disagree with the proposal that reached it, which one context holding both roles cannot do. If the host exposes no way to run them as independent agents, report that as a blocker and stop the audit. Do not run the waves inline and hand the result to Phase 4. The same context arguing both sides still emits confident `cut` rulings, and Phase 4 deletes on them. That is the demolition this reference warns about below, arriving through the dispatch layer instead of a weak model tier.

Two waves, one agent per unit each way.

**Wave 1, the proposers.** One agent per corpus unit. Give it the unit's **full directory**, not just its entry file. Conditionally-loaded reference files are usually the bulk of a unit, and the entry file alone hides them. Instruct it to return the finding schema below and nothing else: no summary prose, no recommendations about other units.

**Wave 2, the defenders.** One agent per unit, given that unit's directory *and* its proposal set, with the opposite job: find a reason each targeted line exists. A defender must search three sources, and must say which it searched:

1. The project's own documented learnings and solution docs.
2. The test suite. Grep a distinctive substring of the target text.
3. Version history. Run `git log -S '<substring>'` for the commit that introduced the line, then read that commit message and the PR it belongs to.

Source 3 is unavailable in a corpus checkout with no history, which is the normal shape of an installed or vendored copy. A defender working without history must say so in `sources_searched` and cannot return `cut` on the strength of the other two alone. That combination is "no provenance found in two of three sources", which is a verification task, not a cut. Point defenders at a checkout that has history, or record the whole audit's provenance basis as partial.

**Pipeline the waves.** Start a unit's defense as soon as its proposal set returns. Do not wait for wave 1 to finish; the two waves share no state across units. It is 2N dispatches on N units, so plan the wave count against the host's concurrency cap (`references/workflow-shapes.md`).

Keep defenders on a capable model tier. A defender that cannot read a test suite and reconstruct intent from a commit message returns "no provenance found" for everything, which silently converts the audit into a demolition.

## Finding schema

Per finding:

| Field | Shape |
|---|---|
| `id` | stable, `<unit>-<nnn>`; defenses and later passes reference findings by id |
| `target` | `path/to/file.md:line` or a line range |
| `class` | exactly one value from the closed enum below |
| `size` | estimated words removed |
| `text` | the offending text verbatim, long enough to relocate after other edits land |
| `why_unneeded` | what a capable model does here without being told |
| `replacement` | the thinner line, or empty for pure deletion |
| `risk` | low / medium / high; high means it touches a contract, a guard, or a string a test may pin |

Per unit, in addition to the findings:

- `must_survive`: every line that must not be cut, each with its reason. See "Protocol regardless of model tier" below for what belongs here.
- `halt_sites`: every place where prose could make an orchestrating model end its turn while naming work it did not do. Quote each verbatim. This list is the highest-value output of the whole phase. A unit that reports zero halt sites has almost certainly not looked at its phase boundaries. `references/halt-taxonomy.md` carries the classes.

## Classes worth hunting

Ordered by expected behavior change per finding, not by word count. The last entry yields the most words and the least behavior.

| Class | Diagnostic question |
|---|---|
| `phantom-handoff` | Does the party this sentence hands off to (a reviewer, a caller, a next agent, a consumer of the artifact) exist in this run? |
| `step-machinery` | Would a different order, or skipping the ceremony, produce a different artifact? |
| `capability-restatement` | Would the model do this if the line were deleted? |
| `filler-rationale` | Does the rule survive intact if the sentence after it is removed? |
| `vestigial-mode` | Grep the corpus for a caller that sets this mode, flag, or branch. Is there one? |
| `mandatory-fanout` | Is this helper dispatch or self-verification pass conditional on anything, or does it run every time regardless of need? |
| `cross-unit-duplication` | Is this block near-identical elsewhere, and is factoring it out actually permitted? |
| `oversized-reference` | Could this file's entire content be five lines? |

Two notes that change how the results read:

- **`phantom-handoff` is the class that causes halts.** Prose written as if a second party were waiting teaches the model to stop and wait. Hunt it at phase boundaries first.
- **Separate always-loaded prose from conditionally-loaded references when you tally `size`.** They cost differently. Always-loaded prose is context cost on every run; a conditionally-loaded file is runtime cost only when loaded. A corpus can be mostly conditional, which makes raw word count overstate context cost and understate runtime cost. Report the split.

## Defender rulings

Exactly three, one per finding:

- **`cut`**: a real search over all three sources found no provenance. The proposal stands.
- **`reduce`**: the constraint is real and the prose states it at several times the length needed. The defender returns the minimal form that preserves the constraint.
- **`keep`**: concrete citable provenance a capable model could not infer, such as a test asserting it, a documented learning, or a commit that added it to fix a named bug. The ruling must carry the citation: path, test name, or sha.

A defender returns one row per finding, in these fields, and nothing else:

| Field | Shape |
|---|---|
| `id` | the proposer's finding id, unchanged; a ruling that cannot be joined back is discarded |
| `ruling` | `cut` / `reduce` / `keep` |
| `sources_searched` | which of the three, named; plus the query used, so an empty search is visible |
| `citation` | required on `keep`: path, test name, or sha. Empty is not a `keep` |
| `minimal_form` | required on `reduce`: the shortest text preserving the constraint |
| `pinning_test` | test path and assertion if a grep of the target text hits the suite, else empty |

**"A weaker model might need it" is not grounds for `keep`. Only citable provenance is.** Without that rule every line survives, because a defender asked to defend can always imagine a reader who needed the line. Where the unmeasured weaker tier is a genuine concern, the ruling is still `cut` or `reduce`, and the tier becomes a logged verification task in the Phase 5 limits report, not a keep.

Symmetrically: a defender who returns `cut` without naming which sources it searched has not searched. Send it back once. Treat a second empty search as an unaudited unit and say so.

## Protocol regardless of model tier

For three categories the default inverts: **absence of provenance is not grounds to cut.** A capable model cannot re-derive them, so keep them even when no citation turns up. This is the one unrecoverable mistake in the phase. Everything else a later pass can put back after a failed run, and these fail silently.

1. **Machine-readable strings other units parse**: field names, enum values, output paths, greppable markers, sentinel lines. These are arbitrary shared conventions. Nothing makes them true except agreement, so no amount of model capability recovers a changed one.
2. **Security guards.**
3. **Platform gotchas where the wrong behavior looks like success.** Their unifying property: *the model cannot discover them by trying*, because trying returns something that looks fine. A flag that returns empty instead of erroring, so a run reports zero findings and calls itself clean. A config key whose commented-out example matched a naive substring check and silently forced every user into the wrong mode.

What *is* cuttable around all three is the justification clause, the sentence explaining that a separate consumer is waiting on the string. Keep the data, cut the story about who wants it. That is usually a `reduce`, and it is frequently also a `phantom-handoff`.

`cross-unit-duplication` collides with this category more than any other class. Before proposing a factor-out, check whether the duplication is mandated: a documented decision, or a test that forbids sharing the block. The most-duplicated block in a corpus is often the one thing that must stay duplicated. In the engagement it was a security guard whose duplication a test explicitly required, and the largest duplication mandate was itself the documented fix for a bug that had regressed twice.

## Synthesis: rank contradiction above confirmation

The synthesis leads with what contradicts the premise the audit started from, before any cut list. Answer three questions explicitly:

- **Which proposals did defenders save, with citations?** Report the count and the strongest saves. A defense rate near zero means the defenders did not search, not that the corpus is pure junk.
- **Which unit was leaner than its word count implied?** Use the always-loaded / conditionally-loaded split. The largest unit by words is often not the largest by context cost.
- **Where does the corpus already contain a documented argument against its own ceremony that nobody had grepped for?** Corpora that document their own decisions accumulate these. A duplication mandate that is itself the recorded fix for a bug that regressed twice is the shape to look for. The ceremony you were about to cut may be a scar.

A synthesis that only confirms the premise is a finding about the audit, not about the corpus. Say so rather than shipping it.

**The per-unit partition does not carry forward.** Findings arrive grouped by unit because that is how they were gathered. Phase 4 groups them by problem, and one problem's findings cross many units (`references/cut-passes.md`). Emit the ruled findings as a flat, id-addressable set with `class` and `target` intact so a pass can select across units. Handing Phase 4 a per-unit bundle produces one agent per unit editing one problem each, which is the collision shape Phase 4 exists to avoid.

## Test pinning is the rate limiter

Expect roughly half of `reduce` items to be pinned by a test asserting exact strings. That, not authoring effort, sets how many items a pass can carry.

- Grep each `reduce` target's distinctive substring against the test suite **during the audit** and record the pinning test in the finding. Discovering the pin during Phase 4 costs a whole pass.
- Budget the assertion retarget as part of the item, not as follow-up. Moving an assertion from a wording to the invariant it was protecting is often larger than the prose edit.
- **A half-applied structural fix is worse than either endpoint.** If a pass cannot land both the prose change and the retarget, leave the item unstarted. A corpus half-migrated between two shapes has the ceremony of both and the guarantees of neither, and the harness cannot attribute a resulting failure to either shape.
- If the invariant a pinned string was protecting cannot be named, that is a finding to report, not a license to loosen the assertion.

## Two rules separate an audit from a demolition

- **A cut with no provenance found after a real search is a confident cut. A cut the defender saves with a citation is off the list.** Do not relitigate a defended keep.
- **Absence of evidence is weaker than the project's own standard for a change.** Where the guidance requires a reproduced failure or an exact failing path, a search that found nothing is a verification task, not a change. Say which of your cuts rest on that weaker basis.

Dispatch shape: one agent per skill, each reading that skill's full directory and proposing cuts with a target and a reason; then a second agent per skill whose job is the opposite, to **defend the existing prose** using the project's own documented learnings, its tests, and git history. Expect the audit to contradict the premise you started with. That is its value.

# Cut Passes (Phase 4)

A pass applies **one problem class** across the corpus and stops. The work fails in two places: agents overwriting each other, and edits left half-applied. Both are prevented by protocol, not care.

`references/workflow-shapes.md` carries the cross-phase orchestration catalog. This file is the Phase 4 procedure only.

## The pass loop

1. Pick one class from the Phase 3 findings (`references/corpus-audit.md`), or one regression class from `references/halt-taxonomy.md`. One class per pass, no bundling.
2. Write the **file-assignment manifest**: unit -> the agent that edits it -> exact paths. Every copy of a shared asset goes to a single named agent (below).
3. If the rewrite has cross-referencing strings, author the **contract file** first, serially (below).
4. Dispatch one agent per unit through whatever sub-agent primitive the platform provides. Each prompt carries the class, the contract path if any, its own paths, and the forbidden paths.
5. **Reconcile** every block touched (below). This is the step that gets skipped.
6. Run the project's own test suite. A pinned string that disappeared is a finding to report with its test path, never a test to edit.
7. Collect each agent's applied/skipped report. Then measure (Phase 5) and commit the pass alone.

Eight passes landed in the engagement that produced this skill. Every one reduced to the same class. Resist widening a pass to "also fix the obvious thing". A pass that changed two classes cannot be attributed by the next measurement.

## One problem per agent, disjoint files

Fanning out by **problem** looks natural and collides immediately. A single class, say a phrasing that implies an absent reader, appears in twenty files, and the next class appears in eleven of the same twenty. Two agents open one file, both write, the second write wins, and the loss is silent because each agent's own diff looks correct.

Fan out by **unit** instead: one agent edits one skill directory and applies the class everywhere inside it. The file assignment is then a partition of the filesystem, and the invariant is checkable before dispatch: every path appears in exactly one manifest row.

| Manifest column | Content |
|---|---|
| unit | the directory the agent edits, e.g. `skills/<name>/` |
| paths | explicit glob or file list inside it |
| forbidden | shared assets and anything outside `unit`, listed by path |
| class | the one problem being applied |
| contract | path to the canonical mapping, or `none` |

State the forbidden set in the prompt as paths, not as a rule to infer. An agent told "do not touch shared files" will decide for itself what is shared.

## Isolation: separate worktrees or disjoint paths in one tree

Disjoint paths in one tree are enough when nothing an agent runs mutates state outside its own paths. That covers most cut passes: edits are text, the manifest is a partition, and a single tree keeps the diff readable and the commit trivial.

Pay for a worktree (or equivalent per-agent checkout) when any of these is true:

- Agents run builds, formatters, generators, or anything that writes outside its unit, such as lockfiles, caches, generated output, or a repo-root config.
- An agent needs to run the suite or the harness to check its own edit. Concurrent runs in one tree race on scratch and on git index state.
- Agents commit, stage, or use branch operations. One git index shared by parallel agents corrupts staging.
- A pass may need to be abandoned wholesale, and a clean discard is worth more than a shared diff.

Otherwise the isolation cost is real: N checkouts to create, N results to merge, and merge conflicts reintroduced on exactly the files the manifest was designed to keep apart.

## The shared-asset trap

Some corpora hold byte-identical copies of a file inside several units, deliberately, with a parity test asserting the copies match. A per-unit agent editing "its" copy breaks parity, and the breakage shows up as a test failure in a *different* pass, attributed to the wrong change.

Discover them before dispatch. Shape (POSIX shell; hash every candidate file, key by basename, report pairs appearing in more than one path):

```
find . -type f \( -name '*.md' -o -name '*.py' -o -name '*.sh' \) -exec shasum {} + \
  | awk '{ n = $2; sub(/.*\//, "", n); print $1, n }' \
  | sort | uniq -c | awk '$1 > 1'
```

Read the output two ways. A basename with **one** hash across many paths is a maintained shared asset: assign every copy to exactly one agent, list it as forbidden for everyone else, and have that agent propagate the edit to all copies in the same pass. A basename with **several** hashes across paths is drift that already happened. That is a finding, not necessarily yours to fix in this pass.

Whether a duplication should exist at all is a proposal-time question, decided in `references/corpus-audit.md`. A pass never settles it by factoring out. Assign the copies to one agent, propagate to every copy, leave the mandate alone.

## Author the contract before a parallel rewrite

When one class spans many files and the strings **cross-reference each other** (a phrase in one unit that another unit quotes, a marker other prose routes to, a shared field name), the rewrite cannot be decided in parallel. `references/workflow-shapes.md` carries what breaks if you skip either the contract or the fan-out.

Serialize the decision. One high-effort agent (or you) writes the canonical mapping to a file first:

```
old string (exact)  ->  new string (exact)  |  rationale  |  units affected
```

Per-unit agents then apply it **verbatim** and report any occurrence the contract does not cover rather than improvising a variant. An uncovered occurrence is a contract gap to resolve serially, not an invitation to extend the contract locally.

Corollary: **gathering the inventory is parallelizable; deciding the canonical rewrite is not.** Fan out to find every occurrence, then collapse to one author to decide the mapping, then fan out to apply it.

## Reconcile

After editing, re-read every block touched and fix what the edit itself broke. A half-applied cut is worse than no cut. It leaves prose that is internally inconsistent, which is a defect the corpus did not have before.

Check each of these on every touched file:

- A reference (path, section name, phase number) pointing at something the pass removed.
- A numbered or ordered sequence with a hole, or with a step whose ordinal no longer matches what it depends on.
- An instruction whose precondition was deleted, so it now applies unconditionally or never.
- A reference file nothing routes to any more. Either restore the route or remove the file. An orphan reference is loaded by nobody and rots.
- Two surviving sentences that now contradict each other. Pick one and delete the other. Do not leave both and let the model choose.
- Frontmatter, description, or activation text that no longer matches what the unit does after the cut.
- A cut that landed in a duplicated shared asset without the sibling copies following.

## Assertions on mechanical edits

When a pass applies many exact replacements, do it under assertions rather than by hand. For each target, assert the string matches **exactly once** in its file, and abort before writing anything if any target matches zero times or more than once.

Exactly-once is the part that makes the check worth running. Zero matches means an earlier pass already rewrote the anchor. More than one means the anchor is ambiguous and the edit would land in the wrong place. In the engagement this caught an anchor a previous pass had already changed, and because the check ran before any write, nothing was partially applied. The pass was re-derived against current content instead of repaired afterward.

Fail closed, all-or-nothing per file at minimum. A script that writes files 1 through 7 and dies on 8 leaves a state no one can review.

## Report the skips

**Report what you deliberately did not cut, and why.** An agent that applied 100% of its proposals has almost certainly over-cut. A meaningful skip rate is the expected outcome, not underperformance. In the engagement's audit, 81 of 616 proposed cuts were defended and kept.

## The over-cut failure mode

Removing a "you must" does not remove the decision. It hands the decision to the model. `references/halt-taxonomy.md` class 10 carries the mechanism and the observed failure. The line that resolved it without restoring the old prose was **a unit decides its own internal delegation; whether a step runs at all is not its call.**

The pass-loop rule: for every mandate you remove, name what now decides, and check that the new decider is allowed to decide it. If the answer is "the model, at its discretion, whether a required step happens", then that mandate is a required check, and it stays. This class is also invisible to a probe that never enters the skipped phase. That is why Phase 5 audits the phases the instrument cannot reach, and why one clean run proves nothing (`references/noise-floor.md`).

## Discipline that survives contact

- **Fix at the smallest layer that defines the behavior.** Reword only when rewording is the smallest mechanism; prefer deleting the structure that made the wording necessary.
- **Field names, enums, greppable markers and security guards are data.** They stay. What goes is the justification clause around them that teaches the model a separate consumer is waiting.
- **Never edit tests to make a suite green.** A removed string a test pins is a finding to report, not a test to weaken.
- One problem per agent, each editing a disjoint file set so parallel work cannot collide.

## Reading the failure (Phase 5)

A failure that moves to a later phase is progress and names the next target. A failure at the same site means the fix missed. A run that completes the task while skipping the workflow is a different defect than a halt, and only shows up if Phase 1's two metrics stayed separate.

**Audit the phases the instrument cannot reach.** A probe that skips a phase can never fail in it, so a green streak certifies only what it exercised. List the phases your task never enters, read those files, and treat what you find there as equal in weight to what the runs found. Some of the most consequential defects live where no test looks.

**Report the limit.** Name the paths that remain unmeasured and what would be needed to measure them. Do not let a cleared bar imply coverage it does not have.

## Ship (Phase 6)

Commit each pass separately with its own message so the history says which change was made and why, and so release tooling can classify intent. Keep the measurement artifacts.

Then write the finding down where the next person will hit it: the mechanism, the before and after, the measured numbers, and the hypotheses that died. **Record the ones that died.** They are what stops the next attempt from re-running a dead end, and they are the part every write-up omits.

# Halt Taxonomy

## The mechanism

A workflow "phase" is often invoked by loading its instructions into the **same conversation** (a skill load, an include, a prose route to "the review phase") rather than by spawning a separate process. When that happens, the model on both sides of that seam is one model in one turn. In an unattended run there is no second party and no next user message, and a reply that contains no tool call ends the session. So an instruction to return, hand back, or report to a caller has exactly one expressible form at that seam: stop. The prose is not wrong about its protocol. The protocol has no runtime.

**The diagnostic that settles which seams are real.** In the run archive, find one session id whose trace contains *both* phase-invocation calls (skill loads, reads of another unit's `SKILL.md`) and real subagent dispatches. The phase invocations carry the same session id as the orchestrator. That is one conversation and a fictional boundary. The subagent dispatches carry a distinct child id. That is a real boundary that can carry a hand-off. One line of log settles it for the whole corpus. Run this before classifying anything. Class 9 and survival screen 3 both depend on the answer.

If the archive carries no session or child ids (`references/baseline-mining.md` extracts them where it can), substitute the weaker structural test: a seam is real only where the trace shows a **dispatch tool call whose result returns into the parent trace**. A skill load, an include, or a file read is not one. Do not treat an unlabeled trace as evidence of a real boundary. That reading keeps every halt.

Each class below gives what to grep, why the match halts, and the replacement. Grep terms are starting points for a corpus that never used exactly these words. Read the matches; do not batch-edit them. Case-insensitive, whole corpus including reference files, one class at a time:

```
grep -rniE 'return control|hand back|hand off to|the caller (owns|applies)' <corpus-dir>
```

## 1. Waits on helpers the runtime already terminated

**Grep:** `wait for all`, `in parallel`, `concurrently`, `background`, `up to <n> minutes`, `poll until`, `dispatch all <n>`

**Why it halts:** a mandated fan-out plus a wait ceiling. If the helpers were never spawned concurrently, or the runtime reaped them at turn end, the wait resolves to nothing. And "wait" has no tool call to express it, so the turn ends inside the ceiling.

**Falsifiable check before touching it:** from the archive, measure the maximum number of helpers dispatched *concurrently* per run (see `references/baseline-mining.md` for the extraction). In the engagement, 331 of 332 runs never exceeded one at a time, which falsified the wall-clock justification the corpus had written for its own fan-out machinery.

**Replacement:** dispatch serially, or state parallelism as an allowance rather than a mandate, and delete the wait ceiling with the wall-clock rationale that justified it.

## 2. Hand-off across a same-model seam

**Grep:** `return control`, `hand back`, `hand off to`, `the caller owns`, `the caller applies`, `report to the orchestrator`, `your caller will`

**Why it halts:** it names a recipient that does not exist on this side of the diagnostic above.

**Replacement:** record the structured result and continue. "Write the result to `<path>`, then continue to `<next step>`." Keep every field name in the result untouched. The fields are data; the hand-off verb is the defect.

## 3. A step ending with no instruction to advance

**Diagnostic (count, not grep):** at each boundary between consecutive steps, classify the last sentence of the step as terminal (`stop`, `done`, `await`, `report`, `wait for confirmation`, `end of step`) or forward (`then`, `next`, `continue to`, `proceed to`). A corpus with more terminal than forward endings has a structural halt, not a wording problem.

**Why it halts:** a step that closes on a terminal word and says nothing about advancing leaves ending the turn as the only complete reading.

**Replacement:** an explicit continue-to-next at **each** transition, placed at the transition and not once in a preamble, plus one statement that a checkpoint means verify and continue rather than pause.

## 4. Completion defined without a successor

**Grep:** `is done once`, `is complete when`, `you are finished when`, `definition of done`, `success criteria`

**Why it halts:** "X is done once ..." with nothing after it reads as the end of the turn, not the end of a step.

**Replacement:** say what follows being done. "X is done once <criteria>; then <next step>."

## 5. A structured output labeled as a return

**Grep:** `Return the following`, `Return:`, `your return value`, `respond with the following object`

**Why it halts:** the verb, and only the verb. A return is a control transfer; there is nowhere to transfer to.

**Replacement:** verb swap. `Return the following:` becomes `Write the following to <path>:`. The field list is untouched. This is the cheapest class and the easiest to over-cut. An agent told to fix "return" language will happily also delete the schema under it.

## 6. An output-format rule doubling as a turn boundary

**Grep:** `output nothing else`, `no other text`, `respond with only`, `a single bare`, `do not add commentary`, `the entire response must be`

**Why it halts:** a rule forbidding anything after the payload forbids the next tool call. It is a strictness constraint that has quietly become a stop instruction.

**Replacement:** write the machine-readable payload to a file at a stated path and let the reply continue. The strictness moves rather than being lost. A file at a fixed path is a *more* deterministic parse target for a programmatic consumer than the end of a transcript, so say that in the replacement text or someone will restore the old rule.

## 7. A missing optional capability treated as fatal

**Grep:** `not installed`, `must be available`, `required tool`, `abort if`, `did not start`, `if <x> is unavailable, stop`

**Why it halts:** a step that cannot execute takes the whole run with it, when the correct outcome is a partial result.

**Replacement:** record the step as skipped with the reason and what it would have covered, report partial, continue. Keep the stop only where the missing capability makes every later step meaningless. That is a real precondition, not this class.

## 8. An unresolvable precondition with no unattended path

**Grep:** `resolve ... first`, `ask the user to`, `credentials`, `log in`, `confirm before`, `requires approval`

**Why it halts:** "resolve credentials first" is satisfiable only by a human, and no human is present.

**Replacement:** report the unmet precondition and continue without the step. Where the step is mutating and unsafe to perform unattended, skip the mutation and record the exact command a human would run. Do not resolve a halt by widening what the run is allowed to change.

## 9. The subagent-prompt trap

**Grep:** the vocabulary of classes 2 and 6, restricted to the corpus's own prompt assets: its persona or agent-prompt directories and prompt templates. Also `report and stop`, `return only a gist`, `the orchestrator owns that decision`, `do not act on this yourself`.

**Why a naive sweep misses it entirely:** while the asset is sent to a real subagent, every one of those lines is correct and required. They become a live session terminator the moment a documented no-subagent fallback routes the asset **inline** into the orchestrator's own conversation. Same words, different consumer. A sweep that reads only the asset sees a correct file. A sweep that reads only the orchestrator sees no offending prose.

**Resolution:** classify each prompt asset by who actually consumes it. Grep every unit that dispatches the asset for an inlining fallback: `if you cannot dispatch`, `if subagents are unavailable`, `run this yourself`, `inline`. Then:

| Consumer | Action |
|---|---|
| Always a real subagent | Leave as is. The boundary is real. |
| Any documented inline fallback | Rewrite so the text is safe in both paths, or remove the fallback. |
| Asset shared by several units | Resolve once, for the union of consumers (see the shared-asset rule in `references/cut-passes.md`). |

Do not blanket-rewrite prompt assets. That deletes the one boundary in the corpus that was not fiction.

## 10. Over-correction: removing the mandate that made phases required

This class belongs here because the cure for 1 through 9 causes it.

A dispatch mandate carries two payloads: the fictional seam, and the requirement that the phase run at all. Remove the first without preserving the second and nothing in the corpus says the workflow's own phases are required.

**Observed:** after a dispatch-mandate removal, a run did the task inline and reported success with no artifacts. The transcript looked clean. That defect is invisible to a did-it-finish metric and shows up only if process adherence and task completion are tracked as separate numbers (`references/baseline-mining.md`).

**Replacement:** keep the requirement, drop the seam. "Phase N is required. Do it in this session; do not skip it." Then verify the phase by artifact existence, never by the final message.

## Screens for a stop that must survive

Treating every stop as the enemy is the failure mode of applying this file. Run these before editing any match above. **Any one** is sufficient to keep the stop as written.

1. **The user is genuinely the subject.** The instruction concerns a human's decision, consent, or preference, not a fabricated caller.
2. **It is a terminal no-op.** Nothing is left for anyone: "no findings, stop." A continuation here invents work.
3. **The roles split across a real runtime boundary.** The opening diagnostic shows two distinct sessions or processes. Then "return to the caller" is accurate and the halt is somebody else's turn ending, correctly.
4. **The site already has an unattended default.** "If no reply within X, choose Y", "in non-interactive mode, do Z." The path exists; the stop is the interactive branch.
5. **The word bounds an activity, not the turn.** "Stop expanding the search once you have three candidates", "halt the retry loop after two failures." A grep for `stop` catches these. They are scope quantifiers, and removing them removes a threshold.

**Sixth case, stated separately because it is the one that gets deleted anyway:** a workflow whose product **is** stopping to ask. An interview that must take one question at a time. A teaching check-in that pauses for the learner's answer. An approval step that exists in order to make the user decide. Deleting the pause deletes the unit. These get an unattended degradation path (a stated default, a skip-with-reason, or an explicit refusal to run unattended), never a deletion. If such a unit fails the benchmark, the benchmark's task is wrong for it. Exclude the unit and report the exclusion rather than editing the unit to pass.

## Why the fix is never more capability

Every class above reduces to prose written as if a second party were waiting. The fix is never to add capability. It is to remove the fictional seam and keep the requirement.

# Noise Floor and the Registered Bar

Phase 2 protocol. Measure what changing nothing produces, then write down the bar before a change exists.

## What the A/A run buys

Two builds of the corpus at the same commit, run under one harness on one task, produce a distribution rather than a result. That distribution is the floor. Any later claim smaller than it is unsupported no matter how confidently it was reported. In the engagement this was the single most valuable measurement of the session. 12 runs across two identical builds gave workflow adherence 7 of 12 and output tokens from 21,872 to 155,682, a 7.12x spread on identical code. It retired every small-sample claim in flight, including an outside analyst's "2 of 8 improved to 5 of 8", which sits entirely inside the range that doing nothing produces.

The A/A also tests the instrument. Identical builds that differ significantly are not evidence about the corpus. They are a harness, provenance, or scoring bug. Chase that before continuing.

## Setup

Required capability: a harness that can point a run at a specific source checkout of the corpus (Phase 0's build selector) and writes a per-run artifact you can parse. Both arms must go through the *same* runner, task, and model configuration.

1. Materialize two checkouts of the corpus at the same commit. Record the commit for each arm.
2. Hash both trees and assert equality before the first run (`find <dir> -type f | sort` then a checksum over the file list and contents). An accidental difference between arms gets read as noise and poisons the floor silently.
3. Label the arms concretely by path, not by intent (`build-a`, `build-b`). Nothing downstream should be able to guess an arm from a filename that also encodes a hypothesis.
4. **Prove the selector is honored, in one run, before planning any.** Point a single run at `build-a`, then open the finished artifact and confirm it names `build-a` in the durable field below. Two failures both look like a normal run: a harness that silently falls back to its installed copy of the corpus, and one that records the arm nowhere. Either makes all 12 runs unlabeled and unusable, and both are invisible until you try to score. If you want a positive control, put a harmless unique string in a **third**, throwaway checkout and confirm it reaches that run's trace. Never put it in either arm, which step 2 requires to stay byte-identical, and re-assert the hashes before the counted runs begin.
5. Plan 12 or more runs total, split evenly. Below about 10 the floor estimate is itself noise; the spread ratio in particular needs the tails.

## Interleave, never batch

Alternate arms run by run (A, B, A, B) and alternate which arm goes first across pairs (A/B, then B/A). Never run all of arm A and then all of arm B.

Batching confounds the comparison with API load, rate-limit backoff, transient service degradation, and time of day. Those effects are large and they are indistinguishable after the fact from a real difference between builds. There is no post-hoc correction, because the confound and the effect are the same column. Interleaving costs nothing. It is the same number of runs, reordered, and it makes each pair a matched observation you can analyze paired if you want the extra power.

If a run dies for an infrastructure reason, re-run *that arm's* slot rather than dropping the pair, and mark the row so the archive's broken-run taxonomy (`references/baseline-mining.md`) still classifies it correctly.

## Provenance per row

Every scored row must carry which build produced it, **read from the artifact the harness wrote**, never inferred from run order, timestamp, or the order you launched things.

The engagement's first scorer read the field the harness exposes for the source-checkout override *while a run is in flight*. On completion the harness folds that value into the run's metadata and clears the live field, so every finished row had an empty build column and the arms were unverifiable from the scored data. The runs were fine. The measurement was worthless until re-scored.

The rule that generalizes: **read the durable post-completion field first, fall back to the transient in-flight one.**

```
build = run.metadata.<override-field> or run.<live-override-field> or None
```

Then check it before scoring. Assert every row has a non-empty build value before computing anything, and fail loud with the offending run ids rather than scoring a partially-labeled table. A row whose arm you cannot establish is not a data point.

Minimum row schema:

| field | why it is here |
|---|---|
| `run_id` | join key back to the raw trace |
| `build` | arm, from the durable field |
| `commit` | proves the arms were the same source |
| `pair_index`, `position_in_pair` | recovers the interleave for paired analysis |
| `adherence` | followed the workflow (separate from outcome) |
| `outcome` | did the job |
| `broken` | empty transcript / error exit / infra death |
| `output_tokens` | the variance channel, and usually the noisier one |
| `terminal_marker` | how the run ended, verbatim |

Adherence and outcome stay separate columns here for the same reason they do in Phase 1.

## Statistics that survive small n

Required capability: a scripting environment with an exact-test library, or the closed forms. Wilson and the risk difference are arithmetic on the counts; Fisher's exact is a sum of hypergeometric terms. If the required analysis is unavailable, do not substitute a normal approximation. The streak below can provide descriptive confirmation; its statistical interpretation requires the assumptions stated there.

At n = 6 per arm the normal approximation to a proportion is wrong in ways that matter. It produces intervals that exclude values the data plainly permit, including 0 and 1. Use:

- **Wilson score intervals** for each arm's rate. Not Wald, not "plus or minus 1.96 times the standard error".
- **Risk difference with a confidence interval** as the headline comparison. Report the interval, not the point estimate. An interval that spans zero *is* the finding.
- **Fisher's exact test** for the 2x2, not chi-square, at these counts.
- **A spread ratio** for continuous channels: max over min of output tokens, per arm and pooled. In this corpus the token spread moved earlier and further than the completion rate, and it is what makes a corpus unmeasurable.

Report both channels. A build whose rate is unchanged but whose spread halved is a real result. The reverse, a rate that improved while the spread widened, is a warning that you got lucky.

### Make "inconclusive" a printed outcome

The summarizer must, whenever a result is not significant, print the runs-per-arm that *would* have been needed for 80% power at the observed effect. Otherwise "no significant difference" reads as an empty slot in the report and someone fills it with narrative.

Two-proportion sample size at 80% power, alpha 0.05, two-sided:

```
n_per_arm = 7.849 * (p1*(1-p1) + p2*(1-p2)) / (p2 - p1)^2
```

Against the engagement's 58% baseline, detecting +30 points gives about 31 per arm from that formula, and the power tool used reported about 39 once it applied a continuity correction. **Take the larger.** Corrections differ by method and land roughly 20-40% above the plain figure at these rates, so with no tool to hand, plan on the plain figure inflated by a third rather than treating 31 as the budget. Print the number next to the non-result, in the same line, so the cost of the claim is visible where the claim is.

## One-armed confirmation

A pre-registered streak can serve as an operational confirmation bar on one frozen build under fixed measurement conditions. Under a null with a **known, fixed success probability `p` and independent runs**, one pre-specified attempt at N consecutive clean runs has all-success probability `p^N`. That calculation alone is not a significance level for an adaptive search.

Establish the baseline before the change, from runs other than the ones being tested. Estimating `p` from the streak's own runs makes the calculation circular and meaningless. A baseline measured independently from A/A and archive runs is still an estimate. Account for its sample size and uncertainty in any statistical claim. When baseline uncertainty, changed conditions, or dependence between runs is unaccounted for, report the observed confirmation descriptively rather than as an exact test.

For one pre-specified attempt under the fixed `p = 0.58` null:

| consecutive clean runs | probability of all successes | odds |
|---|---|---|
| 3 | 0.195 | 1 in 5 |
| 5 | 0.066 | 1 in 15 |
| 8 | 0.0128 | 1 in 78 |

**Keep the search history when interpreting the result.** Across 20 independent attempts under that same null, the probability of at least one eight-run success is `1 - (1 - 0.58^8)^20`, about 0.2272, not 0.0128. This is an illustrative calculation, not a measured CE error rate; adaptive candidates need not be independent. A search-wide significance claim must account for candidate selection and the stopping rule. Changing a candidate resets its streak, not the search history.

**Stop an attempt at the first behavioral failure.** A failed streak is a finding to diagnose before changing the candidate, not permission to repeat it until it passes. Keep every attempt and classify infrastructure retries under the registered broken-run rule. Never pool successes across builds. A cleared operational bar without a supported statistical analysis remains descriptive confirmation. A streak probability is not an effect-size estimate or a percentage improvement.

**Separate diagnostic runs from final confirmation.** One or two diagnostic runs after a cut can locate the next failure; they do not count toward a streak. Begin final confirmation only after a clean diagnostic run, with the candidate and confirmation rule fixed. Use fresh runs whose results were not used to select it. If those results guide another edit or another confirmation attempt, they enter the selection history and cannot be ignored in a later significance claim.

## Registering the bar

The registration is a written artifact at a path you can point a skeptic at, not an intention (`references/workflow-shapes.md` for where the engagement's artifacts live). It must name:

- the metric, in the exact form the summarizer computes it;
- the effect size worth detecting, and why that size and not a smaller one;
- the design (two-armed with n per arm, or streak with N), and whether the result is descriptive confirmation or a statistical claim;
- for a statistical claim, the baseline evidence and uncertainty, independence assumptions, and error control covering candidate selection and repeated attempts;
- the stopping rule, including what counts as a broken run that does not consume the streak;
- the phases the probe task traverses, from the validation below.

**When the affordable n cannot detect the effect you care about, say so and change the design, not the interpretation.** Consider a cheaper metric that moves more per run (token spread usually moves before completion rate) or a bigger effect. A streak can instead support the limited confirmation described above; it is not a substitute for the two-armed design's power or effect-size estimate. Running an underpowered two-armed test and reporting its point estimate is the failure mode this whole phase exists to prevent, and it is exactly what the outside analyst's 2-of-8 versus 5-of-8 was.

## Sizing the probe task

A full realistic task costs too much to repeat 12 to 40 times. Build a probe whose **work** is small but whose **path** is complete. It must cross every phase boundary of the corpus, produce each phase's artifact, and require each handoff, with the substance of each step shrunk to near-nothing.

Then validate that it does, before spending n on it:

1. Run it once with tracing on.
2. For each phase, check the trace for its boundary marker: the dispatch, the artifact write, the check that decides the phase. Score against the phase-marker map Phase 1 already built (`references/baseline-mining.md`) rather than re-deriving markers here. A probe scored against a second, differently-derived map is not comparable to the archive baseline.
3. List the phases the probe never entered.

**A probe that structurally cannot enter a phase can never fail in it.** Its green result certifies only the phases it traversed, and the unentered ones stay unmeasured while the streak makes the corpus look verified. Two of the engagement's landed cut passes came out of auditing exactly those unreachable phases; none of them could have come from the runs. That is why Phase 5 requires reading the phases the instrument cannot reach, and why the unentered list belongs in the registration rather than in a footnote afterward.

## Reading a tie honestly

If both builds succeed on a capable model, that shows **no regression**, not improvement. A ceiling result on the strong axis is compatible with the change having done nothing, and it says nothing at all about a more literal model or a different harness. Report it as "no regression detected, effect indistinguishable from zero at n = X" and keep the improvement claim unmade.

# Workflow Shapes

Which orchestration shape fits each phase, and what breaks when you pick the wrong one.

Two primitives are assumed, both platform-neutral:

- **A dispatch primitive** that launches an independent agent with its own context window and returns a result to the orchestrator (in one host it is a subagent-spawning tool; in another a job runner). Only two properties matter: fresh context per agent, and a result the orchestrator can read.
- **A concurrency cap** the host enforces on how many agents run at once (often around 10). Dispatches above the cap queue rather than fail, so a 31-unit fan-out is three waves, not one. Plan the wave count; do not assume flat cost.

Classify a rejected dispatch by whether an agent launched. Correct a pre-launch argument rejection once, and leave capacity-limited work queued. Any other launch failure follows the phase's own failure direction. In the corpus audit, the proposal and defense contexts must be independent for the audit to mean anything, so a launch failure there is a blocker: stop the audit rather than running the missing side inline.

## The shapes

| Phase | Shape | Why | Failure when mismatched |
|---|---|---|---|
| Archive mining | Serial, local, zero model calls | Classification is a deterministic function of fields present in each log; you will re-run it as the taxonomy changes | Agent-per-log gives a different taxonomy each pass, so the exclusion set cannot be re-derived or audited |
| A/A and A/B runs | Serial within an arm, interleaved across arms, nothing else running | Contention is a confound; interleaving spreads service drift evenly across arms | Concurrent runs inflate duration and can manufacture a timeout that reads as a halt |
| Corpus audit | Fan out by unit, pipelined into per-unit adversarial defense | Units are independent and each needs a full directory read; defense for a unit needs only that unit's findings | A barrier between audit and defense costs the slowest unit and buys nothing |
| Cross-cutting rewrite | One agent authors the contract, then fan out application | The replacement text must be identical everywhere to stay greppable and revertable | No contract: N phrasings of one rewrite. No fan-out: context exhaustion, silent skips |
| Cut passes | Fan out by disjoint file sets, one agent edits every copy of a shared byte-identical asset, barrier before verification | Concurrent edits to one file lose each other; verification must see one tree | Overlapping file sets produce a tree nobody wrote and nobody can bisect |
| Verification | Fan out by independent check, never by file | The checks need different tools and different judgment, and one must be a read no grep can do | Splitting by file runs the same suite N times and skips the coherence read entirely |
| Residual critic | One agent, last, scoped to what did not land | A fresh reader with no stake in the passes is the only one who will call a pass partial | A critic asked to verify the work reports green, which is worth nothing |

## Per phase

**Archive mining is serial and local.** The temptation is to dispatch one agent per log file because there are hundreds of them. Resist it. A script is faster, deterministic, and re-runnable, and re-runnability is the point. The correction that decides whether a baseline is usable at all, excluding empty transcripts and error exits, changes the taxonomy after you have already classified everything, so you will run the classifier more than once. A script re-runs for free and produces the same answer. An agent fleet re-costs the whole archive and produces a slightly different answer. See `references/baseline-mining.md` for the fields and the taxonomy.

**A/A and A/B runs are serial per arm and interleaved across arms.** Never run measurement concurrently with anything else you intend to measure. Metrics differ in how they survive contention, and the distinction decides whether a contended run is salvageable:

- **Robust to contention:** did it complete, did it follow the workflow, which phase it reached, token counts. A run either entered a phase or it did not, regardless of what else was executing.
- **Destroyed by contention:** wall-clock duration and everything derived from it, including anything that depends on a timeout. A contended run that times out mid-phase is indistinguishable from a corpus-caused halt, which is the exact failure this skill is trying to attribute.

So a contended run can still support a completion or adherence claim. It cannot support a latency claim, and it cannot be counted as a halt. Protocol and sample-size math are in `references/noise-floor.md`.

**The corpus audit fans out by unit and pipelines into defense.** One agent per skill directory, each reading that directory in full. That full read is what makes per-unit parallelism worth its cost, and it is why one agent cannot do all the units. Pipeline the adversarial defense stage per unit. A unit's defender needs that unit's findings plus the project's own documented learnings, tests, and history, never the global finding list, so a defense can start the moment its audit returns. A global barrier here only makes every defender wait on the slowest auditor.

Budget the defense stage as first-class work, not a review formality. It is the stage that removes cuts, and the removals are the phase's product as much as the proposals are. If you compress the phase for time, compress the number of units in flight, not the defense. Dispatch shape and finding schema live in `references/corpus-audit.md`.

**A cross-cutting rewrite writes the contract first, then fans out.** When one pattern must change in many units, have a single agent author the canonical contract: the exact text to remove, the exact text to replace it with, and the boundary cases where it does not apply. Then fan out application against that contract.

- **Skip the contract** and N agents produce N phrasings of one rewrite. The change stops being greppable, so you cannot confirm coverage. It stops being revertable as a unit. And when the next measurement fails you cannot tell whether the cause is the change or one agent's variant of it.
- **Skip the fan-out** and one agent reads 30 directories until its usable context is gone. The failure is not an error. It is degradation: the late units get shallow work or get silently skipped, and the output looks complete.

**Cut passes use disjoint file sets, one agent for every copy of a shared asset, then a barrier.** One problem per pass, one agent per pass, and file sets that do not intersect. Byte-identical duplicated assets are the trap. If two passes each touch their own copy of the same file, you get divergence that a parity test catches at best and ships at worst. Assign every copy of a duplicated asset to exactly one agent, even when that agent's problem is not the one the copy most obviously belongs to. Close the fan-out with a barrier before verification. Verification reads a tree, and a tree with a pass still landing in it is not the tree you will ship. Loop mechanics are in `references/cut-passes.md`.

**Verification fans out by independent check.** The axes are different tools and different judgment, so they parallelize cleanly while splitting by file does not:

| Check | What it can see | What it cannot |
|---|---|---|
| Mechanical suite | Pinned strings, schemas, parity between duplicated assets | Whether the surviving prose still makes sense |
| Packaging and manifest checks | Install-time validity, catalog and inventory drift | Runtime behavior |
| Residual-pattern sweep | Remaining instances of the class you just removed, including in files no pass edited | Instances phrased differently from the pattern |
| Coherence read | Dangling cross-references, orphaned "as described above", a phase whose predecessor was deleted | Anything requiring the full suite |

Run the mechanical suite once, as one check, not once per agent. Keep the coherence read in the fan-out and give it a human-style instruction: read the changed files end to end and name anything that no longer follows. It is the only check that finds a reference left pointing at removed text, and no grep will do it.

**The residual critic is one agent, run last.** Give it the pass list, the registered bar, and the measured results, and ask for one line per intended change: landed, partial, or not landed, plus what a next pass would have to do. Instruct it explicitly that declaring success is not its job and that returning an empty list requires naming what it checked to reach that conclusion. Without that framing it will summarize the work approvingly, which tells you nothing you did not already believe.

## Cross-cutting rules

**Fan out by disjoint file ownership, never by item.** Disjoint file ownership means each agent edits its own set of files and no file appears in two agents' sets. One item routinely touches several files and one file routinely hosts several items, so an item-partitioned fan-out silently produces two agents writing the same file. Partition the file set first, then assign each item to the agent that edits its files. An item spanning two agents' file sets is either one agent's job with both files, or it is a contract-then-fan-out (above), not a split.

**A barrier is only justified when the next stage needs cross-item context.** Legitimate: deduplicating findings across all units, an early exit that depends on a total count, a synthesis that compares items against each other. Not legitimate: "I want the list flattened before I continue." That is orchestrator convenience, and it costs you the slowest agent in every wave.

**Pass large context by path plus a short gist.** Give an agent the file paths it must read and two or three sentences of why, not the inlined contents. Inlining spends the orchestrator's context to fill an agent's, which is backwards. The fresh context window is the reason you dispatched.

**Every artifact of this method needs a path decided before the phase that writes it.** The registration and the scored run table (`references/noise-floor.md`), the phase-marker map and the extractor (`references/baseline-mining.md`), the ruled finding set (`references/corpus-audit.md`), the file-assignment manifest and the rewrite contract (`references/cut-passes.md`). Pick one directory for the whole engagement and keep them together. The registration must be re-readable by a skeptic after results exist. The extractor and map must be re-runnable when the taxonomy changes. A contract or manifest passed only through a dispatch prompt cannot be checked afterward against what the agents actually did. The durable ones (registration, final scored table, write-up) belong in the repo's own docs location; per-run scratch does not.

**Give a schema to anything you will aggregate.** If the orchestrator will count, sort, dedup, or compare results, specify the fields and the enum values in the dispatch prompt. Parsing prose across dozens of returns is where item counts quietly stop matching.

**Budget the serial segments.** A one-agent contract phase blocks its entire fan-out, and a barrier before verification blocks on the slowest pass. When reporting elapsed time, name the serial segments separately instead of letting the total read as the shape's natural speed. Otherwise the next person concludes the fan-out was slow and removes the contract.

**State what a fan-out did not cover.** Units that queued past the cap and never ran, units an agent skipped, files outside every agent's set: list them. Silent truncation is indistinguishable from full coverage in the result, and a coverage claim you did not verify is the same defect as a bar chosen after the fact.

---
name: ce-retune
description: "Retune a skill corpus for a new model, measurement-first: mine the run archive for a baseline, establish a noise floor, audit the corpus adversarially, then cut in measured passes until a pre-registered bar clears. Requires a benchmark harness that can A/B two builds of the corpus; refuses without one."
disable-model-invocation: true
argument-hint: "[target model or symptom] [path to the corpus, defaults to ./skills] [bar:<n> consecutive clean runs]"
---

# Retune a Corpus for a New Model

A corpus that degrades on a new model is a measurement problem before it is a writing problem: rewriting what looks wrong produces a plausible fix list and no way to know whether any item mattered.

**Outcome:** a corpus whose measured behavior on the target model clears a bar registered before any change, with the regression classes removed and each removal attributable.

**Done:** the bar is cleared, or the run reports the specific claim it could not support. A green test suite is not done: it proves nothing broke, not that behavior improved.

**Non-goal:** word reduction. Leanness and performance are separate programs that share a corpus; only one of them is the result here. Report completion, not word count.


## Phase 0: the measurement gate — check this first

This skill cannot run without a way to observe behavior. Check for all three, and name whichever is missing:

1. **A run archive or a harness that produces one** — per-run logs carrying the tool-call trace, a terminal marker, token counts, and the final message.
2. **A build selector** — the harness can point a run at a specific source checkout of the corpus (a `--plugin-dir`-style override, a configurable skills path, an env var), so two builds are comparable under one runner.
3. **A repeatable task** the corpus actually executes end to end.

If any is missing, **stop and say so**, naming what to build. Do not fall back to a static audit and present it as retuning: an audit can say what looks cuttable and never whether cutting helped. An audit-only pass is a legitimate thing to want; it is a different request.

State the target model and the harness you found before continuing.

## The phases

They run in order, and each names the reference it cannot start without. Read `references/workflow-shapes.md` before dispatching any phase: the wrong orchestration shape is the common failure. Fan out by disjoint file ownership, never by item. Items cross files, and agents that share a file lose each other's edits.

Before assessing whether the registered bar is met or interpreting its results, read `references/noise-floor.md`.

1. **Mine the archive** before spending a run — `references/baseline-mining.md`. Historical runs are a free baseline, usually larger than any experiment affordable now.
2. **Establish the noise floor** — `references/noise-floor.md`. Run the harness against **two identical copies** of the corpus, same commit on both sides; whatever difference appears is the floor every later claim must clear. **Register the bar now, in writing, before any change exists.** A bar chosen after seeing results is not a bar.
3. **Audit the corpus adversarially** — `references/corpus-audit.md`. One agent per skill proposes cuts; a second per skill does the opposite and defends the existing prose. **The two passes require independent contexts.** If the host exposes no way to run them as separate agents, report that as a blocker and stop the audit — do not argue both sides in one context and present the result as an audit.
4. **Cut in surgical passes**, one problem per agent — `references/cut-passes.md`, and `references/halt-taxonomy.md` when the symptom is stalling, halting, or a run that ends while naming work it did not do. Two rules bound every pass, whatever class it is cutting. **Never edit a test to make a suite green**: a removed string a test pins is a finding to report, not a test to weaken. And **not every stop is the enemy.** Some workflows exist to stop and ask; that is the product. Sort every stop by who is actually on the other side before touching it. `references/halt-taxonomy.md` carries the screens that decide, so read them before cutting any stop.
5. **Measure, then let the failure choose the next fix** — `references/cut-passes.md` again for what each failure site means and for auditing the phases the instrument never enters. Loop 4 and 5 until the registered bar clears. Then stop; a bar cleared is done. Also report what stayed unmeasured: a cleared bar never implies coverage it does not have.
6. **Ship** — `references/cut-passes.md` carries what the commits and the write-up must preserve.