references/grader-templates.md
Last verified: 2026-07-14
# Grader Templates
Runnable examples for the four grader shapes named
in `SKILL.md`'s Graders section: schema-compliance,
exact-match with normalization, execution-based, and
LLM-judge. Every grader returns a binary pass/fail —
never a Likert score — per the plugin-wide rule.
Wire each one to exactly one failure bucket from
error analysis; don't blend buckets into one grader.
## Schema Compliance
For buckets where the failure mode is "the output
isn't shaped right" — tool calls, structured
extraction, JSON responses:
```python
import json
from jsonschema import validate, ValidationError
RESPONSE_SCHEMA = {
"type": "object",
"required": ["action", "arguments"],
"properties": {
"action": {"type": "string"},
"arguments": {"type": "object"},
},
"additionalProperties": False,
}
def grade_schema_compliance(completion: str) -> bool:
"""Binary pass/fail: does the completion parse as
JSON and match RESPONSE_SCHEMA? Malformed JSON is
an automatic fail, not an exception to handle
upstream — the grader owns that decision.
"""
try:
payload = json.loads(completion)
except json.JSONDecodeError:
return False
try:
validate(instance=payload, schema=RESPONSE_SCHEMA)
except ValidationError:
return False
return True
```
## Exact Match with Normalization
For buckets with a single ground-truth string (math
final answers, extracted entities, classification
labels) where naive `==` fails on formatting noise:
```python
import re
def normalize(text: str) -> str:
"""Lowercase, collapse whitespace, strip
punctuation and surrounding markup — apply the
identical normalization to both prediction and
ground truth so neither side gets an unfair pass.
"""
text = text.strip().lower()
text = re.sub(r"[^\w\s.]", "", text)
text = re.sub(r"\s+", " ", text)
return text
def grade_exact_match(completion: str, ground_truth: str) -> bool:
return normalize(completion) == normalize(ground_truth)
```
## Execution-Based
For buckets where correctness means "the code runs
and does the right thing" — the strongest signal
available when applicable, since it needs no
normalization or judgment call.
**WARNING — this grader executes model-generated
code and REQUIRES an isolated environment:** a
network-disabled container, gVisor/firejail, or a
dedicated CI sandbox, with **no secrets or
credentials in the environment** — no HF tokens,
experiment-tracker keys, cloud credentials, or SSH
keys. Never run it directly on a host holding
credentials. The timeout below protects grading-loop
liveness only — **it is NOT a security boundary**;
isolation comes entirely from `sandbox_cmd`.
```python
import logging
import subprocess
import tempfile
from pathlib import Path
logger = logging.getLogger(__name__)
def grade_execution(completion: str, test_code: str, sandbox_cmd: list[str]) -> bool:
"""Write the completion plus a pytest test file to
a scratch dir, run pytest via `sandbox_cmd`, and
pass only on a clean exit code. Timeouts and
non-zero exits are both failures — never treat a
hung process as a pass by default.
SECURITY: executes model-generated code. This
function REQUIRES an isolation boundary — it does
not run anything on the host by itself.
`sandbox_cmd` (list[str], required) is a command
prefix that wraps pytest in that boundary, e.g. a
network-disabled, resource-capped Docker container:
# sandbox_cmd = [
# "docker", "run", "--rm", "--network=none",
# "--memory=1g", "--cpus=1",
# "-v", f"{workdir}:/work:ro", "-w", "/work",
# "python:3.12-slim",
# ]
If `sandbox_cmd` is falsy, this function refuses to
execute anything and returns False — it never falls
back to running pytest on the host. The subprocess
environment is scrubbed to a minimal PATH.
"""
if not sandbox_cmd:
logger.warning(
"grade_execution: no sandbox boundary provided "
"— refusing to execute model-generated code"
)
return False
scrubbed_env = {"PATH": "/usr/bin:/bin"}
with tempfile.TemporaryDirectory() as tmp:
solution = Path(tmp) / "solution.py"
test_file = Path(tmp) / "test_solution.py"
solution.write_text(completion)
test_file.write_text(test_code)
try:
result = subprocess.run(
[*sandbox_cmd, "python", "-m", "pytest",
str(test_file), "-q"],
cwd=tmp,
capture_output=True,
timeout=30,
env=scrubbed_env,
)
return result.returncode == 0
except subprocess.TimeoutExpired:
return False
```
## LLM-Judge Template
Only for buckets that failed the deterministic-first
check in `SKILL.md` — genuinely subjective criteria.
Few-shot slots and a binary output contract are both
mandatory; a judge without few-shot anchors drifts
toward its own prior instead of the calibrated
labels.
```python
JUDGE_PROMPT = """You are grading whether a response
meets the following criterion: {criterion}
Examples of PASS:
{few_shot_pass_examples}
Examples of FAIL:
{few_shot_fail_examples}
Now grade this response. Output exactly one word,
PASS or FAIL, with no other text.
Task: {task}
Response: {response}
Verdict:"""
class JudgeParseError(Exception):
"""Raised when the judge returns anything other than
exactly PASS or FAIL — a transport or format failure,
not a grading verdict. Never caught and coerced to
False; the caller retries the judge call or routes the
item to human review."""
def grade_llm_judge(task: str, response: str, criterion: str,
few_shot_pass: str, few_shot_fail: str,
judge_client) -> bool:
prompt = JUDGE_PROMPT.format(
criterion=criterion,
few_shot_pass_examples=few_shot_pass,
few_shot_fail_examples=few_shot_fail,
task=task,
response=response,
)
# judge_client is pinned to a fixed snapshot per
# SKILL.md's Judge Calibration section — never an
# unpinned "latest" alias.
verdict = judge_client.complete(prompt, temperature=0).strip().upper()
if verdict not in ("PASS", "FAIL"):
raise JudgeParseError(f"unparseable judge verdict: {verdict!r}")
return verdict == "PASS"
```
The grader's return value stays binary pass/fail per this
file's plugin-wide rule — `JudgeParseError` is not a third
grading state, it's an operational failure. Never catch it
and coerce to `False`; log it and re-run the judge call or
route the item to human review instead.
## drift-suite.yaml Example
Frozen general-capability benchmarks plus a
domain-adjacent slice, per `SKILL.md`'s Directory
Contract. Benchmark subsets are frozen at a fixed
seed/split so re-runs are comparable run over run:
```yaml
# eval/drift-suite.yaml
frozen_benchmarks:
- name: mmlu-subset
source: mmlu
split: test
n_items: 500
seed: 42
- name: gsm8k-subset
source: gsm8k
split: test
n_items: 250
seed: 42
- name: ifeval
source: ifeval
split: test
n_items: 300
seed: 42
domain_adjacent:
path: eval/goldens.jsonl
filter: "tag == 'drift-suite'"
n_items_range: [200, 500]
drift_budget:
noise_tolerance_pts: 1
rerun_seed_variation_pts: [2, 5]
hard_fail_threshold_pts: 5 # >5pt drop is a hard fail regardless of task-metric gains
```
**Minimum viable drift suite for a small/dogfood run:** the
500/250/300 general-benchmark sizes and the 200-500
`domain_adjacent` range above are scoped for production-scale
runs and can be hours of wall-clock at greedy single-stream
decode on modest hardware. Scope down deliberately rather than
silently truncating — pick n per benchmark using
`checkpoint-promotion`'s item-count-from-budget rule (95% CI
half-width comfortably under half the hard-fail threshold), and
document the scoped-down suite inline in `drift-suite.yaml`
(a comment naming the budget it was sized against). The
`domain_adjacent` block itself is **optional when the golden
set is small and already fully consumed by the task harness** —
if `eval/goldens.jsonl` has no separate drift-suite-tagged
holdout because every golden is already spoken for by the task
grader, omit the block rather than fabricating a slice with
nothing behind it; the frozen general benchmarks alone still
provide a capability-drift signal.
## Drift-Suite MMLU-Style Scoring: Logprob Over Generate-and-Extract
"Letter-choice logprob or generate-and-extract" are not
equivalent scoring methods, though earlier guidance in this
plugin implied they were interchangeable:
- **Generate-and-extract** (sample a completion, extract the
first A–D letter from a tight token budget) is **parse-brittle**
for models that preamble before answering. Observed on a real
drift run: a near-base checkpoint hit 9/100 unparsed-scored-
as-wrong items (all prose restarts like "The energy levels
of...") on a `max_new_tokens=8` budget, versus 0–1 unparsed
for other checkpoints in the same series — this conflates
answer-format compliance with the knowledge MMLU is supposed
to measure, and can hard-fail a checkpoint on formatting alone.
- **Logprob scoring** (compare the model's log-probability on
each of the four answer-letter tokens directly, no generation
or parsing involved) is robust to this failure mode entirely —
there is nothing to parse, so a verbose or preambling model
scores on its actual answer distribution.
**Prefer logprob scoring whenever the harness has logit access**
(local inference, not a hosted API that only returns text). When
logit access isn't available and generate-and-extract is the
only option, use a generous token budget and retry with a longer
budget on any unparsed output before scoring it as wrong — don't
let a tight budget silently convert "the model reasoned before
answering" into "the model failed the benchmark."
references/judge-calibration.md
Last verified: 2026-07-14
# Judge Calibration Protocol
The full procedure behind `SKILL.md`'s "Judge
Calibration Is a Prerequisite" section. Any grader
routed to an LLM-judge follows this before its
verdicts count toward a pass rate or a checkpoint
promotion decision.
## N/A Path: All-Deterministic Harness
If error analysis produced zero buckets that route to
an LLM-judge — every grader is regex, schema, or
execution-based — this entire protocol is N/A for the
run, not an unsatisfiable checklist item. Phase 0
Exit Checklist item 4 in `SKILL.md` is satisfied by
stating this explicitly (e.g. "Judge calibration:
N/A — all N graded criteria are deterministic") rather
than leaving it blank or blocking Phase 0 completion
on a judge that was never going to exist. This is
common on a greenfield strict-schema or exact-match
task with synthetic goldens — don't invent a
subjective criterion just to have something to
calibrate.
## 1. Label
Collect human labels for **≥100 items** covering the
failure bucket the judge will grade. Use the same
labelers (or a labeling rubric tight enough to be
interchangeable) that produced the axial-coding
buckets in `SKILL.md`'s Building Goldens section —
a judge calibrated against a different notion of
"pass" than the one used to build goldens will
silently diverge from what the harness is supposed
to measure.
## 2. Split
Divide the labeled set three ways and keep the
splits separate for the whole calibration cycle:
| Split | Purpose | Size |
|---|---|---|
| train | Few-shot examples embedded in the judge prompt | ~20-30% |
| dev | Iterate the prompt, catch obvious misses | ~30-40% |
| sealed test | Report TPR/TNR once; never re-touch after | ~30-40% |
The sealed-test split is sealed: if a dev-split
iteration cycle causes the reported test-split
number to move, that number is no longer a valid
generalization estimate — re-seal a fresh test split
instead of re-running against the same one.
## 3. Compute TPR/TNR
Run the judge (with train-split few-shot examples in
the prompt) against the sealed test split and compute:
```python
def tpr_tnr(judge_verdicts: list[bool], human_labels: list[bool]) -> tuple[float, float]:
"""True Positive Rate (sensitivity) and True
Negative Rate (specificity) against human labels.
A single blended accuracy number hides which
direction the judge is biased toward — always
report both, never accuracy alone.
Raises ValueError on empty or mismatched-length
input, or if the sealed test split lacks either
class — a `zip()` over unequal lists silently
drops the extra items, which can produce
apparently valid metrics from a partial or
miscollected split.
"""
if not judge_verdicts or not human_labels:
raise ValueError("tpr_tnr: judge_verdicts and human_labels must be non-empty")
if len(judge_verdicts) != len(human_labels):
raise ValueError(
f"tpr_tnr: length mismatch ({len(judge_verdicts)} verdicts vs "
f"{len(human_labels)} labels) — check the split for a collection bug"
)
tp = sum(j and h for j, h in zip(judge_verdicts, human_labels))
fn = sum((not j) and h for j, h in zip(judge_verdicts, human_labels))
tn = sum((not j) and (not h) for j, h in zip(judge_verdicts, human_labels))
fp = sum(j and (not h) for j, h in zip(judge_verdicts, human_labels))
if (tp + fn) == 0:
raise ValueError("tpr_tnr: no positive-labeled items in this split — TPR undefined")
if (tn + fp) == 0:
raise ValueError("tpr_tnr: no negative-labeled items in this split — TNR undefined")
return tp / (tp + fn), tn / (tn + fp)
```
Agree on a TPR/TNR bar with whoever owns the eval
harness before running this step — a common starting
bar is ≥0.85 on both, tightened per bucket based on
how costly a false pass or false fail is downstream.
## 4. Bias-Correct Reported Rates
A judge with unequal TPR/TNR does not report the true
pass rate of the model under test — it reports a
rate skewed by its own asymmetric error pattern.
Correct the observed pass rate before publishing it:
```python
def bias_corrected_pass_rate(observed_pass_rate: float, tpr: float, tnr: float) -> float:
"""Rogan-Gladen style correction: recovers the
true pass rate from the judge's observed rate and
its TPR/TNR, rather than reporting the judge's raw
output as if it were ground truth.
"""
denom = tpr + tnr - 1
if denom <= 0:
raise ValueError("Judge is at or below chance — do not correct, recalibrate")
return (observed_pass_rate + tnr - 1) / denom
```
If `tpr + tnr - 1` is small (judge close to chance),
the correction blows up and becomes unreliable — that
is itself a signal the judge needs a prompt rewrite,
not a correction formula.
## 5. Pin the Snapshot
Record the exact judge model snapshot/version used
for calibration alongside the TPR/TNR numbers. An
unpinned judge (a "latest" alias that moves under
you) invalidates the calibration the moment the
underlying model changes — the TPR/TNR numbers stop
describing the judge actually in use.
## 6. Recalibrate
Two triggers, either one is sufficient:
- **Judge-model change** — any change to the pinned
snapshot, intentional or forced (deprecation).
- **Quarterly, regardless** — schedule recalibration
even with no judge change, since the underlying
task distribution (what "hard" cases look like)
drifts as the model under test improves and error
analysis surfaces new failure modes.
Recalibration reruns steps 1-4 in full — it is not a
partial refresh of just the test split.
## 7. Miscalibration Fallback
If the judge cannot hit the agreed TPR/TNR bar after
prompt iteration on the dev split:
- The judge ships **advisory-only**: its verdicts
surface in review tooling for a human to consider,
but do not gate a checkpoint promotion and are not
counted into a reported pass rate.
- Do not lower the TPR/TNR bar to make an
uncalibrated judge "pass" — that defeats the point
of calibrating in the first place.
- Prefer routing the bucket to a deterministic grader
if the criterion can be reframed as one (see
`SKILL.md`'s Graders section) over shipping a
permanently advisory-only judge.
## Different Model Family, Always
The judge model must come from a different model
family than the model under test, for every
calibration cycle and every recalibration — a judge
evaluating outputs from its own family is a biased
grader, and this cannot be corrected away with the
TPR/TNR formula above since the bias is systematic
rather than random.
SKILL.md
---
name: eval-harness-first
description: Build the evaluation harness that gates every fine-tuning run — golden sets, per-failure-mode graders, judge calibration, and base-model baselines. Use when starting a fine-tuning effort, when converting traces into an eval set, or when calibrating a judge against human labels.
---
# Eval Harness First
The Phase 0 gate for the whole plugin:
`finetuning-method-selection` and every downstream
skill assume this harness exists before a training
config gets written. The harness is not a run-end
side artifact — it is the data-curation engine. The
same labeled traces that build the goldens feed
training data, minus an explicit holdout.
**Input:** production/agent traces if they exist, or
a task spec if they don't, plus labelers willing to
grade ≥100 examples.
**Output format:** the `eval/` directory below —
goldens, graders, drift suite, and the base-model
baseline that later phases gate on.
## The Gate
No eval harness, no fine-tune. Skip to a training
config and there is nothing to measure against,
nothing to catch regressions, and no labeled data
to train on. The flywheel:
1. **Collect traces** — production/agent spans, or
synthetic tasks if none exist yet.
2. **Error analysis** — open coding on ≥100 traces,
axial coding into 4–8 failure buckets.
3. **One grader per bucket** — deterministic first;
calibrated LLM-judge only for genuinely
subjective criteria.
4. **Prioritize** by frequency × severity × value.
5. **The labeled traces feed dataset curation, minus
an explicit holdout.** Every `eval/goldens.jsonl`
ID stays excluded from training data by ID.
6. **Train.**
7. **Re-run the same harness** on the checkpoint —
not a different, looser one.
8. **Drift detection feeds back to step 2** — new
production failure modes re-open error analysis.
Steps 2–4 build the harness; steps 5–8 are why it
must exist first — it is both the training data
source and the checkpoint's exit gate.
## Building Goldens
- **From traces, when they exist:** run error
analysis — open coding on ≥100 real traces (read
them, tag failures in your own words, no fixed
taxonomy yet), then axial coding to collapse those
tags into 4–8 named failure buckets. Fewer than 4
means the coding pass was too shallow; more than 8
means buckets need merging. **Exception:**
single-failure-surface tasks (e.g. strict-schema
extraction) may land at 1–2 buckets with per-field
sub-metrics inside one grader — don't invent
artificial splits with no evidence behind them.
- **Synthetic, when traces don't exist yet:**
dimension-based generation — enumerate the axes
that matter (task type, difficulty, edge case,
persona) and sample the cross-product; free-
generated prompts cluster around whatever's
easiest to write.
- **Goldens are versioned like code** — commit
`eval/goldens.jsonl`, diff it in review, tag it per
release. It doubles as the CI regression suite.
## Graders
One grader per failure bucket from error analysis —
not one for the whole eval set. A single blended
score hides which bucket regressed.
- **Deterministic first.** Regex, schema validation,
or execution checks are cheaper, reproducible, and
need no calibration.
- **LLM-judge only for genuinely subjective
criteria** — tone, faithfulness, "which response
is better" — where no deterministic check can
express it.
- **Binary pass/fail over Likert.** A 1–5 or 1–10
scale is noisier to calibrate and harder to apply
consistently; collapse to pass/fail.
- **Drift-suite MMLU-style scoring: prefer logprob
over generate-and-extract** — a tight token budget
makes generate-and-extract parse-brittle for models
that preamble, conflating format compliance with
the knowledge being measured. Templates for all
four grader shapes and this scoring note:
`references/grader-templates.md`.
## Judge Calibration Is a Prerequisite
Any bucket routed to an LLM-judge needs calibration
before its verdicts count for anything beyond
exploration — a hard prerequisite, not a
nice-to-have. **N/A when no bucket routes to a
judge** — an all-deterministic harness has nothing
to calibrate; state that rather than leaving this
section unaddressed.
- Label ≥100 items, split **train**/**dev**/**sealed
test** (report once, no re-touching after).
- Report **TPR and TNR**, not one blended accuracy
number — a judge can hit 90% by always saying
"pass" on a skewed set.
- **Pin the judge to a fixed model snapshot** and
recalibrate on judge-model change, quarterly
regardless.
- **The judge must come from a different model family
than the model under test.**
- A judge that misses the agreed TPR/TNR bar ships
**advisory-only** — flags for human review, never
gates a promotion. Full protocol, bias correction,
and recalibration checklist:
`references/judge-calibration.md`.
## The Baseline
Before Phase 1 (method selection) starts, run the
full harness — goldens plus the capability-drift
suite — against the unmodified base model. This is
the number every later checkpoint gets compared
against.
`eval/baseline-<model>.json` is the gate token. No
baseline file, no comparison basis for
`checkpoint-promotion` — a checkpoint that "looks
better" against nothing measured isn't a finding.
## Directory Contract
```
eval/
├── goldens.jsonl # labeled traces + synthetic goldens, versioned
├── graders/ # one module per failure bucket
│ ├── schema_compliance.py
│ ├── exact_match.py
│ └── rubric_judge.py
├── drift-suite.yaml # frozen benchmarks + 200-500 domain-adjacent items
└── baseline-<model>.json # gate token: harness + drift suite vs the base model
runs/
└── <run-id>/
└── results.json # per-run harness output, one per checkpoint
```
`eval/` persists across runs and lives outside
`runs/` — the fixed measuring stick, not a run
artifact. `runs/` is disposable; `eval/` is not.
Never let a run script write into `eval/`. **Canonical
location:** every per-trace `results.json` — the
Phase 0 baseline included — lives at
`runs/<run-id>/results.json`, never under
`eval/runs/...`; an instruction requesting the
latter is wrong, not this contract.
### Phase 0 Exit Checklist
Before `finetuning-method-selection`, confirm:
1. ≥100 traces open-coded; 4–8 failure buckets (N/A
floor for synthetic goldens on a single-failure-
surface task — see the Building Goldens exception;
bucket count then comes from post-baseline error
analysis instead).
2. `eval/goldens.jsonl` committed and versioned.
3. One grader per bucket, deterministic first.
4. Judges calibrated — TPR/TNR, snapshot pinned,
different family (**N/A when no bucket routes to
an LLM-judge**; state that explicitly).
5. `eval/drift-suite.yaml` frozen.
6. `eval/baseline-<model>.json` written.
Missing any of the six (or its stated N/A)? Not
Phase 0 complete — `/finetune` checks the baseline
file before a run.
## Related Skills
General-purpose evaluation guidance (dashboards, A/B
testing, non-fine-tuning harnesses) lives in the
`llm-application-dev` plugin's `llm-evaluation`
skill — this skill covers only the fine-tuning
coupling: goldens that double as training data, and
the baseline that gates a checkpoint.
- `finetuning-method-selection` — routes here first.
- `dataset-curation` — formats these traces into
training rows.
- `trace-to-training-data` — turns graded traces into
training examples.
- `checkpoint-promotion` — consumes
`baseline-<model>.json`, re-runs this harness on
each candidate checkpoint.
## References
- `references/grader-templates.md` — runnable grader
examples per shape, plus a `drift-suite.yaml`
example and MMLU logprob-scoring note.
- `references/judge-calibration.md` — the
calibration protocol, including the all-
deterministic N/A path.