references/conversion-recipes.md
Last verified: 2026-07-13
# Conversion Recipes
Concrete JSONL-to-JSONL conversions for every
pattern in `SKILL.md`: a graded trace to an SFT
row, a pair of graded traces to a DPO pair, an
expert correction to an SFT row, the
rejection-sampling loop with reward-threshold
selection, and the goldens-holdout check that
must run before any converted row merges into a
training set. Input records match
`eval-harness-first`'s `runs/<run-id>/results.json`
shape; output records match `dataset-curation`'s
Format Selection table exactly — field names are
copied, not paraphrased. No base-model names
appear below; where a model matters, load it from
`finetuning-method-selection`'s
`references/model-catalog.md`.
## 1. Graded Trace to SFT Example
Input — one row from `runs/<run-id>/results.json`,
a single-turn trace that passed with a reward
above the batch's top-fraction threshold:
```json
{"task_id": "t-118", "trace_id": "t-118-a1", "messages": [{"role": "user", "content": "Summarize the incident report in two sentences."}, {"role": "assistant", "content": "A misconfigured retry policy caused a 12-minute outage in the billing service. Root cause was fixed and a regression test was added."}], "verdict": "pass", "reward": 0.94, "grader": "rubric_judge"}
```
Output — `dataset-curation`'s ChatML shape, grading
metadata stripped since the trainer only needs
`messages`:
```json
{"messages": [{"role": "user", "content": "Summarize the incident report in two sentences."}, {"role": "assistant", "content": "A misconfigured retry policy caused a 12-minute outage in the billing service. Root cause was fixed and a regression test was added."}]}
```
Only `messages` survives the conversion. `task_id`
and `trace_id` still get written to the dataset
card's provenance field (see the holdout check
below) — they are dropped from the training row
itself, not discarded entirely.
## 2. Two Graded Traces to a DPO Pair
Input — two traces sharing one `task_id`, from the
same rollout batch, with different rewards:
```json
{"task_id": "t-204", "trace_id": "t-204-a1", "messages": [{"role": "user", "content": "Write a commit message for a null-check fix."}, {"role": "assistant", "content": "Fix null pointer exception in user lookup by validating the session before dereferencing it."}], "verdict": "pass", "reward": 0.88, "grader": "rubric_judge"}
{"task_id": "t-204", "trace_id": "t-204-a4", "messages": [{"role": "user", "content": "Write a commit message for a null-check fix."}, {"role": "assistant", "content": "misc changes"}], "verdict": "fail", "reward": 0.11, "grader": "rubric_judge"}
```
Selection, per `preference-optimization`'s Pair
Construction formula — `chosen` is the top-reward
trace for the `task_id`; `rejected` is whichever
trace in that task's trajectory set sits closest
to μ−2σ of the reward distribution, not the
lowest-reward trace by default (here, with only
two candidates, the low trace happens to be the
μ−2σ pick; a batch with more sampled candidates
per task selects a rejected member above the
minimum):
```python
def select_pair(trajectories):
"""trajectories: same task_id, each a dict with
'reward' and 'messages'. Returns (chosen, rejected) —
always two distinct records; raises if fewer than two
trajectories are given."""
if len(trajectories) < 2:
raise ValueError("select_pair needs >=2 trajectories to form a pair")
ranked = sorted(trajectories, key=lambda t: t["reward"])
chosen = ranked[-1]
candidates = [t for t in trajectories if t is not chosen]
rewards = [t["reward"] for t in trajectories]
mu = sum(rewards) / len(rewards)
variance = sum((r - mu) ** 2 for r in rewards) / len(rewards)
sigma = variance ** 0.5
target = mu - 2 * sigma
rejected = min(candidates, key=lambda t: abs(t["reward"] - target))
return chosen, rejected
```
Output — `dataset-curation`'s DPO pair shape, with
`prompt` pulled from the shared user turn and
`chosen`/`rejected` from each trace's final
assistant turn:
```json
{"prompt": "Write a commit message for a null-check fix.", "chosen": "Fix null pointer exception in user lookup by validating the session before dereferencing it.", "rejected": "misc changes"}
```
## 3. Correction Record to SFT Example
Input — a failing trace plus a human expert's
corrected output, no reward field required since a
human already validated the correction:
```json
{"task_id": "t-311", "trace_id": "t-311-a2", "messages": [{"role": "user", "content": "Extract the invoice total as a JSON number."}, {"role": "assistant", "content": "The total is around $4,200"}], "verdict": "fail", "grader": "schema_compliance", "correction": {"content": "{\"total\": 4200.00}", "corrected_by": "reviewer-07"}}
```
Output — the corrected content replaces the
failing assistant turn; the original failing
content never enters the training set:
```json
{"messages": [{"role": "user", "content": "Extract the invoice total as a JSON number."}, {"role": "assistant", "content": "{\"total\": 4200.00}"}]}
```
Route corrections into the SFT set directly, per
`SKILL.md`'s SFT From Traces section — skip the
reward-threshold gate below for these rows.
## 4. Rejection-Sampling Loop
Sample several candidate completions per prompt,
grade each, and keep only the top-reward fraction
— the Agent-lightning pattern named in `SKILL.md`:
```python
MAX_CANDIDATES = 32 # ceiling on model calls per prompt for this recipe
def rejection_sample(prompt, policy, grader, n=8, keep_fraction=0.25):
"""Generate n candidates for prompt, grade each, and
keep the top keep_fraction by reward. This recipe is
for small fixed batches — n is capped at
MAX_CANDIDATES; a larger sampling budget needs a
dedicated rollout pipeline with its own concurrency
and cost controls, not this loop."""
if n > MAX_CANDIDATES:
raise ValueError(f"n={n} exceeds MAX_CANDIDATES={MAX_CANDIDATES}")
candidates = [policy.generate(prompt) for _ in range(n)]
graded = [(c, grader.score(prompt, c)) for c in candidates]
graded.sort(key=lambda pair: pair[1], reverse=True)
keep_n = max(1, int(len(graded) * keep_fraction))
kept = graded[:keep_n]
return [
{"messages": [
{"role": "user", "content": prompt},
{"role": "assistant", "content": completion},
]}
for completion, reward in kept
]
```
At `keep_fraction=0.25` and `n=8`, two candidates
per prompt survive into the SFT set — tune
`keep_fraction` against the batch's reward
distribution rather than a fixed count, since a
harder prompt set shifts the whole distribution
down.
## 5. Goldens-Holdout Check
Run this before any converted batch merges into
the training set — per `SKILL.md`'s Hygiene
section, a golden ID leaking into training data
silently inflates every later eval run against
that same golden:
```python
import json
def load_golden_ids(goldens_path):
with open(goldens_path) as f:
return {json.loads(line)["task_id"] for line in f}
def filter_holdout(candidate_rows, golden_ids):
"""candidate_rows: dicts still carrying task_id
before provenance stripping. Returns only rows
whose task_id never appears in the goldens."""
kept, dropped = [], []
for row in candidate_rows:
if row["task_id"] in golden_ids:
dropped.append(row)
else:
kept.append(row)
return kept, dropped
```
Run `filter_holdout` before the `messages`-only
stripping shown in recipe 1 — once `task_id` is
gone, the check has nothing to match against.
Log `dropped` rather than silently discarding it;
a large `dropped` count usually means the trace
collection step is resampling goldens instead of
production traffic.
SKILL.md
---
name: trace-to-training-data
description: Convert evaluation traces and production logs into SFT examples and preference pairs. Use when graded traces or failure examples exist and need to become training data, when applying rejection sampling to model outputs, or when building DPO pairs from passing and failing runs.
---
# Trace To Training Data
This skill assumes `eval-harness-first`
already graded the traces being
converted here — goldens, graders,
and `runs/<run-id>/results.json`
all exist before conversion
starts. This is the flywheel edge
that skill names in its own flow:
"the same labeled traces become
the training set." Conversion
happens here; grading already
happened upstream.
**Input:** graded traces —
`eval/goldens.jsonl` plus
`runs/<run-id>/results.json`, each
row carrying a `task_id`, a
`verdict` from the grader, and a
`reward` when the task supports a
scalar score (judge score,
execution partial-credit, or an
RLVR verifier):
```json
{"task_id": "t-042", "trace_id": "t-042-a3",
"messages": [{"role": "user", "content": "..."}],
"verdict": "pass", "reward": 0.91,
"grader": "exact_match"}
```
**Output format:** rows shaped
exactly like `dataset-curation`'s
Format Selection table — SFT
`messages` rows or DPO
`prompt`/`chosen`/`rejected`
pairs — so this skill's output is
that skill's input with no
reshaping step in between.
## The Principle
The eval harness already did the
labeling work: every trace in
`results.json` carries a verdict,
and often a reward, before this
skill ever touches it. Converting
a graded trace into a training
row is mechanical — pick a shape
from `dataset-curation`'s table,
map fields, write JSONL.
**Curation is the work that
remains** — which traces clear a
quality bar, which pairs are
informative, and which rows must
never enter the training set at
all.
Treat any conversion step that
requires re-judging a trace as a
sign the harness is missing a
grader, not a gap this skill
should paper over. A trace with
no verdict or reward isn't
convertible yet — route it back
to `eval-harness-first` first,
don't hand-label it here to
unblock conversion.
## SFT From Traces
- **Keep the top-reward fraction
of successful trajectories**,
not every passing one. Rank
passing traces by reward and
take a fraction (the
Agent-lightning pattern) rather
than every trace that merely
cleared the pass bar — a trace
that barely passed is a weaker
SFT signal than one that scored
well above threshold.
- **Expert-corrected failures
become gold SFT examples
directly** (the Langfuse
pattern) — when a human edits a
failing trace's output into a
correct one, that correction
needs no reward threshold; a
human already validated it.
Route corrections straight into
the SFT set.
- **Step-level masking beats
whole-trajectory discard for
multi-step traces.** When only
some steps in a multi-step
trajectory are bad, mask the
loss on the bad steps and keep
the good ones, rather than
discarding the whole trajectory.
SRFT reports 32.2% vs. 30.9% on
SWE-bench for step-level critic
masking over trajectory discard
— a real, if modest, gap from
the finer-grained cut.
## Preference Pairs From Traces
- **Build pairs from
passing-vs-failing trajectories
on the SAME task**, never from
unrelated best- and
worst-scoring traces pulled
across different tasks —
cross-task pairs teach the
model to prefer one task over
another, not one response over
another.
- **Select the rejected member at
μ−2σ of the reward distribution
for that task, never the
absolute minimum.**
`preference-optimization`'s
Pair Construction section owns
the full selection formula;
this skill supplies the graded
trajectories it consumes.
- **Judge-scored delta selection
cuts pair volume without
cutting signal.** Score each
candidate pair by
chosen-minus-rejected judge
delta and keep only the
highest-delta subset — the top
5k of a 16.5k candidate pool
matched the full pool's
downstream result. Build the
full candidate set first, then
filter by delta; don't cap
generation at 5k up front.
## Hygiene
- **Scan for secrets and PII before any row ships,
and redact what's found.** Traces sourced from
production logs can carry credentials, API keys,
tokens, or customer data — run a secret/PII scan
over every SFT and DPO row and redact matches;
conversion fails closed (the row is dropped, not
shipped with the raw content) if sensitive fields
remain after redaction. Never commit secrets.
- **Eval goldens must never leak
into training data.** Hold
every `eval/goldens.jsonl` ID
out of every converted SFT and
DPO set — a trace that also
appears as a golden trains on
the exact item the checkpoint
gets graded against later,
silently inflating every
subsequent eval run.
- **Dedup against the training
set**, not just within the
newly converted rows —
exact-match or
embedding-similarity, matching
`dataset-curation`'s dedup
method field, run against
whatever training data already
exists before this batch merges
in.
- **Provenance goes into the
dataset card.** Every converted
row must trace back to its
source `run_id` and `trace_id`
— `dataset-curation`'s
Provenance field checks for
exactly this link back to
`trace-to-training-data`
output; a row with no traceable
source isn't ready to merge.
## Related Skills
- `eval-harness-first` — produces
the graded traces this skill
converts; a trace with no
verdict or reward isn't
convertible yet, route it back
there before conversion.
- `dataset-curation` — owns the
target formats and the dataset
card this skill's provenance
data feeds; converted rows must
match its Format Selection
table field names exactly, not
an approximation of them.
- `preference-optimization` —
consumes the DPO pairs this
skill builds and owns the full
μ−2σ rejection-selection
formula referenced above.
Worked JSONL-to-JSONL conversions
— graded trace to SFT row, trace
pair to DPO pair, correction to
SFT row, the rejection-sampling
loop, and the goldens-holdout
check — live in
`references/conversion-recipes.md`.