From .atraj rollouts to a trained specialist

B22 + B29 + B30 + posttrainllm sft chained — the closed substrate→training loop, V1.

This recipe walks the substrate-refinement cycle that closes between B22 (token-preserving trajectory recorder), B29 (trace-to-training-data), B30 (reasoning-depth classifier for mix balancing), and posttrainllm sft.

Inputs in / model out, no LLM judge in V1, no DPO.

The loop

agent rollouts → .atraj files (B22)
   ↓
posttrainllm traces-to-data        (B29)   →  raw-sft.jsonl
   ↓
posttrainllm reasoning-classify --train    →  reason.tgfr
posttrainllm reasoning-classify --score    →  scored.jsonl     (B30)
posttrainllm reasoning-classify --filter   →  balanced.jsonl
   ↓
posttrainllm sft --data balanced.jsonl --base <gallery-pin>

Step 1 — collect trajectories

posttrainllm agent specialist.tinygpt --tools tools.json \
  --trajectory-dir ~/.cache/posttrainllm/atraj/tool-call-v1 \
  --trajectory-task "tool-call" \
  --single "look up the weather in Paris"

Every rollout emits one <uuid>.atraj. Keep them under a per-task directory so B29 can --task tool-call them as a batch. See docs/agent_runtime.md §“Token-preserving trajectories (B22)” for the format.

Step 2 — turn the trajectories into SFT JSONL

posttrainllm traces-to-data ~/.cache/posttrainllm/atraj/tool-call-v1 \
  --task tool-call \
  --out raw-sft.jsonl

Two export modes (--export, issue #159):

• answer-only (default) — one row per (user → final assistant answer) pair. Intermediate tool calls and tool results are dropped; the row teaches the answer, not the actions that produced it.
• trajectory — one row per assistant turn, each carrying the full conditioning context (system, prior turns, tool calls, reconstructed tool results) with per-message supervise flags. This is the mode that teaches the tool-use sequence: every assistant turn — including bare tool calls — is a supervised target under the exact context the model saw. sft renders these rows per-block through the agent loop’s ChatML convention, consumes recorded assistant output_ids when present, and masks loss to the flagged sampled spans. New recordings include the system prefill as the first context step; older .atraj files cannot recover a system prompt that was never recorded.

Default filters: tool-echo drop (answer-only only — assistant turns that emitted a tool-call JSON but never reached {"answer": ...} are excluded), exact dedup on the full row, and MinHash near-dedup at Jaccard ≥ 0.85 — sketched on the prompt for answer-only, on the full conditioning context for trajectory.

Tune the MinHash threshold per corpus:

posttrainllm traces-to-data ~/.cache/posttrainllm/atraj/tool-call-v1 \
  --task tool-call \
  --minhash-threshold 0.7 \
  --out raw-sft.jsonl

Run with --dry-run first to preview the per-stage drop counts before committing to a write.

Step 3 — balance the depth mix

Trace dumps are single-hop heavy by default (those are the prompts that succeed soonest in production). Without balancing, the trained specialist regresses on multi-hop and comparison prompts even though both are present in raw-sft.jsonl. B30 fixes this:

# Train the depth classifier ONCE per corpus family.
posttrainllm reasoning-classify \
  --train labeled-seed.jsonl \
  --heldout labeled-heldout.jsonl \
  --out reason.tgfr

posttrainllm reasoning-classify --score raw-sft.jsonl \
  --model reason.tgfr --out scored.jsonl --field user

posttrainllm reasoning-classify --filter scored.jsonl \
  --target-mix "single=0.3,multi=0.5,comparison=0.2,other=0.0" \
  --out balanced.jsonl

The Castform-aligned 0.3/0.5/0.2 mix biases toward the depth that regresses most often. Adjust per the per-class evals you care about. See balanced-training-mix.md for the full B30 details.

Step 4 — fine-tune

posttrainllm sft --base <pin-from-gallery> --data balanced.jsonl ...

What V1 is NOT

These are explicit deferrals, all flagged in the CLI so the operator knows what’s missing:

• No LLM-pivot judge filter. --judge-model <id> is reserved but exits non-zero in V1 — wiring the judge would need a live LLM, and this recipe is intentionally compute-light. Coming in B29 V2 via posttrainllm judge (E7) as a subprocess.
• No DPO-pair construction. --mode dpo is reserved but rejected; V1 ships --mode sft only. DPO needs either a per-step reward signal or a judge-score margin; PRD §“Open questions” picks the shape.
• No external observability ingest (Braintrust / Langfuse). The .atraj files are the V1 substrate; external ingest is V2.

Where this lives in code

Pairs with

• [[balanced-training-mix.md]] — B30’s standalone recipe.
• [[B35-local-agent-vertical-poc.md]] — downstream consumer.
• [[B32-eval-ci-gate.md]] — gate the trained specialist before promoting it to a gallery pin.