B32 `posttrainllm eval` as a CI / pre-commit gate

status: scaffolding-shipped
owner: unassigned
created: 2026-06-13
parent_plan: docs/PLAN.md §3 Tier B (B32)
parent_strategy: docs/sessions/2026-06-13-market-landscape-mac-first.md (move #4)
related_prds: E1-bfcl-eval.md, E2-tau-bench-eval.md (the shipped harnesses this reframes), B23-agent-eval-protocol.md (the rigor B32 invokes)

PRD — Reframe the eval harnesses as a developer-workflow gate

Status (updated 2026-06-18): scaffolding shipped. Pure gate logic (TinyGPTModel/EvalGate.swift: direction heuristic, pp thresholds, per-suite override, missing-baseline handling, K-pass mean + repeated-run stdev/stderr/95% CI + optional B23 budget metadata in gate-result.json) with unit tests in EvalGateTests.swift. CLI (Sources/TinyGPT/EvalGate.swift): --spec / eval-gate.json / posttrainllm.project.json eval block resolution, --candidate (no-GPU path), --baseline, --threshold, --passes, --budget, --update-baseline, gate-result.json, exit 0/1. Action at .github/actions/posttrainllm-eval-gate/ forwards spec, candidate, passes, and budget; recipe docs/recipes/eval-gate.md, smoke evals/eval-gate-smoke.sh (asserts both exit codes against committed fixtures, including repeated-row CI output). Remaining to flip to ✅: run a real specialist’s suites end to end through the gate (the command-driven path) on a self-hosted Mac runner — exercised by the user, not by CI.

Goal

Ship posttrainllm eval-gate — a single command that runs a project’s declared eval suites against a model + adapter, compares to a stored baseline, and exits non-zero when any suite regresses past a threshold. Plus a GitHub Action wrapper and a pre-commit hook recipe.

The landscape finding (docs/sessions/2026-06-13-market-landscape-mac-first.md): BFCL and τ-bench are academic leaderboards with no product wrapping them into “gate my SLM in CI.” We already wrapped both (E1/E2 shipped). B32 is the framing work that turns a benchmark wrapper into a workflow primitive — near-zero new model code, high product leverage.

Why now

• E1 (BFCL), E2 (τ-bench), E3 (lm-eval), E5 (HumanEval), the unhappy-path suite, and B27 leaderboard all ship. The infrastructure to run evals is done; the missing piece is a one-command, exit-code-driven gate.
• Competitors sell eval as a SaaS that ingests your traces. A local exit-code gate that never phones home is the structural counter — it runs in your CI on your runner.
• Pairs with the B31 project file: the gate reads which models the project pins + which suites it declares, so posttrainllm eval-gate is zero-config in a project that already has a posttrainllm.project.json.

Scope — in

• Sources/TinyGPT/EvalGate.swift — orchestrator:
  • Reads an eval-gate spec (a [eval] block in posttrainllm.project.json, or a standalone eval-gate.json): which suites, which model/adapter, which baseline file, per-suite regression thresholds (default: any drop > 2pp fails; configurable per suite).
  • Runs each declared suite via the shipped harness (E1/E2/E3/E5/unhappy).
  • Compares to the baseline (the same EvalCompare.Row JSONL format).
  • Prints a human summary + writes a machine gate-result.json.
  • Exit code: 0 if all suites pass their thresholds, 1 otherwise.
• --update-baseline flag — re-stamp the baseline after an intentional improvement (the “accept the new numbers” path).
• --passes K — delegates to B23 protocol (K-pass mean ± σ) so the gate isn’t fooled by single-run noise. Default K=1 for CI speed; recipe recommends K=3 for release gates.
• GitHub Action: .github/actions/posttrainllm-eval-gate/action.yml — thin wrapper that runs the binary on a self-hosted Mac runner and annotates the PR with the pass/fail table.
• Pre-commit recipe: docs/recipes/eval-gate.md documents both the .pre-commit-config.yaml hook and the Action.

Scope — out

• Hosted dashboard for gate history. The leaderboard (B27) + train-viewer (C10) cover visualization; the gate just emits JSONL they can read.
• Cloud CI — the gate is designed for a self-hosted Mac runner (that’s the point: the model never leaves the device). GitHub-hosted Linux runners can’t run the MLX path; documented as a constraint.
• Auto-bisect (find which commit regressed the eval). Useful follow-up; defer.

Files to touch

Don’t touch

• The individual eval harnesses (E1/E2/E3/E5) — the gate calls them as library functions, no changes.
• Sources/TinyGPT/TinyGPT.swift beyond the one switch case.

Acceptance criteria

• posttrainllm eval-gate in a dir with a valid spec runs the declared suites and exits 0 when all pass, 1 when any regresses past threshold.
• --update-baseline re-stamps the baseline JSONL.
• The GitHub Action annotates a PR with the suite table + pass/fail.
• eval-gate-smoke.sh asserts both exit codes (a deliberately regressed fixture fails; a matching fixture passes).
• The recipe reproduces on a clean checkout with a self-hosted Mac runner.

Reference patterns

• Sources/TinyGPT/EvalCompare.swift — the baseline-vs-candidate diff logic; the gate adds the threshold + exit-code layer on top.
• scripts/eval_planner_report.py — the existing champion-vs-candidate verdict pattern (the E9 gate’s PASS/HOLD logic is the template for B32’s per-suite verdict).
• B23 agent-eval-protocol — the K-pass rigor the gate invokes.

Open questions

• Default threshold direction. Recommendation: per-suite, default “fail if score drops > 2pp from baseline”; suites where lower-is-better (PPL, latency) invert automatically via the EvalCompare direction metadata.
• Whether the gate spec lives in posttrainllm.project.json or a separate eval-gate.json. Recommendation: an optional block in the project file (one config surface), with the standalone file as an override.