Factory Eval Protocol
Factory Eval Protocol
The eval protocol exists to prevent training against noise.
Rules
- Freeze the eval before training.
- Run the baseline first.
- Keep primary score and regression score separate.
- Report skipped checks.
- Do not ship a specialist without a before/after table.
Required Metrics
Every candidate report should include:
- primary task score
- baseline score
- score delta
- pass/fail
- regression/breadth score
- latency if available
- RAM or peak RSS if available
- token throughput if available
- train time
- eval cost/time
- parse/error rate where relevant
Evals To Prefer
Use the most specific verified eval:
- Tool calling / agentic: BFCL and Pace fixtures.
- Planner: Pace ship gate / unhappy-path fixtures.
- Routing:
eval-router. - SQL:
eval-sql. - Compression:
eval-scaledown. - Escalation:
eval-escalate. - General model sanity:
run-lm-evaloreval-gateover E0 rows.
Frontier Calibration
For benchmark-style evals, keep the existing rule: a frontier or trusted incumbent must be near-ceiling before the eval is used to grade small models. If the eval punishes better-than-gold answers or ungroundable golds, use it for training only, not reporting.
Ship/Reject Discipline
Ship only when:
- primary score clears threshold
- regression/breadth drop is acceptable
- failure classes are understood
- artifact can be reproduced or at least located
- report is complete
Reject when:
- primary score does not beat baseline
- breadth/regression damage erases the gain
- eval was not stable enough to trust
- artifact cannot be loaded or packaged