# Factory Eval Protocol

The eval protocol exists to prevent training against noise.

## Rules

1. Freeze the eval before training.
2. Run the baseline first.
3. Keep primary score and regression score separate.
4. Report skipped checks.
5. Do not ship a specialist without a before/after table.

## Required Metrics

Every candidate report should include:

- primary task score
- baseline score
- score delta
- pass/fail
- regression/breadth score
- latency if available
- RAM or peak RSS if available
- token throughput if available
- train time
- eval cost/time
- parse/error rate where relevant

## Evals To Prefer

Use the most specific verified eval:

- Tool calling / agentic: BFCL and Pace fixtures.
- Planner: Pace ship gate / unhappy-path fixtures.
- Routing: `eval-router`.
- SQL: `eval-sql`.
- Compression: `eval-scaledown`.
- Escalation: `eval-escalate`.
- General model sanity: `run-lm-eval` or `eval-gate` over E0 rows.

## Frontier Calibration

For benchmark-style evals, keep the existing rule: a frontier or trusted
incumbent must be near-ceiling before the eval is used to grade small models.
If the eval punishes better-than-gold answers or ungroundable golds, use it for
training only, not reporting.

## Ship/Reject Discipline

Ship only when:

- primary score clears threshold
- regression/breadth drop is acceptable
- failure classes are understood
- artifact can be reproduced or at least located
- report is complete

Reject when:

- primary score does not beat baseline
- breadth/regression damage erases the gain
- eval was not stable enough to trust
- artifact cannot be loaded or packaged