A local LLM evaluation should answer a product decision, not merely produce a leaderboard number. Freeze the target and baseline first, run the baseline and candidate through the same protocol, inspect task-specific slices and regressions, then record a ship, route, retry, or reject decision.
Define the contract
Specify the inputs, expected outputs, allowed variation, protected content, and failure classes. Choose metrics that match the task: exact execution for SQL, tool and argument correctness for agents, unnecessary edit rate for autocorrection, or grounded answer criteria for retrieval. Keep a qualitative review lane, but do not let it overwrite deterministic gates.
Freeze the evaluation set
Separate training, validation, and test data. Record provenance and deduplicate across splits. A holdout that changes for every candidate cannot support a fair comparison. Mark synthetic fixtures as synthetic and avoid describing a small suite as production proof.
Run baseline and candidate symmetrically
Use the same prompts, decoding settings, runtime, and evaluator. Preserve raw predictions and measurement state. Record timeouts and missing values separately from zero. When the candidate is a routed specialist, evaluate both the target slice and the routing boundary.
Measure regressions explicitly
A specialist can improve the hard gate and still become worse outside it. Report deltas for every protected slice. Define reject conditions before the run, such as meaning changes, leakage, breadth loss beyond a threshold, or latency and memory outside the target machine budget.
Make the decision reproducible
Package the base and candidate identities, dataset revision, evaluator revision, configuration, per-slice results, missing measurements, and final decision. Keep “ship” narrower than “the candidate improved.” A routed ship can be correct when the target gain is real and the routing constraint is enforced.
PostTrainLLM includes fixture and no-GPU gates, live eval commands, canonical factory-run records, specialist packages, and public report cards. Several public results intentionally disclose missing historical latency, RAM, throughput, or raw traces rather than replacing those gaps with estimates.
CTA: Inspect the eval leaderboard and report cards, then use the factory recipes to reproduce the workflow.