# How to evaluate a local LLM | PTLLM > Evaluate a local LLM with paired target and regression checks. Walk through a real report card and validate its evidence format offline without loading a model. Canonical: https://posttrainllm.com/evaluate-local-llm Evaluate a local LLM with paired target and regression checks. Walk through a real report card and validate its evidence format offline without loading a model. A local LLM evaluation should answer a product decision, not merely produce a leaderboard number. Freeze the target and baseline first, run the baseline and candidate through the same protocol, inspect task-specific slices and regressions, then record a ship, route, retry, or reject decision. ## Define the contract Specify the inputs, expected outputs, allowed variation, protected content, and failure classes. Choose metrics that match the task: exact execution for SQL, tool and argument correctness for agents, unnecessary edit rate for autocorrection, or grounded answer criteria for retrieval. Keep a qualitative review lane, but do not let it overwrite deterministic gates. ## Freeze the evaluation set Separate training, validation, and test data. Record provenance and deduplicate across splits. A holdout that changes for every candidate cannot support a fair comparison. Mark synthetic fixtures as synthetic and avoid describing a small suite as production proof. ## Run baseline and candidate symmetrically Use the same prompts, decoding settings, runtime, and evaluator. Preserve raw predictions and measurement state. Record timeouts and missing values separately from zero. When the candidate is a routed specialist, evaluate both the target slice and the routing boundary. ## Measure regressions explicitly A specialist can improve the hard gate and still become worse outside it. Report deltas for every protected slice. Define reject conditions before the run, such as meaning changes, leakage, breadth loss beyond a threshold, or latency and memory outside the target machine budget. ## Make the decision reproducible Package the base and candidate identities, dataset revision, evaluator revision, configuration, per-slice results, missing measurements, and final decision. Keep “ship” narrower than “the candidate improved.” A routed ship can be correct when the target gain is real and the routing constraint is enforced. PostTrainLLM includes fixture and no-GPU gates, live eval commands, canonical factory-run records, specialist packages, and public report cards. Several public results intentionally disclose missing historical latency, RAM, throughput, or raw traces rather than replacing those gaps with estimates. ## Read a real local LLM evaluation report Start with the retained Qwen3-4B ReST file-operations report card. It shows a target improvement alongside a broader regression. These are historical results, not a fresh evaluation performed for this guide. Open the [report card](https://posttrainllm.com/report-cards/qwen3-4b-rest-fused) and its [machine-readable JSON](https://posttrainllm.com/report-cards/qwen3-4b-rest-fused.json). Its file-operations gate rises from 0.75 to 1.00, while breadth falls from 0.667 to 0.556 (rounded to three decimal places). Both pairs are marked historical. A better score on the specialist task does not establish a better general assistant. Check the base model, specialist artifact and recorded configuration first. Then inspect the target and regression gates, the evidence states, the decision and its verification blockers. Missing commands, frozen-eval identity or overlap checks cannot be inferred from a good score. The card says routed-ship and verified: false: retain the file-operations specialist within its routing constraint, reject it as a general successor and do not use it as Pace’s default planner. A recorded decision and a schema-valid report do not establish a fully verified ship. For your own evaluation, declare both a target gate and a protected regression gate before running the candidate. Preserve paired outputs and reject decisions when the candidate fails a requirement. ## Validate the evidence format offline From a clone of the PostTrainLLM repository, with Python 3 available, run the offline checker: ``` python3 scripts/factory/check_fine_tune_report_card.py browser/public/report-cards/qwen3-4b-rest-fused.json ``` This command validates the committed card’s schema, provenance states and decision consistency. It does not rerun a model evaluation or prove the historical scores. To reproduce a model result, you also need the exact dataset split, model and adapter revisions, evaluator, decoding settings and serving runtime. Name any missing input rather than estimating a new result. **CTA:** Inspect the eval leaderboard and report cards, then use the factory recipes to reproduce the workflow. ## Tool-calling evaluation: preserve the denominator Use the [official BFCL definitions](https://gorilla.cs.berkeley.edu/leaderboard) when reporting BFCL results. A selected local slice is not the full leaderboard. Publish the exact cases, scorer, model revision, decoding settings and failure policy; separate valid JSON from correct arguments, task completion and unintended actions. Keep task depth and out-of-domain breadth separate. Our September ReST comparison reported 12/12 versus stock 9/12 on file operations, but 25/45 versus stock 30/45 on breadth. Neither denominator should disappear into a headline percentage. Repeated runs and uncertainty remain important even when a small gate scores perfectly. After quantization, adapter fusion or export, repeat the same gate in the actual serving runtime. A smaller artifact does not prove lower runtime memory or higher throughput. Preserve rejected candidates in the [attempt ledger](https://posttrainllm.com/docs/attempt-ledger) so future readers can see what failed. ## Continue through the factory - [fine tune llm on mac](https://posttrainllm.com/fine-tune-llm-on-mac) - [mlx lora fine tuning](https://posttrainllm.com/mlx-lora-fine-tuning) - [eval leaderboard](https://posttrainllm.com/eval-leaderboard) - [leaderboard](https://posttrainllm.com/leaderboard) - [artifacts](https://posttrainllm.com/artifacts) - [recipes](https://posttrainllm.com/recipes)