> Canonical page: https://posttrainllm.com/evaluate-local-llm

[← research lab](https://posttrainllm.com/)

Mac-local specialist field guide

# How to evaluate a local LLM

An evidence-first workflow for running LLM evaluation locally: test a local LLM against a frozen baseline, task gates, regression slices, and an explicit decision.

A local LLM evaluation should answer a product decision, not merely produce a leaderboard number. Freeze the target and baseline first, run the baseline and candidate through the same protocol, inspect task-specific slices and regressions, then record a ship, route, retry, or reject decision.

## Define the contract

Specify the inputs, expected outputs, allowed variation, protected content, and failure classes. Choose metrics that match the task: exact execution for SQL, tool and argument correctness for agents, unnecessary edit rate for autocorrection, or grounded answer criteria for retrieval. Keep a qualitative review lane, but do not let it overwrite deterministic gates.

## Freeze the evaluation set

Separate training, validation, and test data. Record provenance and deduplicate across splits. A holdout that changes for every candidate cannot support a fair comparison. Mark synthetic fixtures as synthetic and avoid describing a small suite as production proof.

## Run baseline and candidate symmetrically

Use the same prompts, decoding settings, runtime, and evaluator. Preserve raw predictions and measurement state. Record timeouts and missing values separately from zero. When the candidate is a routed specialist, evaluate both the target slice and the routing boundary.

## Measure regressions explicitly

A specialist can improve the hard gate and still become worse outside it. Report deltas for every protected slice. Define reject conditions before the run, such as meaning changes, leakage, breadth loss beyond a threshold, or latency and memory outside the target machine budget.

## Make the decision reproducible

Package the base and candidate identities, dataset revision, evaluator revision, configuration, per-slice results, missing measurements, and final decision. Keep “ship” narrower than “the candidate improved.” A routed ship can be correct when the target gain is real and the routing constraint is enforced.

PostTrainLLM includes fixture and no-GPU gates, live eval commands, canonical factory-run records, specialist packages, and public report cards. Several public results intentionally disclose missing historical latency, RAM, throughput, or raw traces rather than replacing those gaps with estimates.

 CTA: Inspect the eval leaderboard and report cards, then use the factory recipes to reproduce the workflow.

## Tool-calling evaluation: preserve the denominator

Use the [official BFCL definitions](https://gorilla.cs.berkeley.edu/leaderboard) when reporting BFCL results. A selected local slice is not the full leaderboard. Publish the exact cases, scorer, model revision, decoding settings and failure policy; separate valid JSON from correct arguments, task completion and unintended actions.

Keep task depth and out-of-domain breadth separate. Our September ReST comparison reported 12/12 versus stock 9/12 on file operations, but 25/45 versus stock 30/45 on breadth. Neither denominator should disappear into a headline percentage. Repeated runs and uncertainty remain important even when a small gate scores perfectly.

After quantization, adapter fusion or export, repeat the same gate in the actual serving runtime. A smaller artifact does not prove lower runtime memory or higher throughput. Preserve rejected candidates in the [attempt ledger](https://posttrainllm.com/docs/attempt-ledger/) so future readers can see what failed.

Continue through the factory

[fine tune llm on mac](https://posttrainllm.com/fine-tune-llm-on-mac)[mlx lora fine tuning](https://posttrainllm.com/mlx-lora-fine-tuning)[eval leaderboard](https://posttrainllm.com/eval-leaderboard)[leaderboard](https://posttrainllm.com/leaderboard)[artifacts](https://posttrainllm.com/artifacts)[recipes](https://posttrainllm.com/recipes)