<!-- agent-source-envelope -->
# Artifact: Pace Intent Router v8 | PTLLM

> A 49.5M intent classifier that was extremely fast and strong on its source-matched synthetic holdout, then failed the first leakage-checked sealed…

Canonical: https://posttrainllm.com/artifacts/pace-intent-router-v8
<!-- /agent-source-envelope -->

49.5M on-device classifier

Release-ready weights

2026-07-18

This is the most important rejection in the published model set. Pace v8 scored 95.5% on 14,995 source-matched synthetic examples at roughly 3ms, but only 57.1% on a new 63-instance sealed suite that frontier aced. The weights remain useful as a latency floor and as proof that the synthetic generator overfit its template families—not as the production winner.

classification, routing, from scratch, Mac local

## Headline Numbers

Evidence class applies to each value: measured = retained evaluation or
runtime result; historical = recorded result not freshly reproduced;
derived = calculated or decided from records; observed = current artifact
state; not-measured = an explicit evidence gap.

**Sealed accuracy:** 57.1%

63 held-out instances, two passes

Evidence: measured

**Frontier retained:** 57.1%

Codex gpt-5.5 scored 100% on the same ruler

Evidence: measured

**Warm latency:** 3.8ms

mean local sealed-eval latency

Evidence: measured

**Source-matched holdout:** 95.5%

14,995 synthetic rows; did not generalize

Evidence: measured

## Competitive Context

| System | Metric | Score | Size / Class | Comparable? | Readout |
| --- | --- | --- | --- | --- | --- |
| Pace Intent Router v8 | sealed exact intent accuracy | 57.1% | 49.5M | Direct | Fastest local entry, but rejected because fresh phrasing exposed distribution overfit. |
| Qwen3-4B-Instruct-2507 4-bit | same sealed exact intent accuracy | 93.7% | 4B | Direct | Capability winner among local entries at 211ms mean warm latency. |
| Apple FoundationModels | same sealed exact intent accuracy | 92.1% | on-device system model | Direct | 522ms mean warm latency; much slower than Pace v8 but materially more accurate on fresh phrasing. |
| Codex gpt-5.5 | same sealed exact intent accuracy | 100% | frontier anchor | Direct | Passed the frozen 99% frontier-ceiling gate. Its amortized batch latency is not comparable with per-instance local timing. |

Direct rows share this artifact's eval setup. Directional rows are
useful market context but should not be read as leaderboard claims.

## Attempts and decisions

| Evidence stage | Accuracy | Latency | Decision | Lesson |
| --- | --- | --- | --- | --- |
| Source-matched synthetic holdout | 95.5% | 3.1ms p50 | promising | The pipeline learned its generator distribution |
| Sealed V1 | 57.1% | 3.8ms mean | reject-production-winner | Fresh user-like phrasing broke the apparent win |

## Sealed V1 result

| Entry | Accuracy | Unknown recall | Warm latency | Readout |
| --- | --- | --- | --- | --- |
| Codex gpt-5.5 | 100.0% | 100.0% | 519ms amortized batch | Frontier ruler passed |
| Qwen3 4B 4-bit | 93.7% | 77.8% | 211ms | Local capability winner |
| Apple FoundationModels | 92.1% | 77.8% | 522ms | Accurate, slower |
| Pace v8 | 57.1% | 55.6% | 3.8ms | Latency floor; rejected |

## Release Blockers

### Fresh-distribution generalization failed

The earlier 95.5% result was source-matched to the synthetic generator. Accuracy fell to 57.1% on leakage-checked sealed V1.

**Unblock:** Generate public-development examples from failure themes—not sealed prompts—and require a newly generated sealed V2 for any successor claim.

### Rejected as the production winner

Qwen3 4B and Apple FoundationModels both exceeded 92% on the same sealed suite.

**Unblock:** Keep v8 only as a latency floor and training-pipeline artifact until a successor clears the sealed gate.

## Evidence

- [Hugging Face model](https://huggingface.co/posttrainllm/pace-intent-router-v8)

- [Model card](https://github.com/PostTrainLLM/posttrainllm/blob/main/specialists/pace-intent-router-v8/model_card.md)

- [Eval report](https://github.com/PostTrainLLM/posttrainllm/blob/main/specialists/pace-intent-router-v8/eval_report.json)

- [Sealed V1 report](https://github.com/PostTrainLLM/posttrainllm/blob/main/evals/everyday-benchmark/pace-intent-sealed-v1.md)

- [Sealed result receipt](https://github.com/PostTrainLLM/posttrainllm/blob/main/evals/everyday-benchmark/receipts/pace-intent-router-v8-sealed-v1.json)

## Next Release Action

Do not tune on sealed V1. Build a broader public-development corpus from the failure themes, train a successor, and judge it once on a newly generated sealed V2.
