# v11 baselines — committed 2026-06-09

This document records the v9 baseline against each of the six v11 ship-gate dimensions ([pace-planner-v11-ship-gate.md](../prds/pace-planner-v11-ship-gate.md)). Frozen at commit time. **Not for revision** — v11 progress measured against these exact numbers.

## v9-LoRA (currently shipped planner) baselines

| Dim | Eval | v9 score | Threshold | Gap |
|---|---|---|---|---|
| 1 | `fm-fixtures-v2` (16 prompts) | **33.3%** | ≥60% | -26.7pp |
| 2 | `bfcl-pace-12` subset (96 prompts) | structural 0% (v9 grammar has no `intent` field; deferred runtime measurement) | ≥40% | -40pp |
| 3 | `fm-fixtures-oos` (30 prompts) | **0.0%** (measured via `--skip-model` baseline 2026-06-09) | ≥80% | -80pp |
| 4 | `fm-fixtures-ambig` (20 prompts) | **0.0%** (structural, runner verified) | ≥50% | -50pp |
| 5 | Schema validity (AST exact) | **~85%** (v9 grammar is loose) | ≥95% | -10pp |
| 6 | `fm-fixtures-destructive` (10 prompts) | **0.0%** (structural, runner verified) | ≥90% | -90pp |

Non-regression gates:

| | v9 | Threshold |
|---|---|---|
| `fm-fixtures-compose` | 70% | ≥65% |
| TTFW p95 warm | ~119ms | ≤140ms |

## Why three baselines are "structural 0%"

Dimensions 3, 4, and 6 require the model to emit a new intent class that **does not exist in v9's training data, grammar, or system prompt**:

- v9 grammar: `{ "spokenText", "pointAtLabel", "clickLabel", "bodyText" }`
- v9 has no `intent` field, no `out_of_scope`, no `clarify`, no `confirm_destructive`

Probability v9 spontaneously emits a never-seen JSON shape that happens to match the new spec = ~0. We record these as 0.0% rather than running 60 model inferences to verify, because the output language is provably disjoint from the target.

**This shortcut is valid for the baseline-recording purpose**, but should be re-verified by an actual run if anyone ever ports the v9 grammar to include the new classes. As of today, no such port exists.

## Failure-mode distribution (qualitative, not scored)

Quick assessment of what v9 *does* do when fed an out-of-scope or ambiguous prompt (informally observed during prior eval sessions, not scored against new suite):

- **OOS** ("what's the weather"): v9 produces either a `pointAtLabel`/`clickLabel` referencing an element that doesn't exist, or a garbled `spokenText`. Failure mode: over-eager hallucination.
- **Ambig** ("send it" with multiple buttons): v9 picks one button confidently, no clarifying question. Failure mode: confident guess.
- **Destructive** ("delete all my emails"): v9 emits a click on whatever delete-shaped element is present. Failure mode: directly fires, no confirmation gate.

These failure modes are exactly what the new training data + rules + grammar layer should fix.

## Eval runner status

- **Existing runner** (`scripts/pace/pace-eval-fm-fixtures.py`) — handles fm-fixtures-v2 format (POINT_ID/CLICK_ID expectation).
- **New runner** (`scripts/pace/eval_pace_unhappy.py`, written 2026-06-09) — handles OOS / AMBIG / DESTRUCT formats with `EXPECT_INTENT` keys. Supports `--skip-model` for structural-baseline runs and `--serve-url` for model evaluation. Output captured as JSON for downstream tooling.
- **BFCL runner** (`scripts/bfcl/eval_bfcl.py`) — already supports v10 schema (`intent` + `payload`) via #231. Pace-12 subset committed at `~/.cache/posttrainllm/datasets/bfcl/BFCL_v3_pace12.json`.

## What this lets us do today

With these baselines committed, the v11 ship gate from [pace-planner-v11-ship-gate.md](../prds/pace-planner-v11-ship-gate.md) is now fully specified:
- All thresholds locked
- All baselines locked
- All eval suites locked (60 new fixtures live under `pace/evals/`)

The remaining work to ship-or-fail v11 is:
1. BFCL-12 subset construction + v9 score against it (#311)
2. Hand-curated training data for the 3 new intent classes (~450 rows)
3. v11 training (DoRA on combined corpus)
4. Score runner extension for new fixture formats
5. Score v11, compare to this doc

No part of this is allowed to revise the numbers above.

## 2026-06-10 ADDENDUM — Dim1 baseline was measured under the wrong harness config

The frozen 33.3% Dim1 number stands as recorded, but its provenance is now
understood and it must NOT be the comparison point for the v11 verdict:

- Score artifact `~/.cache/posttrainllm/scores/v9-LoRA-fp16-baseline-1780984389.json`
  records no sys-prompt; `score_formula.py` falls back to `eval_pace_v2.py`
  defaults = **v6-label prompt + v6 label schema**. The 33.3% run evaluated
  v9 under v6's harness.
- Re-measured 2026-06-10 with v9's as-shipped config (compose-v2 prompt +
  v9 schema + serve-side `--grammar`): **9/15 (60.0%)**, FakePace identical
  (1/15), same fixtures, same baked-hf. Same model — pure config drift.
- This likely also explains the eval-matrix "v8 73.3% → 33.3%
  irreproducible" mystery (same default-prompt drift on re-eval).
  Unverified; 10-min test when GPU frees.

**Rule for the v11 verdict:** v11 runs with its own prompt+grammar
(as-shipped), so compare it to **v9-as-shipped = 60.0%**, not 33.3%.
Dim1 threshold ≥60% therefore means: v11 ships on Dim1 only if it beats
the corrected v9 baseline. Note v9-as-shipped (60%) already ties
zero-shot Qwen3-14B on this suite.

**Process fix:** evals must always pin --sys-prompt + --schema + serve
--grammar to the model's shipped config; score artifacts now record the
sys-prompt used.

## 2026-06-11 FINAL VERDICT — v11 FAILS the ship gate. Planner freezes on qwen3-30b-a3b.

One run, as committed (plain LoRA r32, 709 rows, after the DoRA infra
failure voided the first attempt — see #326):

| Dim | v11 | Threshold | Verdict |
|---|---|---|---|
| 1 happy-path (fm-fixtures-v2) | ~rules-level | ≥60% | FAIL |
| 2 BFCL pace-12 | 27.1% (26/96) | ≥40% | FAIL |
| 3 out-of-scope | 0/30 | ≥80% | FAIL |
| 4 ambiguous | 0/20 | ≥50% | FAIL |
| 6 destructive-confirm | 5/10 | ≥90% | FAIL |
| nonreg compose/holdout | rules-level | ≥65% | FAIL |

Failure mode (verified in eval-dim3.json, not harness drift): the model
emits clean v11 JSON but `intent:"answer"` with a helpful attempt on
out-of-scope prompts — the exact over-eagerness the 176 OOS rows were
meant to train away. Train loss 0.001 = full memorization of 709 rows
with near-zero generalization of the refusal/clarify DISCIPLINE to
held-out phrasings. The 0.6B memorizes behaviors; it does not learn
rules. confirm_destructive (50%) generalized best — it has the most
surface-regular trigger words.

**Decision (per the one-run freeze):** specialist planner track CLOSED.
Pace ships qwen3-30b-a3b (already the wired default). The factory's
adaptation pipeline survives and moves to 4B+ bases per
`qlora-large-model-finetune.md` — where the zero-shot baseline already
clears most of these dimensions and fine-tuning starts from competence,
not memorization.

## 2026-06-11 zero-shot candidate gate (post-v11, planner replacement)

Same gate, no training. Run artifacts: ~/.cache/posttrainllm/runs/{fm-zeroshot-guided,qwen3-4b-zeroshot}/.

| Dim | Threshold | v11 0.6B trained | Apple FM guided | Qwen3-4B-2507 4bit |
|---|---|---|---|---|
| 1 happy-path | ≥60% | 13% | 13.3% | **66.7% PASS** |
| 2 BFCL pace-12 | ≥40% | 27.1% | 32.3% | **58.3% PASS** (p50 641ms) |
| 3 out-of-scope | ≥80% | 0% | **93.3% PASS** | 66.7% |
| 4 ambiguous | ≥50% | 0% | 5% | 10% |
| 6 destructive | ≥90% | 50% | 60% | **90.0% PASS** |

Notes:
- Apple FM required guided (@Generable) output to be measured fairly —
  plain-text mode refused every OOS prompt CORRECTLY but scored 0 for
  format (scripts/fm_bridge.swift + fm_shim.py are the harness).
- Qwen3-4B zero-shot passes 3/5 dims that eleven trained 0.6B versions
  never passed once. Misses OOS by 13pp; dim4 (clarify-not-guess) is the
  universal gap — every model guesses confidently.
- LM Studio endpoint contention (30B JIT-load mid-suite, shared with the
  other agent) poisoned two partial runs with instant-failure rows
  (p50 ~1ms signature); affected sections re-run clean.

**Planner decision: Qwen3-4B-Instruct-2507 is the fine-tune base** (QLoRA
phase Q3 target). Training closes dim3 (+13pp) and dim4 — starting from
competence, not memorization. Apple FM is the zero-footprint refusal
champion; keep as optional pre-filter/fallback on Apple Intelligence Macs.
30B stays the dev-machine reference, never the shipped default.

## 2026-06-11 held-out (h2) zero-shot baselines — the QLoRA before/after reference

Fresh 60-fixture suites (posttrainllm evals/fm-fixtures-{oos,ambig,destructive}-h2,
zero overlap with training corpus or old fixtures, commit 95dbe2b):

| Suite | Threshold | Qwen3-4B-2507 | Apple FM (guided) |
|---|---|---|---|
| oos-h2 | ≥80% | **80.0% PASS** | 96.7% |
| ambig-h2 | ≥50% | 0% | 5% |
| destructive-h2 | ≥90% | 70% | 20% |

- 4B scored HIGHER on held-out OOS than on the leaked originals — no
  contamination advantage existed; the old suite was just a different mix.
  Zero-shot 4B now passes dims 1, 2, 3.
- Clarify is the single real gap (0% across every model ever measured) —
  the QLoRA run's primary target, plus destructive consistency (+20pp).
- The Q3 fine-tune gates on THESE suites, not the leaked originals.