Skip to content
posttrainllm docs
Esc
↑↓navigate↵open⌘Jpreview
On this page

v11 baselines — committed 2026-06-09

v11 baselines — committed 2026-06-09

This document records the v9 baseline against each of the six v11 ship-gate dimensions (pace-planner-v11-ship-gate.md). Frozen at commit time. Not for revision — v11 progress measured against these exact numbers.

v9-LoRA (currently shipped planner) baselines

Dim Eval v9 score Threshold Gap
1 fm-fixtures-v2 (16 prompts) 33.3% ≥60% -26.7pp
2 bfcl-pace-12 subset (96 prompts) structural 0% (v9 grammar has no intent field; deferred runtime measurement) ≥40% -40pp
3 fm-fixtures-oos (30 prompts) 0.0% (measured via --skip-model baseline 2026-06-09) ≥80% -80pp
4 fm-fixtures-ambig (20 prompts) 0.0% (structural, runner verified) ≥50% -50pp
5 Schema validity (AST exact) ~85% (v9 grammar is loose) ≥95% -10pp
6 fm-fixtures-destructive (10 prompts) 0.0% (structural, runner verified) ≥90% -90pp

Non-regression gates:

v9 Threshold
fm-fixtures-compose 70% ≥65%
TTFW p95 warm ~119ms ≤140ms

Why three baselines are “structural 0%”

Dimensions 3, 4, and 6 require the model to emit a new intent class that does not exist in v9’s training data, grammar, or system prompt:

  • v9 grammar: { "spokenText", "pointAtLabel", "clickLabel", "bodyText" }
  • v9 has no intent field, no out_of_scope, no clarify, no confirm_destructive

Probability v9 spontaneously emits a never-seen JSON shape that happens to match the new spec = ~0. We record these as 0.0% rather than running 60 model inferences to verify, because the output language is provably disjoint from the target.

This shortcut is valid for the baseline-recording purpose, but should be re-verified by an actual run if anyone ever ports the v9 grammar to include the new classes. As of today, no such port exists.

Failure-mode distribution (qualitative, not scored)

Quick assessment of what v9 does do when fed an out-of-scope or ambiguous prompt (informally observed during prior eval sessions, not scored against new suite):

  • OOS (“what’s the weather”): v9 produces either a pointAtLabel/clickLabel referencing an element that doesn’t exist, or a garbled spokenText. Failure mode: over-eager hallucination.
  • Ambig (“send it” with multiple buttons): v9 picks one button confidently, no clarifying question. Failure mode: confident guess.
  • Destructive (“delete all my emails”): v9 emits a click on whatever delete-shaped element is present. Failure mode: directly fires, no confirmation gate.

These failure modes are exactly what the new training data + rules + grammar layer should fix.

Eval runner status

  • Existing runner (scripts/pace-eval-fm-fixtures.py) — handles fm-fixtures-v2 format (POINT_ID/CLICK_ID expectation).
  • New runner (scripts/eval_pace_unhappy.py, written 2026-06-09) — handles OOS / AMBIG / DESTRUCT formats with EXPECT_INTENT keys. Supports --skip-model for structural-baseline runs and --serve-url for model evaluation. Output captured as JSON for downstream tooling.
  • BFCL runner (scripts/eval_bfcl.py) — already supports v10 schema (intent + payload) via #231. Pace-12 subset committed at ~/.cache/posttrainllm/datasets/bfcl/BFCL_v3_pace12.json.

What this lets us do today

With these baselines committed, the v11 ship gate from pace-planner-v11-ship-gate.md is now fully specified:

  • All thresholds locked
  • All baselines locked
  • All eval suites locked (60 new fixtures live under pace/evals/)

The remaining work to ship-or-fail v11 is:

  1. BFCL-12 subset construction + v9 score against it (#311)
  2. Hand-curated training data for the 3 new intent classes (~450 rows)
  3. v11 training (DoRA on combined corpus)
  4. Score runner extension for new fixture formats
  5. Score v11, compare to this doc

No part of this is allowed to revise the numbers above.

2026-06-10 ADDENDUM — Dim1 baseline was measured under the wrong harness config

The frozen 33.3% Dim1 number stands as recorded, but its provenance is now understood and it must NOT be the comparison point for the v11 verdict:

  • Score artifact ~/.cache/posttrainllm/scores/v9-LoRA-fp16-baseline-1780984389.json records no sys-prompt; score_formula.py falls back to eval_pace_v2.py defaults = v6-label prompt + v6 label schema. The 33.3% run evaluated v9 under v6’s harness.
  • Re-measured 2026-06-10 with v9’s as-shipped config (compose-v2 prompt + v9 schema + serve-side --grammar): 9/15 (60.0%), FakePace identical (1/15), same fixtures, same baked-hf. Same model — pure config drift.
  • This likely also explains the eval-matrix “v8 73.3% → 33.3% irreproducible” mystery (same default-prompt drift on re-eval). Unverified; 10-min test when GPU frees.

Rule for the v11 verdict: v11 runs with its own prompt+grammar (as-shipped), so compare it to v9-as-shipped = 60.0%, not 33.3%. Dim1 threshold ≥60% therefore means: v11 ships on Dim1 only if it beats the corrected v9 baseline. Note v9-as-shipped (60%) already ties zero-shot Qwen3-14B on this suite.

Process fix: evals must always pin –sys-prompt + –schema + serve –grammar to the model’s shipped config; score artifacts now record the sys-prompt used.

2026-06-11 FINAL VERDICT — v11 FAILS the ship gate. Planner freezes on qwen3-30b-a3b.

One run, as committed (plain LoRA r32, 709 rows, after the DoRA infra failure voided the first attempt — see #326):

Dim v11 Threshold Verdict
1 happy-path (fm-fixtures-v2) ~rules-level ≥60% FAIL
2 BFCL pace-12 27.1% (26/96) ≥40% FAIL
3 out-of-scope 0/30 ≥80% FAIL
4 ambiguous 0/20 ≥50% FAIL
6 destructive-confirm 5/10 ≥90% FAIL
nonreg compose/holdout rules-level ≥65% FAIL

Failure mode (verified in eval-dim3.json, not harness drift): the model emits clean v11 JSON but intent:"answer" with a helpful attempt on out-of-scope prompts — the exact over-eagerness the 176 OOS rows were meant to train away. Train loss 0.001 = full memorization of 709 rows with near-zero generalization of the refusal/clarify DISCIPLINE to held-out phrasings. The 0.6B memorizes behaviors; it does not learn rules. confirm_destructive (50%) generalized best — it has the most surface-regular trigger words.

Decision (per the one-run freeze): specialist planner track CLOSED. Pace ships qwen3-30b-a3b (already the wired default). The factory’s adaptation pipeline survives and moves to 4B+ bases per qlora-large-model-finetune.md — where the zero-shot baseline already clears most of these dimensions and fine-tuning starts from competence, not memorization.

2026-06-11 zero-shot candidate gate (post-v11, planner replacement)

Same gate, no training. Run artifacts: ~/.cache/posttrainllm/runs/{fm-zeroshot-guided,qwen3-4b-zeroshot}/.

Dim Threshold v11 0.6B trained Apple FM guided Qwen3-4B-2507 4bit
1 happy-path ≥60% 13% 13.3% 66.7% PASS
2 BFCL pace-12 ≥40% 27.1% 32.3% 58.3% PASS (p50 641ms)
3 out-of-scope ≥80% 0% 93.3% PASS 66.7%
4 ambiguous ≥50% 0% 5% 10%
6 destructive ≥90% 50% 60% 90.0% PASS

Notes:

  • Apple FM required guided (@Generable) output to be measured fairly — plain-text mode refused every OOS prompt CORRECTLY but scored 0 for format (scripts/fm_bridge.swift + fm_shim.py are the harness).
  • Qwen3-4B zero-shot passes 3/5 dims that eleven trained 0.6B versions never passed once. Misses OOS by 13pp; dim4 (clarify-not-guess) is the universal gap — every model guesses confidently.
  • LM Studio endpoint contention (30B JIT-load mid-suite, shared with the other agent) poisoned two partial runs with instant-failure rows (p50 ~1ms signature); affected sections re-run clean.

Planner decision: Qwen3-4B-Instruct-2507 is the fine-tune base (QLoRA phase Q3 target). Training closes dim3 (+13pp) and dim4 — starting from competence, not memorization. Apple FM is the zero-footprint refusal champion; keep as optional pre-filter/fallback on Apple Intelligence Macs. 30B stays the dev-machine reference, never the shipped default.

2026-06-11 held-out (h2) zero-shot baselines — the QLoRA before/after reference

Fresh 60-fixture suites (posttrainllm evals/fm-fixtures-{oos,ambig,destructive}-h2, zero overlap with training corpus or old fixtures, commit 95dbe2b):

Suite Threshold Qwen3-4B-2507 Apple FM (guided)
oos-h2 ≥80% 80.0% PASS 96.7%
ambig-h2 ≥50% 0% 5%
destructive-h2 ≥90% 70% 20%
  • 4B scored HIGHER on held-out OOS than on the leaked originals — no contamination advantage existed; the old suite was just a different mix. Zero-shot 4B now passes dims 1, 2, 3.
  • Clarify is the single real gap (0% across every model ever measured) — the QLoRA run’s primary target, plus destructive consistency (+20pp).
  • The Q3 fine-tune gates on THESE suites, not the leaked originals.

Was this page helpful?