B2/B2b/B3/B7 mini-router family (bundled)

status: not-started
owner: unassigned
created: 2026-06-13
parent_plan: docs/PLAN.md §3 Tier B (B2, B2b, B3, B7)
related_prds: factory-pace-fast-router-100ms.md, factory-planner-v7-tools-in-prompt.md (the existing planner-v7 + fast-router PRDs are the building blocks this PRD ties together with real BFCL data and a bake-off)

PRD — Mini-router on real BFCL data + bake-off + FSM-injection (B2-B3-B7)

Goal

Bundle the four router items from PLAN.md Tier B into one coordinated PRD because they are sequentially dependent on each other:

Each is small. Together they answer the architectural question: “is the per-task classifier head worth the deviation from one unified-model + grammar?”

Why now

• A1 ships a specialist; B2 needs that specialist to score against.
• BFCL data is on disk (D2 + A4 done); the mini-router’s BFCL-on-real- data path is just wiring the training data.
• Bake-off is the gating question. Without B2b’s result we don’t know whether to keep iterating on the classifier-head approach or to fall back to pure-FSM.

Scope — in

B2 (mini-router on real BFCL data):

• scripts/recipes/b2-router-train.sh — pulls BFCL train split, reformats as {query, tool} pairs, trains the mini-router via posttrainllm train-extractor.
• Eval the trained router via posttrainllm eval-bfcl with the router in front (intent → tool name → constrained generation).

B2b (bake-off):

• Wire the alternative: posttrainllm serve --tools <catalog> --grammar bfcl-tool.gbnf end-to-end (no router head; FSM only).
• evals/router-bakeoff.sh runs both configurations against the same BFCL eval; reports per-category + average + per-sample latency.
• Decision criterion: classifier-head wins if it beats pure-FSM by ≥ 3pp average AND per-sample latency is within 1.2× pure-FSM.
• The losing approach is parked, not deleted — kept on disk as the reference comparison for future router experiments.

B3 (FSM constraint-injection):

• If classifier wins, Sources/TinyGPTServe/RouterFsmInject.swift — the router’s top-1 tool name selects which sub-grammar the FSM uses for the rest of the response.
• If FSM wins, skip B3; the unified grammar already does this.

B7 (specialist routing):

• Once ≥ 2 specialists ship (A1 + B1), train a 2nd-level router that picks among them based on intent. Reuses the mini-router infra, different label set.
• Requires B22 trajectory recorder for the training data.

Scope — out

• Production multi-specialist routing in serve — that’s downstream; this PRD validates the architecture, doesn’t ship the runtime.
• Cost-aware routing (cheaper specialist for easy queries, expensive for hard). V1 routes by accuracy; cost-aware is V2.

Files to touch

Acceptance criteria

• B2: trained mini-router on BFCL train split; eval accuracy ≥ 70% on the BFCL test split.
• B2b: side-by-side numbers published; decision recorded in b2-router-bakeoff.md with reasoning.
• B3: only if B2b winner = classifier — FSM-injected router ships and gets a BFCL row in the leaderboard.
• B7: post-A1+B1, the specialist router ships and routes with ≥ 80% accuracy on a held-out “which specialist should answer this?” test set.

Reference patterns

• factory-pace-fast-router-100ms.md — existing fast-router PRD, contains the V0 architecture.
• factory-planner-v7-tools-in-prompt.md — the unified-grammar alternative we’re comparing against.
• Sources/TinyGPT/TrainExtractor.swift — the existing trainer (B2 = extend its data path; B7 = extend its label space).

Open questions

• Whether B2b uses a fixed sampling seed (deterministic bake-off) or repeated K=3 (per B23 protocol). Recommendation: K=3. Bake-off conclusions deserve the rigor.