A1 first specialist end-to-end (tool-caller)

status: not-started
owner: unassigned
created: 2026-06-13
parent_plan: docs/PLAN.md §3 Tier A (A1)
related_prds: E1-bfcl-eval.md (gates the score), E2-tau-bench-eval.md (multi-turn score), E8-train-time-eval-hook.md (in-loop scoring), B23-agent-eval-protocol.md (eval rigor), factory-pace-planner-v6_1.md (Pace planner companion; this PRD is the more general tool-caller north-star), B22-trajectory-recorder.md (substrate for off-policy refinement)

PRD — Train the first specialist end-to-end (tool-caller)

Goal

Ship one trained specialist that beats Pace’s current 0-shot floor (qwen3-4b-instruct-2507) on BFCL average by ≥ 3pp on a Mac-runnable artifact (≤ 8 GB on disk). Validates the platform’s north-star thesis: “individuals build and upgrade specialist models for their specific tasks on a Mac, with measurable lift over the same model 0-shot.”

This is the integration PRD — every Tier-A item (datasets, evals, mini-router, FSM, training recipe) lands as inputs here; this PRD ships the executable recipe + the trained adapter + the leaderboard row.

Why now

• All five Tier E eval harnesses are shipped (E1 BFCL, E2 τ-bench, E5 HumanEval, E7 judge, E8 train-time eval-hook). Eval surfaces no longer block training.
• Datasets are pulled (D-tier work mostly complete; xlam still gated but covered by D1’s HF_TOKEN path).
• The Mac-specialist platform thesis (PLAN.md §“Product framing”) needs proof-by-existence — one working specialist that demonstrably beats its base on a published benchmark.
• Until A1 lands, every other Tier-B optimization (B10, B11, B14, etc.) is theoretical lift on top of an unbuilt baseline. A1 unblocks the rest.

Scope — in

A single recipe-shaped artifact:

• Base model: qwen3-4b-instruct-2507 (current Pace floor; Mac-runnable; the artifact users already have).
• Adapter: LoRA (rank-32) trained via posttrainllm sft on the combined dataset below. DoRA is the default PEFT but LoRA gives comparable quality with strictly simpler on-disk handling; revisit once C3 DoRA-on-disk ships.
• Training data (already on disk per docs/dataset_inventory.md):
  • hermes-function-calling-v1 (~10K)
  • function-calling-chatml (public path; per D2)
  • xlam-function-calling-60k (gated; ship without if HF_TOKEN is missing — recipe degrades gracefully)
  • Distillation-from-teacher pairs generated by posttrainllm magpie targeting tool-call distributions (~5K, generated lazily).
• Training recipe: scripts/recipes/a1-tool-caller.sh — sequence packing on, NEFTune α=5, sft 3 epochs, LR cosine 5e-5 → 5e-7 (will switch to WSD once B11 lands), checkpoint every 500 steps with --eval-every 500 --eval-tasks bfcl,humaneval --eval-limit 50.
• Validation: posttrainllm eval-bfcl against the adapter on the full BFCL 10-category suite + the τ-bench retail env.
• Ship gate: BFCL avg of (base + adapter) ≥ BFCL avg of base alone + 3pp. Reject otherwise — back to data + recipe debugging.
• Artifact: adapters/a1-tool-caller.lora (lives in the existing CF R2 bucket via the cloud-save pipeline; PR adds a posttrainllm pull a1-tool-caller example).
• Documentation: docs/specialists/a1-tool-caller.md — motivation + recipe + before/after numbers + caveats.
• PLAN.md: A1 ⬜ → ✅ on ship; row in docs/research/mac_slm_leaderboard_v0.md as qwen3-4b-instruct-2507+a1-lora so the leaderboard shows the lift.

Scope — out

• Pretraining a tool-caller from scratch. Adapter on a strong base wins on Mac compute budget. From-scratch is a Tier-5 arc.
• Mini-router head (B2/B2b/B3) — those are competing designs for tool routing. A1 ships the simpler adapter; the router-vs-FSM bake-off is its own PRD.
• Cloud-escalate training signal (B5) — V1 specialist routes everything locally; learning when to defer to cloud is a follow-up.
• DPO refinement — V1 is SFT only. DPO on top is V2.
• τ-bench airline + the harder BFCL categories beyond the V1 gate. Document those scores; don’t gate on them.

Files to touch (recipe + artifact, mostly)

Don’t touch

• The base model. No upstream changes; this is purely an adapter.
• The Pace planner (pace-system-prompt-v11.txt) — A1 is the tool-caller north-star, separate from Pace’s planner-routing role.

Acceptance criteria

• bash scripts/recipes/a1-tool-caller.sh runs end-to-end on M5 Pro in < 24h wall-clock.
• posttrainllm eval-bfcl qwen3-4b-instruct-2507 --adapter adapters/a1-tool-caller.lora scores ≥ +3pp average vs the same base without the adapter.
• posttrainllm eval-tau-bench qwen3-4b-instruct-2507 --adapter ... --env retail scores ≥ base on retail (no regression on the multi-turn axis; +pp is the stretch).
• B23 protocol-hardened: gate uses K=3 passes, mean reported with ci95.
• Leaderboard row published; the new “rank 1” or “rank 2” entry is the A1 specialist.
• docs/specialists/a1-tool-caller.md exists, ≤ 300 lines, cites external sources for the SFT recipe rather than re-explaining.
• evals/a1-acceptance.sh re-runs the gate from a clean checkout given a pre-trained adapter (i.e., reproducibility check, not training-from-scratch in CI).

Reference patterns

• factory-character-specialist-recipe.md — the existing specialist-recipe template; same shape, different domain.
• Sources/TinyGPT/SFT.swift — the trainer is shipped. A1 is recipe
  • data + eval, not new tooling.
• BFCL v3 leaderboard — the external scoreboard our V1 score is benchmarked against. We don’t need to submit; the local leaderboard renders the delta.
• xLAM paper, Liu et al. 2024 — the tool-call training-data recipe. Cite, don’t redocument.

Open questions

• Whether to attempt the gated xLAM data or ship without it. Recommendation: ship without on first cut; xLAM access is user-config dependent, not platform-blocking. Re-train when available.
• Whether to also produce a posttrainllm-native (.tinygpt-base) specialist instead of (or alongside) the qwen3 LoRA. Recommendation: ship qwen3+LoRA first — it’s the artifact users actually consume. Native-base is V2.