--- title: "A1 first specialist end-to-end (tool-caller)" --- **status:** not-started **owner:** unassigned **created:** 2026-06-13 **parent_plan:** docs/PLAN.md §3 Tier A (A1) **related_prds:** E1-bfcl-eval.md (gates the score), E2-tau-bench-eval.md (multi-turn score), E8-train-time-eval-hook.md (in-loop scoring), B23-agent-eval-protocol.md (eval rigor), factory-pace-planner-v6_1.md (Pace planner companion; this PRD is the more general tool-caller north-star), B22-trajectory-recorder.md (substrate for off-policy refinement) # PRD — Train the first specialist end-to-end (tool-caller) ## Goal Ship one trained specialist that beats Pace's current 0-shot floor (`qwen3-4b-instruct-2507`) on BFCL average by ≥ 3pp on a Mac-runnable artifact (≤ 8 GB on disk). Validates the platform's north-star thesis: "individuals build and upgrade specialist models for their specific tasks on a Mac, with measurable lift over the same model 0-shot." This is the integration PRD — every Tier-A item (datasets, evals, mini-router, FSM, training recipe) lands as inputs here; this PRD ships the executable recipe + the trained adapter + the leaderboard row. ## Why now - All five Tier E eval harnesses are shipped (E1 BFCL, E2 τ-bench, E5 HumanEval, E7 judge, E8 train-time eval-hook). Eval surfaces no longer block training. - Datasets are pulled (D-tier work mostly complete; xlam still gated but covered by D1's HF_TOKEN path). - The Mac-specialist platform thesis (PLAN.md §"Product framing") needs proof-by-existence — one working specialist that demonstrably beats its base on a published benchmark. - Until A1 lands, every other Tier-B optimization (B10, B11, B14, etc.) is theoretical lift on top of an unbuilt baseline. A1 unblocks the rest. ## Scope — in A single recipe-shaped artifact: - **Base model:** `qwen3-4b-instruct-2507` (current Pace floor; Mac-runnable; the artifact users already have). - **Adapter:** LoRA (rank-32) trained via `posttrainllm sft` on the combined dataset below. DoRA is the default PEFT but LoRA gives comparable quality with strictly simpler on-disk handling; revisit once C3 DoRA-on-disk ships. - **Training data (already on disk per docs/dataset_inventory.md):** - `hermes-function-calling-v1` (~10K) - `function-calling-chatml` (public path; per D2) - `xlam-function-calling-60k` (gated; ship without if HF_TOKEN is missing — recipe degrades gracefully) - Distillation-from-teacher pairs generated by `posttrainllm magpie` targeting tool-call distributions (~5K, generated lazily). - **Training recipe:** `scripts/recipes/a1-tool-caller.sh` — sequence packing on, NEFTune α=5, sft 3 epochs, LR cosine 5e-5 → 5e-7 (will switch to WSD once B11 lands), checkpoint every 500 steps with `--eval-every 500 --eval-tasks bfcl,humaneval --eval-limit 50`. - **Validation:** `posttrainllm eval-bfcl` against the adapter on the full BFCL 10-category suite + the τ-bench retail env. - **Ship gate:** BFCL avg of (base + adapter) ≥ BFCL avg of base alone + 3pp. Reject otherwise — back to data + recipe debugging. - **Artifact:** `adapters/a1-tool-caller.lora` (lives in the existing CF R2 bucket via the cloud-save pipeline; PR adds a `posttrainllm pull a1-tool-caller` example). - **Documentation:** `docs/specialists/a1-tool-caller.md` — motivation + recipe + before/after numbers + caveats. - **PLAN.md:** A1 ⬜ → ✅ on ship; row in `docs/research/mac_slm_leaderboard_v0.md` as `qwen3-4b-instruct-2507+a1-lora` so the leaderboard shows the lift. ## Scope — out - **Pretraining a tool-caller from scratch.** Adapter on a strong base wins on Mac compute budget. From-scratch is a Tier-5 arc. - **Mini-router head** (B2/B2b/B3) — those are *competing* designs for tool routing. A1 ships the simpler adapter; the router-vs-FSM bake-off is its own PRD. - **Cloud-escalate training signal** (B5) — V1 specialist routes everything locally; learning when to defer to cloud is a follow-up. - **DPO refinement** — V1 is SFT only. DPO on top is V2. - **τ-bench airline + the harder BFCL categories** beyond the V1 gate. Document those scores; don't gate on them. ## Files to touch (recipe + artifact, mostly) | File | Change | |---|---| | `scripts/recipes/a1-tool-caller.sh` | new — one-command recipe (data prep, SFT, eval, package) | | `docs/specialists/a1-tool-caller.md` | new — the user-facing brief | | `docs/dataset_inventory.md` | new (or updated) — list datasets used, sizes, licenses | | `docs/research/mac_slm_leaderboard_v0.md` | regenerate via `build_slm_leaderboard.py` after A1 evals | | `docs/PLAN.md` | A1 ⬜ → ✅ + the delta vs the 0-shot floor | | `evals/a1-acceptance.sh` | new — re-runs the gate check on a fresh checkout | | `../archive/HANDOFF.md` | bumped to "A1 shipped" milestone | ## Don't touch - The base model. No upstream changes; this is purely an adapter. - The Pace planner (`pace-system-prompt-v11.txt`) — A1 is the tool-caller north-star, separate from Pace's planner-routing role. ## Acceptance criteria - [ ] `bash scripts/recipes/a1-tool-caller.sh` runs end-to-end on M5 Pro in < 24h wall-clock. - [ ] `posttrainllm eval-bfcl qwen3-4b-instruct-2507 --adapter adapters/a1-tool-caller.lora` scores ≥ +3pp average vs the same base without the adapter. - [ ] `posttrainllm eval-tau-bench qwen3-4b-instruct-2507 --adapter ... --env retail` scores ≥ base on retail (no regression on the multi-turn axis; +pp is the stretch). - [ ] B23 protocol-hardened: gate uses K=3 passes, mean reported with ci95. - [ ] Leaderboard row published; the new "rank 1" or "rank 2" entry is the A1 specialist. - [ ] `docs/specialists/a1-tool-caller.md` exists, ≤ 300 lines, cites external sources for the SFT recipe rather than re-explaining. - [ ] `evals/a1-acceptance.sh` re-runs the gate from a clean checkout given a pre-trained adapter (i.e., reproducibility check, not training-from-scratch in CI). ## Reference patterns - `factory-character-specialist-recipe.md` — the existing specialist-recipe template; same shape, different domain. - `Sources/TinyGPT/SFT.swift` — the trainer is shipped. A1 is recipe + data + eval, not new tooling. - [BFCL v3 leaderboard](https://gorilla.cs.berkeley.edu/leaderboard.html) — the external scoreboard our V1 score is benchmarked against. We don't need to submit; the local leaderboard renders the delta. - [xLAM paper, Liu et al. 2024](https://arxiv.org/abs/2406.18518) — the tool-call training-data recipe. Cite, don't redocument. ## Open questions - Whether to attempt the gated xLAM data or ship without it. **Recommendation:** ship without on first cut; xLAM access is user-config dependent, not platform-blocking. Re-train when available. - Whether to also produce a posttrainllm-native (`.tinygpt`-base) specialist instead of (or alongside) the qwen3 LoRA. **Recommendation:** ship qwen3+LoRA first — it's the artifact users actually consume. Native-base is V2.