Planner v7 — tools-in-prompt (generalizable function calling)

status: implementation-v0-2026-06-07-long-sft-and-external-evals-pending
owner: unassigned (parallel-agent task — Swift training + serve + Python eval)
created: 2026-06-08
priority: P1 — unblocks Pace tool churn without retrain; foundation for any multi-app planner consumer
depends-on: v6 ANE arc landing (#263), BFCL eval shipping (#231)
authorized-by: maintainer 2026-06-08 (architectural direction confirmed — implementation gated on owner go-ahead after ANE arc completes)

PRD — Planner v7: tools-in-prompt architecture

Ship note — 2026-06-07

Owner approved the previously gated v7 work. The first implementation slice landed:

• posttrainllm serve --tools <tools.json> injects the tool catalog into the ChatML system prompt
• --tools also installs a default constrained output schema shaped as { "verb": <tool-name-enum>, "args": {}, "spoken_text": "" }
• The larger tools system prompt is included in prompt-cache prefixing
• Added grammars/pace-system-prompt-v7-tools.txt
• Added data/eval scaffolding:
  • scripts/archive/v7-data/normalize-xlam.py
  • scripts/archive/v7-data/build-pace-topup.py
  • scripts/archive/v7-data/build-heldout.py
  • scripts/archive/v7-eval/run-heldout.py
  • scripts/archive/v7-eval/run-bfcl.py
  • scripts/archive/v7-eval/run-tau-bench.py
  • scripts/archive/v7-eval/heldout-tools.jsonl
• Generated local Pace top-up data at ~/.cache/posttrainllm/datasets/pace-v7-topup.jsonl

Known v0 limitation: the current posttrainllm JSON-Schema FSM does not support oneOf / conditional schemas, so v0 constrains the verb enum but does not yet enforce per-verb argument schemas. That remains the next grammar milestone before claiming full v7 acceptance. The 6-8 hour xLAM-scale SFT and external BFCL / tau-bench runs are still pending.

Why this PRD exists

v6’s grammar hardcodes Pace’s action set. Adding a new action type (drag-and-drop, multi-select, keyboard shortcut sequences, novel app integrations) requires re-training. That’s manageable when Pace evolves slowly, but the moment Pace gains 5+ new tool types in a sprint, or a second posttrainllm consumer wants its own tool set, the v6 architecture becomes a bottleneck.

v7 inverts the design: tools live in the system prompt at inference time, not in training data. The model learns the pattern of “fill this schema correctly,” not the specific tool list. Adding tools = one prompt line, no retrain. Matches OpenAI function calling, Anthropic tool use, and MCP.

This PRD is a draft. The case for shipping v7 NOW vs deferring is genuinely close (see “Cost-benefit” below). The owner should weigh deferring until Pace tool churn becomes real pain.

Goal

Train a Qwen3-0.6B (or xLAM-1B — see leverage choice below) specialist that:

1. Takes a system prompt containing a list of available tool schemas
2. Takes a user intent (text or transcribed voice)
3. Emits a structured tool call: which tool, with which arguments
4. Generalizes to tools NOT seen during training, given only their schemas in the prompt
5. Drops into Pace via posttrainllm serve --tools <tools.json> — and into any future posttrainllm consumer the same way

Architectural shift — what changes from v6

Leverage-first choices (per [[feedback_leverage_first]])

Base model: xLAM-1B-fc-r (Salesforce, Apache 2.0). Already trained on function calling at scale; we LoRA-fine-tune for Pace specifics rather than teaching function-calling from scratch on bare Qwen3-0.6B. Saves weeks of SFT and lifts the floor.

Training data: xlam-function-calling-60k (already on the queue as task #227 — D1 pending). Diverse synthetic function calls. Use as warmstart. Pace-specific top-up (~500 examples) for the long tail.

Schema patterns: MCP tool specification + Anthropic tool use + Apple App Intents. Don’t invent a new schema; align with the three industry references. (App Intents study is its own PRD — factory-app-intents-taxonomy-study.md.)

App Intents study landed: see docs/learn/app-intents-comparison.md. Before v7 SFT lock-in, replace the draft taxonomy below with the study’s recommended set: query, perform, set, compose, clarify, say, query_memory, wait, open, schedule. Also add a shared EntityRef schema, fold top-level search into query(mode=search), rename navigate to open, and split broad act examples into perform and set.

Action collapse plan — many actions, few verbs

The model should not learn one verb per surface action. Pace can execute a large and growing action set — click, double click, type, key chord, scroll, drag, multi-select, open app, open file, set value, search, summarize, clarify, wait, schedule — but the planner should choose from a small stable verb set and put the variability inside typed args.

The core rule:

{
  "verb": "perform",
  "args": {
    "action": "click",
    "target": {"type": "ui_element", "label": "save draft button"}
  },
  "spoken_text": "opening save draft"
}

click, type, key, scroll, drag, multi_select, and similar UI operations are actions under perform, not top-level verbs. This keeps the model’s classification problem small while keeping the executor expressive.

Use specialized top-level verbs only when they imply a different executor, latency model, safety policy, or product behavior:

This is the bridge from v6 to v7:

• v6 legacy tags ([CLICK:x,y], [TYPE:text], [KEY:cmd+s]) remain an executor compatibility layer.
• v7 supervision should prefer structured calls such as perform(action=type, text=...); tags can be rendered downstream for old Pace clients.
• Dynamic schemas should enum-constrain the things visible in the current request: verb, perform.action, target labels, key names, scroll directions, and available app/file/entity refs.
• Adding a new primitive action should usually mean adding a new perform.action enum value and executor handler, not adding a new planner verb or retraining the whole model.
• Add a top-level verb only after the executor semantics diverge from perform enough that the model benefits from a separate bucket.

Eval: BFCL (Berkeley Function Calling Leaderboard, #231) + τ-bench (#232) + held-out-tools test set (our own).

Verb taxonomy (the architectural decision)

The taxonomy IS the design. Get this wrong and v7 fails to generalize. Proposed taxonomy:

5 core verbs (cover ~90% of intents)

5 specialized verbs (carve out edge cases that don’t fit cleanly)

Why this carving

• Bounded vocabulary: model learns 10 verbs deeply, not 500 endpoints shallowly. Sample-efficient.
• Apps as values, not tools: adding Bluesky = enum value under act.target, not a new tool.
• Composable: model can chain (read → compose → act) for multi-step task planning.
• MCP-shaped: each verb maps cleanly to MCP tool definitions.
• App Intents-aligned: Apple’s verb taxonomy is similar (read the App Intents study before final taxonomy lock).

Held-out verbs for eval

Train on verbs A/B/C/D, eval on verb E (schema-only in prompt). If the model calls E correctly with no training examples, generalization is real. Pick compose or recall as the held-out — they’re the most semantically distinct from the action-oriented verbs.

System prompt structure

You have these tools available. Use them to fulfill the user's intent.

Tools:
[
  {
    "name": "read",
    "description": "Pull information from a known source.",
    "schema": {
      "source": {"enum": ["calendar", "email", "files", "current_screen"]},
      "query": {"type": "string", "optional": true},
      "filter": {"type": "object", "optional": true}
    }
  },
  ...
]

Emit a JSON object with:
  {
    "verb": "<verb_name>",
    "args": {...},
    "spoken_text": "<what to say to the user, optional>"
  }

The full tools list (~10 verbs × schemas) comes to roughly 1500 tokens. Prompt cache (#260, already shipped) means this is parsed once per process, not per request.

Grammar generation at request time

v6 uses a static GBNF. v7 needs dynamic GBNF generated from the tools list passed at request time. Implementation:

1. serve accepts --tools <tools.json> flag (or tools field in request body for OpenAI-compat)
2. On request, generate a GBNF that enforces:
   • verb is one of the tool names
   • args matches the schema of the selected verb (mutually exclusive: if verb=read, args must match read’s schema)
3. Cache the generated GBNF by (tools.json hash) — same tools list reuses the same compiled grammar
4. JSON Schema → GBNF is a known transformation (we already do part of this for v6); needs extension to handle the verb-conditional args.

This is non-trivial but not novel — llama.cpp’s grammar generator already handles JSON Schema. Borrow from there.

Training data plan

Stage A — public function-call warmstart:

• xlam-function-calling-60k (Salesforce, Apache 2.0) — diverse synthetic function calls. ~60K (intent, tools, call) triples.
• Format-normalize to our verb schema (one-off Python script).

Stage B — Pace-specific top-up:

• ~500-1000 hand-crafted (intent, Pace_tools, correct_call) triples
• Cover Pace’s actual tool set across all 10 verbs
• Include ambiguity / clarification cases for ask
• Include held-out verb’s examples (to verify generalization, NOT to train on it)

Stage C — adversarial / generalization examples:

• (intent, tools_with_one_unfamiliar_verb, correct_call) — model should use the unfamiliar verb correctly from schema alone
• (intent, tools_missing_obvious_verb, ask_for_help_call) — model should emit ask instead of fabricating

Total: ~62K training examples. Roughly 4× v6’s data set. SFT budget: ~6-8 hours on M5 Pro with current pipeline.

Eval plan

Three layers:

1. BFCL (#231 — ship this before v7 SFT starts) — objective number vs xLAM, GPT-4-mini, Claude Haiku, others.
2. τ-bench (#232) — multi-turn agent eval, more realistic than one-shot function calling.
3. Held-out tools (our own) — 50 (intent, tools_with_held_out_verb, gold_call) triples we craft ourselves. The verb is in the prompt but NOT in training. Generalization check.

Target: BFCL within 5% of xLAM-7B (we’ll be 7× smaller), τ-bench within 10% of teacher, held-out >70% accuracy.

Cost-benefit (the “should we even do this” check)

Cost: ~2-3 weeks of work (data prep + SFT + grammar generator + serve wire-in + eval).

Benefit: every future Pace tool change costs zero retrain. Every non-Pace posttrainllm consumer gets a planner that works for their tools.

When v7 pays off: when Pace gains 5+ new tool types, OR when a second consumer shows up, OR when the user wants to expose 3rd-party MCP servers to Pace dynamically.

When v7 is premature: if Pace’s tool set stays stable for 3+ months, the ~40 min × N retrains for v6 stays cheaper than 2-3 weeks of v7 work.

Recommendation: hold off on v7 SFT until ANE arc lands and v6 has been daily-driven for ~2 weeks. If Pace’s tool set churns visibly in that window, ship v7. If it’s stable, v6 keeps winning on specialization-for-known-scope.

Migration from v6

v7 output schema is a superset of v6’s. The click executor PRD in clickyLocal (docs/prds/click-executor-improvements.md) already plans for this — it ships schema-tolerant code that accepts both v6’s single-label output and v7’s tool-call output.

When v7 lands:

1. Pace’s LocalPlannerClient schema decode already handles both
2. Pace’s system prompt template gains a tools: section
3. The planner endpoint URL stays the same
4. Roll back to v6 if v7 regresses (binary swap)

Scope — in

• Stage A/B/C training data pipeline
• LoRA SFT on chosen base (xLAM-1B preferred per leverage)
• Dynamic GBNF generator (JSON Schema → grammar)
• serve --tools <tools.json> flag
• BFCL + τ-bench + held-out-tools eval
• Migration smoke test (Pace runs against v7 endpoint)

Scope — out

• Multi-tool calls in one response (model emits ONE call; orchestrator chains them)
• Streaming tool calls (single output, streamed token-by-token is fine)
• Tool description in non-English (English schemas only for v1)
• Tool versioning (caller assumes the tools list is current truth)
• Multi-LoRA for v6→v7 hot-swap (separate concern)

Files involved

New:

• scripts/archive/v7-data/normalize-xlam.py — convert xlam-60k to our verb schema
• scripts/archive/v7-data/build-pace-topup.py — hand-curate Stage B
• scripts/archive/v7-data/build-heldout.py — generate Stage C
• native-mac/Sources/TinyGPTServe/DynamicGrammar.swift — JSON Schema → GBNF
• scripts/archive/v7-eval/run-bfcl.py — wrap BFCL harness for v7 model
• scripts/archive/v7-eval/run-tau-bench.py — wrap τ-bench
• scripts/archive/v7-eval/heldout-tools.jsonl — our held-out eval set

Modified:

• native-mac/Sources/TinyGPTServe/Serve.swift — add --tools flag, per-request grammar generation
• grammars/pace-system-prompt-v7-tools.txt — new v7 system prompt template

Don’t touch:

• v6 LoRA artifacts (parallel ship, v6 keeps running)
• ANE arc (separate moat)
• VLM arc (separate moat)

Acceptance

1. xlam-60k normalized to our verb schema; format-validated
2. SFT completes; LoRA artifact written
3. posttrainllm serve --tools tools.json loads and accepts requests
4. BFCL: within 5% of xLAM-7B (subject to held-out result; if too ambitious, target xLAM-1B parity)
5. Held-out-tools eval: ≥70% correct call on tools never seen in training
6. Pace smoke: v7 endpoint serves Pace’s actual workload without regression in fm-fixture pass rate vs v6
7. Round-trip latency: ≤1.5× v6 (the longer system prompt costs some prefill; prompt cache mitigates)

Why this matters — the moat argument

v6 makes Pace specialist-grade. v7 makes the posttrainllm planner factory generalizable. Pace becomes the first consumer; the platform is then usable for any future Mac-native voice-companion / agent that wants a small fast planner. That’s the difference between “posttrainllm trained one good model for one app” and “posttrainllm is the way you ship local planners on Mac.”

v7 is what turns a specialist into a platform.