B1 second specialist (SQL POC)

status: expanded-qwen06-poc-complete; retry-data
owner: unassigned
created: 2026-06-13
parent_plan: docs/PLAN.md §3 Tier B (B1)
related_prds: A1-first-specialist-tool-caller.md (the template; B1 cookie-cuts off it)

PRD — Second specialist after A1 ships

Goal

Run the factory recipe shape against SQL to validate that the platform is not accidentally tied to a tool-calling target. Same factory loop, different data and eval.

Two specialists ship the platform thesis: “you can do this for any task” is much stronger than “we did this for one task.”

Why now

• The active factory cleanup selected SQL as the first low-compute POC because posttrainllm eval-sql is already self-contained and smoke-tested.
• The initial fixture is deliberately tiny and deterministic. The first live Qwen3-0.6B runs proved the loop but were marked retry-data; Spider remains the later public benchmark once preference tuning is tested.

Scope — in

• Freeze the POC fixture in evals/sql-poc/.
• Use scripts/recipes/b1-sql.sh for the live SFT/generate/eval path.
• Score with posttrainllm eval-sql.
• Write row-level failure traces from eval-sql --out.
• Ship/retry gate: candidate execution accuracy ≥ baseline + 3pp.
• Artifact: adapters/b1-sql.lora or specialists/b1-sql/.

Scope — out

• Combining A1 + B1 into a single multi-task adapter. Distinct adapters keep the experiment clean; merging is its own arc.
• Cross-domain transfer ablation. Useful research; defer.

Files to touch

Acceptance criteria

• Low-compute SQL fixture exists and is smoke-tested.
• Row-level eval traces capture SQL failures.
• Live baseline score recorded from Qwen3-0.6B on the toy SQL eval.
• First SQL adapter beats base by ≥ 3pp on the toy SQL eval.
• Expanded non-overlapping synthetic SQL set generated and evaluated.
• Failure labels and DPO-style preference rows emitted from the expanded candidate failures.
• Factory run folder contains config, dataset, train log, baseline/candidate evals, row traces, report, artifact metadata, and decision.
• Run preference tuning or move to a public benchmark slice before any ship claim.

Reference patterns

• docs/prds/A1-first-specialist-tool-caller.md — the template. This PRD is the second instance; if the recipe doesn’t generalize easily, that’s the finding.
• InterCode-Bash — the shell-side eval scaffolding.
• Spider — the SQL-side eval scaffolding.

Open questions

• Which public benchmark replaces the toy fixture after the POC: Spider is the default unless another SQLite-backed text-to-SQL set is lower friction.