# Fine-Tune Report Card — sql-routed-specialist-poc — routed-sft-lora > Retry — eval: The routed setup is the current best SQL candidate and passes the current public exact plus synthetic execution gates, but it is not a shipped… Canonical: https://posttrainllm.com/report-cards/qwen06-sql-routed-v1 PostTrainLLM fine-tune report card · schema v1 ## Subject **Artifact:** **Artifact id:** qwen06-sql-routed-v1; **Kind:** report-routed-adapter; **Package dir:** Not recorded; **Path:** docs/factory/public-artifacts.md#qwen06-sql-routed-v1; **Routing constraint:** Not recorded — Evidence state: missing — The run folder records no routing constraint for this artifact. — Sources: artifact.json#routing_constraint; **Shipped:** false **Base model:** Qwen/Qwen3-0.6B (bf16, c1899de289a04d12100db370d81485cdf75e47ca) — Evidence state: measured — Sources: config.json#base_model **Candidate model:** qwen06-sql-routed-v1 — Evidence state: measured — Sources: eval-candidate.json#model_id **Method:** routed-sft-lora — Evidence state: measured — Sources: config.json#candidate.method **Owner goal:** Publish the current SQL routed POC as a transparent factory report artifact with measured blockers before packaging. — Evidence state: measured — Sources: config.json#owner_goal **Target:** sql-routed-specialist-poc — Evidence state: measured — Sources: config.json#target ## Decision **Blocked by:** - public execution DB bundle not local - latency/RAM/tok-s not measured - clean-SQL output hygiene gate missing **Decision:** retry-eval **Evidence sources:** - docs/factory/public-artifacts.md - docs/specialists/b1-sql-poc.md - runs/2026-07-02-sql-routed-qwen06-v1/report.md **Failure reason:** The routed artifact lacks public execution benchmarking, performance measurements, and a clean-SQL package gate. **Failure reason confidence:** exact **Lesson:** Routing preserves both measured SQL slices, but artifact status still depends on public execution and performance evidence. **Next action:** Run scripts/sql/build_sql_spider_execution_gate.py against a local Spider DB bundle, score the routed candidate, then re-render this report with latency/RAM/tok-s. — Evidence state: measured — Sources: decision.json#next_action **Outcome label:** report-only **Reason:** The routed setup is the current best SQL candidate and passes the current public exact plus synthetic execution gates, but it is not a shipped specialist until a public execution benchmark and performance measurements exist. **Verification blockers:** - Decision is `retry-eval`, not `ship`; verification applies to ship claims only. - Primary gate `sql-poc-expanded-synthetic-execution` has no frontier-ceiling evidence, so the eval is unverified as a ruler. - Train/eval overlap (leakage) was not checked with current evidence. - Open blocker: public execution DB bundle not local - Open blocker: latency/RAM/tok-s not measured - Open blocker: clean-SQL output hygiene gate missing **Verified:** false ## Before and after - **Baseline:** 0.16 — Evidence state: measured — Sources: eval-baseline.json#score; **Candidate:** 0.86 — Evidence state: measured — Sources: eval-candidate.json#score; **Delta:** 0.7 — Evidence state: derived — Derived from: - eval-baseline.json#score - eval-candidate.json#score — Sources: eval-baseline.json#score, eval-candidate.json#score; **Eval identity:** **Command:** scripts/sql/run_sql_routed_generate.py + eval-sql/score_sql_public_exact — Evidence state: measured — Sources: eval-candidate.json#command; **Date:** 2026-07-02 — Evidence state: measured — Sources: eval-candidate.json#date; **Frozen:** locked public64 + synthetic50 — Evidence state: measured — Identity comes from the dataset manifest's locked-split label; no eval-validity.json records a hashed frozen-eval identity. — Sources: dataset.json#processing.heldout_split; **Suite:** sql-poc-expanded-synthetic-execution; **Frontier ceiling:** Not recorded — Evidence state: missing — No frontier-ceiling score is recorded for this benchmark. Per docs/factory/eval-protocol.md an unvalidated benchmark cannot certify a verified ship. — Sources: eval-validity.json#frontier.by_suite.sql-poc-expanded-synthetic-execution; **Metric:** sql-poc-expanded-synthetic-execution; **Name:** sql-poc-expanded-synthetic-execution; **Passed:** true — Evidence state: measured — Sources: eval-candidate.json#passed; **Role:** primary; **Sample size:** 50 — Evidence state: measured — Sample size from slice `synthetic_sqlite_execution`, named by config.eval.primary_slice. — Sources: slice-metrics.json#slices.synthetic_sqlite_execution.rows; **Threshold:** 0.86 — Evidence state: measured — Sources: config.json#eval.threshold.primary_min - **Baseline:** 0.484 — Evidence state: measured — From slice `public_bmc2_exact`, named by config.eval.regression_slice. — Sources: slice-metrics.json#slices.public_bmc2_exact.baseline; **Candidate:** 0.531 — Evidence state: measured — From slice `public_bmc2_exact`, named by config.eval.regression_slice. — Sources: slice-metrics.json#slices.public_bmc2_exact.candidate; **Delta:** 0.047 — Evidence state: derived — Derived from: - slice-metrics.json#slices.public_bmc2_exact.baseline - slice-metrics.json#slices.public_bmc2_exact.candidate — Sources: slice-metrics.json#slices.public_bmc2_exact.baseline, slice-metrics.json#slices.public_bmc2_exact.candidate; **Eval identity:** **Command:** Not recorded — Evidence state: missing — No eval command is recorded for this gate. — Sources: eval-candidate.json#command; **Date:** 2026-07-02 — Evidence state: measured — Sources: eval-candidate.json#date; **Frozen:** Not recorded — Evidence state: missing — The run folder does not record whether the regression suite is frozen. — Sources: dataset.json; **Suite:** sql-public-bmc2-exact-and-router-smoke; **Frontier ceiling:** Not recorded — Evidence state: missing — No frontier-ceiling score is recorded for this benchmark. Per docs/factory/eval-protocol.md an unvalidated benchmark cannot certify a verified ship. — Sources: eval-validity.json#frontier.by_suite.sql-public-bmc2-exact-and-router-smoke; **Metric:** normalized_exact_match; **Name:** sql-public-bmc2-exact-and-router-smoke; **Passed:** true — Evidence state: measured — Sources: slice-metrics.json#slices.public_bmc2_exact.pass; **Role:** regression; **Sample size:** 64 — Evidence state: measured — Sources: slice-metrics.json#slices.public_bmc2_exact.rows; **Threshold:** 0 — Unit: pp max drop — Evidence state: measured — Sources: config.json#eval.threshold.breadth_drop_max_pp ## Per-slice evidence - **Baseline:** Not recorded — Evidence state: missing — This slice records no baseline score. — Sources: slice-metrics.json#slices.known_hard_slice_join.baseline; **Candidate:** 0.6875 — Evidence state: measured — Sources: slice-metrics.json#slices.known_hard_slice_join.candidate; **Delta:** Not recorded — Evidence state: missing — Delta cannot be derived: baseline has no recorded value. No change is implied. — Sources: slice-metrics.json#slices.known_hard_slice_join.baseline, slice-metrics.json#slices.known_hard_slice_join.candidate; **Metric:** synthetic_execution_accuracy; **Name:** known_hard_slice_join; **Passed:** Not recorded — Evidence state: missing — This slice records no gate result. — Sources: slice-metrics.json#slices.known_hard_slice_join.pass; **Sample size:** 16 — Evidence state: measured — Sources: slice-metrics.json#slices.known_hard_slice_join.rows - **Baseline:** 0.484 — Evidence state: measured — Sources: slice-metrics.json#slices.public_bmc2_exact.baseline; **Candidate:** 0.531 — Evidence state: measured — Sources: slice-metrics.json#slices.public_bmc2_exact.candidate; **Delta:** 0.047 — Evidence state: derived — Derived from: - slice-metrics.json#slices.public_bmc2_exact.baseline - slice-metrics.json#slices.public_bmc2_exact.candidate — Sources: slice-metrics.json#slices.public_bmc2_exact.baseline, slice-metrics.json#slices.public_bmc2_exact.candidate; **Metric:** normalized_exact_match; **Name:** public_bmc2_exact; **Passed:** true — Evidence state: measured — Sources: slice-metrics.json#slices.public_bmc2_exact.pass; **Sample size:** 64 — Evidence state: measured — Sources: slice-metrics.json#slices.public_bmc2_exact.rows - **Baseline:** 0.14 — Evidence state: measured — Sources: slice-metrics.json#slices.synthetic_sqlite_exact.baseline; **Candidate:** 0.84 — Evidence state: measured — Sources: slice-metrics.json#slices.synthetic_sqlite_exact.candidate; **Delta:** 0.7 — Evidence state: derived — Derived from: - slice-metrics.json#slices.synthetic_sqlite_exact.baseline - slice-metrics.json#slices.synthetic_sqlite_exact.candidate — Sources: slice-metrics.json#slices.synthetic_sqlite_exact.baseline, slice-metrics.json#slices.synthetic_sqlite_exact.candidate; **Metric:** normalized_exact_match; **Name:** synthetic_sqlite_exact; **Passed:** true — Evidence state: measured — Sources: slice-metrics.json#slices.synthetic_sqlite_exact.pass; **Sample size:** 50 — Evidence state: measured — Sources: slice-metrics.json#slices.synthetic_sqlite_exact.rows - **Baseline:** 0.16 — Evidence state: measured — Sources: slice-metrics.json#slices.synthetic_sqlite_execution.baseline; **Candidate:** 0.86 — Evidence state: measured — Sources: slice-metrics.json#slices.synthetic_sqlite_execution.candidate; **Delta:** 0.7 — Evidence state: derived — Derived from: - slice-metrics.json#slices.synthetic_sqlite_execution.baseline - slice-metrics.json#slices.synthetic_sqlite_execution.candidate — Sources: slice-metrics.json#slices.synthetic_sqlite_execution.baseline, slice-metrics.json#slices.synthetic_sqlite_execution.candidate; **Metric:** execution_accuracy; **Name:** synthetic_sqlite_execution; **Passed:** true — Evidence state: measured — Sources: slice-metrics.json#slices.synthetic_sqlite_execution.pass; **Sample size:** 50 — Evidence state: measured — Sources: slice-metrics.json#slices.synthetic_sqlite_execution.rows ## Cost and performance **Eval time seconds:** Not recorded — Evidence state: missing — No cost.json fragment records this field, so no timing or cost measurement exists for this run. — Sources: cost.json#eval_time_seconds **Latency ms:** Not recorded — Evidence state: missing — The candidate eval recorded no latency_ms. It is reported as not measured rather than as zero. — Sources: eval-candidate.json#latency_ms **Peak rss mb:** Not recorded — Evidence state: missing — The candidate eval recorded no peak_rss_mb. It is reported as not measured rather than as zero. — Sources: eval-candidate.json#peak_rss_mb **Tokens per second:** Not recorded — Evidence state: missing — The candidate eval recorded no tokens_per_second. It is reported as not measured rather than as zero. — Sources: eval-candidate.json#tokens_per_second **Training cost usd:** Not recorded — Evidence state: missing — No cost.json fragment records this field, so no timing or cost measurement exists for this run. — Sources: cost.json#training_cost_usd **Training time seconds:** Not recorded — Evidence state: missing — No cost.json fragment records this field, so no timing or cost measurement exists for this run. — Sources: cost.json#training_time_seconds ## Eval validity and leakage **Frontier ceiling:** Not recorded — Evidence state: missing — No frontier-ceiling score is recorded for this benchmark. Per docs/factory/eval-protocol.md an unvalidated benchmark cannot certify a verified ship. — Sources: eval-validity.json#frontier.by_suite.sql-poc-expanded-synthetic-execution **Frozen eval:** locked public64 + synthetic50 — Evidence state: measured — Identity comes from the dataset manifest's locked-split label; no eval-validity.json records a hashed frozen-eval identity. — Sources: dataset.json#processing.heldout_split **Known limitations:** - Routed report combines two metric families: public b-mc2 exact match and synthetic SQLite execution. Do not collapse this into one public SQL quality score. - Gate `sql-poc-expanded-synthetic-execution` has no frontier-ceiling evidence, so its absolute score is not calibrated against frontier capability. - Gate `sql-public-bmc2-exact-and-router-smoke` has no frontier-ceiling evidence, so its absolute score is not calibrated against frontier capability. **Leakage:** Not recorded — Evidence state: missing — No train/eval overlap check is recorded. Publication cannot certify the held-out set is uncontaminated. — Sources: eval-validity.json#overlap_check.result ## Caveats - Baseline eval note: Synthetic SQLite execution baseline on 50 heldout rows; exact match was 0.140. - Candidate eval note: Routed public-v4 + synthetic-expanded adapters: public b-mc2 exact 0.531 on 64 rows, synthetic exact 0.840 on 50 rows, router smoke 64/50 high-confidence. ## Source evidence - **Kind:** local-run-fragment; **Label:** run config; **Note:** Local run folder (gitignored); regenerate to inspect.; **Path:** runs/2026-07-02-sql-routed-qwen06-v1/config.json - **Kind:** local-run-fragment; **Label:** dataset manifest; **Note:** Local run folder (gitignored); regenerate to inspect.; **Path:** runs/2026-07-02-sql-routed-qwen06-v1/dataset.json - **Kind:** local-run-fragment; **Label:** frozen baseline eval; **Note:** Local run folder (gitignored); regenerate to inspect.; **Path:** runs/2026-07-02-sql-routed-qwen06-v1/eval-baseline.json - **Kind:** local-run-fragment; **Label:** candidate eval; **Note:** Local run folder (gitignored); regenerate to inspect.; **Path:** runs/2026-07-02-sql-routed-qwen06-v1/eval-candidate.json - **Kind:** local-run-fragment; **Label:** decision record; **Note:** Local run folder (gitignored); regenerate to inspect.; **Path:** runs/2026-07-02-sql-routed-qwen06-v1/decision.json - **Kind:** local-run-fragment; **Label:** per-slice metrics; **Note:** Local run folder (gitignored); regenerate to inspect.; **Path:** runs/2026-07-02-sql-routed-qwen06-v1/slice-metrics.json - **Kind:** local-run-fragment; **Label:** reproducibility provenance; **Note:** Local run folder (gitignored); regenerate to inspect.; **Path:** runs/2026-07-02-sql-routed-qwen06-v1/provenance.json - **Kind:** local-run-fragment; **Label:** human report; **Note:** Local run folder (gitignored); regenerate to inspect.; **Path:** runs/2026-07-02-sql-routed-qwen06-v1/report.md - **Kind:** local-run-fragment; **Label:** qualitative trace review; **Note:** Local run folder (gitignored); regenerate to inspect.; **Path:** runs/2026-07-02-sql-routed-qwen06-v1/trace_review.md - **Kind:** local-run-fragment; **Label:** artifact metadata; **Note:** Local run folder (gitignored); regenerate to inspect.; **Path:** runs/2026-07-02-sql-routed-qwen06-v1/artifact.json - **Kind:** local-run-fragment; **Label:** training log; **Note:** Local run folder (gitignored); regenerate to inspect.; **Path:** runs/2026-07-02-sql-routed-qwen06-v1/train.log - **Kind:** cited-evidence; **Label:** docs/factory/public-artifacts.md; **Path:** docs/factory/public-artifacts.md; **Sha256:** f82d9e5b6c92dcca90a615a28aa3df84738402d1d2cd4c2032fc2dc086fb3ee5 - **Kind:** cited-evidence; **Label:** docs/specialists/b1-sql-poc.md; **Path:** docs/specialists/b1-sql-poc.md; **Sha256:** f862be06d44c97aae43fbdcf0a7c93b16888193fe14771ffc9895848e7b03b83 ## Compiled from **Compiler:** scripts/factory/build_fine_tune_report_card.py **Compiler version:** 1.0.0 **Dataset hashes:** - **Path:** evals/sql-poc-expanded/train.jsonl; **Rows:** 108; **Sha256:** b82c44eed469340e967d7c8eb593a73e16ac1b8d3bfac4747bdcfbe879f6fd44 - **Path:** evals/sql-poc-expanded/dev.jsonl; **Rows:** 50; **Sha256:** 01ad0f5136324507f229e3302e8be53621b7462896db8c60854932df91018c93 - **Path:** evals/sql-poc-expanded/preferences.jsonl; **Rows:** 108; **Sha256:** 0dc7d074683a97eadabd6d6eb45eb8c84f1a9bf48df9f7024a7382abfbf75ca8 - **Path:** evals/sql-public-bmc2-train-v4-joinweighted/train.jsonl; **Rows:** 5567; **Sha256:** 749332268f66756625a563c5fb54b6157acc07ef1ab21c39ddab86b1317a318b - **Path:** evals/sql-public-bmc2-train-v4-joinweighted/dev.jsonl; **Rows:** 64; **Sha256:** d5b374d646e348dfb751a321d5701d06cc4f20c6734735c59771215037349f6c - **Path:** evals/sql-public-bmc2-train-v4-joinweighted/preferences.jsonl; **Rows:** 2048; **Sha256:** 4a9d12703eb6cd2ca4ff56cb2a79d4fe942b8e41c0ad322a7085c3342c267186 - **Path:** evals/sql-routed-mixed-v1/mixed114.jsonl; **Rows:** 114; **Sha256:** c2ba6579df56c3ef2d39827fd54ec8d51c01df1cdf307e1954ffeaf70ab6b183 **Source id:** 2026-07-02-sql-routed-qwen06-v1 **Source kind:** factory-run ## Report card id 2026-07-02-sql-routed-qwen06-v1