> Canonical page: https://posttrainllm.com/report-cards/qwen06-sql-routed-v1

PostTrainLLM fine-tune report card · schema v1 · compiler 1.0.0

# sql-routed-specialist-poc — routed-sft-lora

 Publish the current SQL routed POC as a transparent factory report artifact with measured blockers before packaging.

**Target**
 sql-routed-specialist-poc

**Base model**
 Qwen/Qwen3-0.6B (bf16, c1899de289a04d12100db370d81485cdf75e47ca)

**Candidate**
 qwen06-sql-routed-v1

**Method**
 routed-sft-lora

**Compiled from**
 2026-07-02-sql-routed-qwen06-v1 (factory-run)

## Decision

 Retry — eval Report-only artifact — no model to use

The routed setup is the current best SQL candidate and passes the current public exact plus synthetic execution gates, but it is not a shipped specialist until a public execution benchmark and performance measurements exist.

### Verification status

Not fully verified. This report does not claim a verified ship. Reasons:

- Decision is `retry-eval`, not `ship`; verification applies to ship claims only.

- Primary gate `sql-poc-expanded-synthetic-execution` has no frontier-ceiling evidence, so the eval is unverified as a ruler.

- Train/eval overlap (leakage) was not checked with current evidence.

- Open blocker: public execution DB bundle not local

- Open blocker: latency/RAM/tok-s not measured

- Open blocker: clean-SQL output hygiene gate missing

**Failure reason**
 The routed artifact lacks public execution benchmarking, performance measurements, and a clean-SQL package gate.

**Failure-reason confidence**
 exact

**Lesson**
 Routing preserves both measured SQL slices, but artifact status still depends on public execution and performance evidence.

### Open blockers

- public execution DB bundle not local

- latency/RAM/tok-s not measured

- clean-SQL output hygiene gate missing

### Next action

 Run scripts/sql/build_sql_spider_execution_gate.py against a local Spider DB bundle, score the routed candidate, then re-render this report with latency/RAM/tok-s.

## Before and after

 Every gate with baseline, candidate, derived delta, threshold, result, sample size, and frontier-ceiling evidence.

**Gate**

**Role**

**Metric**

**Baseline**

**Candidate**

**Delta**

**Threshold**

**Result**

**n**

**Frontier ceiling**

**sql-poc-expanded-synthetic-execution**

 primary
 sql-poc-expanded-synthetic-execution
 0.1600
 0.8600
 0.7000 derived
 0.8600
 yes
 50 Sample size from slice `synthetic_sqlite_execution`, named by config.eval.primary_slice.
 — not recorded No frontier-ceiling score is recorded for this benchmark. Per docs/factory/eval-protocol.md an unvalidated benchmark cannot certify a verified ship.

**sql-public-bmc2-exact-and-router-smoke**

 regression
 normalized_exact_match
 0.4840 From slice `public_bmc2_exact`, named by config.eval.regression_slice.
 0.5310 From slice `public_bmc2_exact`, named by config.eval.regression_slice.
 0.0470 derived
 0 pp max drop
 yes
 64
 — not recorded No frontier-ceiling score is recorded for this benchmark. Per docs/factory/eval-protocol.md an unvalidated benchmark cannot certify a verified ship.

### Eval identity

 Which suite produced each gate, how it was invoked, and whether it is frozen.

**Gate**

**Suite**

**Command**

**Date**

**Frozen**

**sql-poc-expanded-synthetic-execution**

 sql-poc-expanded-synthetic-execution
 scripts/sql/run_sql_routed_generate.py + eval-sql/score_sql_public_exact
 2026-07-02
 locked public64 + synthetic50 Identity comes from the dataset manifest's locked-split label; no eval-validity.json records a hashed frozen-eval identity.

**sql-public-bmc2-exact-and-router-smoke**

 sql-public-bmc2-exact-and-router-smoke
 — not recorded No eval command is recorded for this gate.
 2026-07-02
 — not recorded The run folder does not record whether the regression suite is frozen.

## Per-slice evidence

 Primary metric broken down by task slice, so an overall win cannot hide a weak slice.

**Slice**

**Metric**

**Baseline**

**Candidate**

**Delta**

**Result**

**n**

**known_hard_slice_join**

 synthetic_execution_accuracy
 — not recorded This slice records no baseline score.
 0.6875
 — not recorded Delta cannot be derived: baseline has no recorded value. No change is implied.
 — not recorded This slice records no gate result.
 16

**public_bmc2_exact**

 normalized_exact_match
 0.4840
 0.5310
 0.0470 derived
 yes
 64

**synthetic_sqlite_exact**

 normalized_exact_match
 0.1400
 0.8400
 0.7000 derived
 yes
 50

**synthetic_sqlite_execution**

 execution_accuracy
 0.1600
 0.8600
 0.7000 derived
 yes
 50

## Cost and performance

 Latency, memory, throughput, and training cost/time. Absent evidence is reported as not measured, never as zero.

**Metric**

**Value**

**Source**

**Latency**

 — not recorded The candidate eval recorded no latency_ms. It is reported as not measured rather than as zero.
 eval-candidate.json#latency_ms

**RAM / peak RSS**

 — not recorded The candidate eval recorded no peak_rss_mb. It is reported as not measured rather than as zero.
 eval-candidate.json#peak_rss_mb

**Decode throughput**

 — not recorded The candidate eval recorded no tokens_per_second. It is reported as not measured rather than as zero.
 eval-candidate.json#tokens_per_second

**Training time**

 — not recorded No cost.json fragment records this field, so no timing or cost measurement exists for this run.
 cost.json#training_time_seconds

**Training cost**

 — not recorded No cost.json fragment records this field, so no timing or cost measurement exists for this run.
 cost.json#training_cost_usd

**Eval time**

 — not recorded No cost.json fragment records this field, so no timing or cost measurement exists for this run.
 cost.json#eval_time_seconds

## Eval validity and leakage

 Whether the benchmark is a trustworthy ruler: frontier ceiling, frozen-eval identity, and train/eval overlap.

**Check**

**Result**

**Source**

**Frontier ceiling**

 — not recorded No frontier-ceiling score is recorded for this benchmark. Per docs/factory/eval-protocol.md an unvalidated benchmark cannot certify a verified ship.
 eval-validity.json#frontier.by_suite.sql-poc-expanded-synthetic-execution

**Frozen eval**

 locked public64 + synthetic50 Identity comes from the dataset manifest's locked-split label; no eval-validity.json records a hashed frozen-eval identity.
 dataset.json#processing.heldout_split

**Train/eval overlap**

 — not recorded No train/eval overlap check is recorded. Publication cannot certify the held-out set is uncontaminated.
 eval-validity.json#overlap_check.result

### Known eval limitations

- Routed report combines two metric families: public b-mc2 exact match and synthetic SQLite execution. Do not collapse this into one public SQL quality score.

- Gate `sql-poc-expanded-synthetic-execution` has no frontier-ceiling evidence, so its absolute score is not calibrated against frontier capability.

- Gate `sql-public-bmc2-exact-and-router-smoke` has no frontier-ceiling evidence, so its absolute score is not calibrated against frontier capability.

## Caveats

- Baseline eval note: Synthetic SQLite execution baseline on 50 heldout rows; exact match was 0.140.

- Candidate eval note: Routed public-v4 + synthetic-expanded adapters: public b-mc2 exact 0.531 on 64 rows, synthetic exact 0.840 on 50 rows, router smoke 64/50 high-confidence.

## Source evidence

Every number above traces to one of these artifacts. Content hashes are recorded where the source file is committed.

- run config runs/2026-07-02-sql-routed-qwen06-v1/config.json · local-run-fragment · Local run folder (gitignored); regenerate to inspect.

- dataset manifest runs/2026-07-02-sql-routed-qwen06-v1/dataset.json · local-run-fragment · Local run folder (gitignored); regenerate to inspect.

- frozen baseline eval runs/2026-07-02-sql-routed-qwen06-v1/eval-baseline.json · local-run-fragment · Local run folder (gitignored); regenerate to inspect.

- candidate eval runs/2026-07-02-sql-routed-qwen06-v1/eval-candidate.json · local-run-fragment · Local run folder (gitignored); regenerate to inspect.

- decision record runs/2026-07-02-sql-routed-qwen06-v1/decision.json · local-run-fragment · Local run folder (gitignored); regenerate to inspect.

- per-slice metrics runs/2026-07-02-sql-routed-qwen06-v1/slice-metrics.json · local-run-fragment · Local run folder (gitignored); regenerate to inspect.

- reproducibility provenance runs/2026-07-02-sql-routed-qwen06-v1/provenance.json · local-run-fragment · Local run folder (gitignored); regenerate to inspect.

- human report runs/2026-07-02-sql-routed-qwen06-v1/report.md · local-run-fragment · Local run folder (gitignored); regenerate to inspect.

- qualitative trace review runs/2026-07-02-sql-routed-qwen06-v1/trace_review.md · local-run-fragment · Local run folder (gitignored); regenerate to inspect.

- artifact metadata runs/2026-07-02-sql-routed-qwen06-v1/artifact.json · local-run-fragment · Local run folder (gitignored); regenerate to inspect.

- training log runs/2026-07-02-sql-routed-qwen06-v1/train.log · local-run-fragment · Local run folder (gitignored); regenerate to inspect.

- [docs/factory/public-artifacts.md](https://github.com/PostTrainLLM/posttrainllm/blob/main/docs/factory/public-artifacts.md) docs/factory/public-artifacts.md · cited-evidence · sha256 f82d9e5b6c92dcca…

- [docs/specialists/b1-sql-poc.md](https://github.com/PostTrainLLM/posttrainllm/blob/main/docs/specialists/b1-sql-poc.md) docs/specialists/b1-sql-poc.md · cited-evidence · sha256 f862be06d44c97aa…

### Dataset hashes

 SHA-256 of each dataset the run recorded, for independent reproduction.

**Dataset**

**Rows**

**SHA-256**

**evals/sql-poc-expanded/train.jsonl**

 108
 b82c44eed469340e967d7c8eb593a73e16ac1b8d3bfac4747bdcfbe879f6fd44

**evals/sql-poc-expanded/dev.jsonl**

 50
 01ad0f5136324507f229e3302e8be53621b7462896db8c60854932df91018c93

**evals/sql-poc-expanded/preferences.jsonl**

 108
 0dc7d074683a97eadabd6d6eb45eb8c84f1a9bf48df9f7024a7382abfbf75ca8

**evals/sql-public-bmc2-train-v4-joinweighted/train.jsonl**

 5567
 749332268f66756625a563c5fb54b6157acc07ef1ab21c39ddab86b1317a318b

**evals/sql-public-bmc2-train-v4-joinweighted/dev.jsonl**

 64
 d5b374d646e348dfb751a321d5701d06cc4f20c6734735c59771215037349f6c

**evals/sql-public-bmc2-train-v4-joinweighted/preferences.jsonl**

 2048
 4a9d12703eb6cd2ca4ff56cb2a79d4fe942b8e41c0ad322a7085c3342c267186

**evals/sql-routed-mixed-v1/mixed114.jsonl**

 114
 c2ba6579df56c3ef2d39827fd54ec8d51c01df1cdf307e1954ffeaf70ab6b183

## How to read the evidence states

**measured**
 Read directly from a source artifact for this candidate.

**derived**
 Computed from other recorded values (for example a delta).

**historical**
 Imported from a legacy record without current canonical provenance. Treat as weaker than a measurement.

**skipped**
 Deliberately not run for this candidate.

**not recorded**
 Evidence should exist but does not. No number is implied.

**not applicable**
 The check does not apply to this candidate.