posttrainllm fine-tune report card · schema v1 · compiler 1.0.0

sql-routed-specialist-poc — routed-sft-lora

Publish the current SQL routed POC as a transparent factory report artifact with measured blockers before packaging.

Target: sql-routed-specialist-poc
Base model: Qwen/Qwen3-0.6B (bf16, c1899de289a04d12100db370d81485cdf75e47ca)
Candidate: qwen06-sql-routed-v1
Method: routed-sft-lora
Compiled from: 2026-07-02-sql-routed-qwen06-v1 (factory-run)

Decision

Retry — eval Report-only artifact — no model to use

The routed setup is the current best SQL candidate and passes the current public exact plus synthetic execution gates, but it is not a shipped specialist until a public execution benchmark and performance measurements exist.

Verification status

Not fully verified. This report does not claim a verified ship. Reasons:

Decision is `retry-eval`, not `ship`; verification applies to ship claims only.
Primary gate `sql-poc-expanded-synthetic-execution` has no frontier-ceiling evidence, so the eval is unverified as a ruler.
Train/eval overlap (leakage) was not checked with current evidence.
Open blocker: public execution DB bundle not local
Open blocker: latency/RAM/tok-s not measured
Open blocker: clean-SQL output hygiene gate missing

Failure reason: The routed artifact lacks public execution benchmarking, performance measurements, and a clean-SQL package gate.
Failure-reason confidence: exact
Lesson: Routing preserves both measured SQL slices, but artifact status still depends on public execution and performance evidence.

Open blockers

public execution DB bundle not local
latency/RAM/tok-s not measured
clean-SQL output hygiene gate missing

Next action

Run scripts/build_sql_spider_execution_gate.py against a local Spider DB bundle, score the routed candidate, then re-render this report with latency/RAM/tok-s.

Before and after

Every gate with baseline, candidate, derived delta, threshold, result, sample size, and frontier-ceiling evidence.
Gate	Role	Metric	Baseline	Candidate	Delta	Threshold	Result	n	Frontier ceiling
sql-poc-expanded-synthetic-execution	primary	sql-poc-expanded-synthetic-execution	0.1600	0.8600	0.7000 derived	0.8600	yes	50Sample size from slice `synthetic_sqlite_execution`, named by config.eval.primary_slice.	— not recordedNo frontier-ceiling score is recorded for this benchmark. Per docs/factory/eval-protocol.md an unvalidated benchmark cannot certify a verified ship.
sql-public-bmc2-exact-and-router-smoke	regression	normalized_exact_match	0.4840From slice `public_bmc2_exact`, named by config.eval.regression_slice.	0.5310From slice `public_bmc2_exact`, named by config.eval.regression_slice.	0.0470 derived	0 pp max drop	yes	64	— not recordedNo frontier-ceiling score is recorded for this benchmark. Per docs/factory/eval-protocol.md an unvalidated benchmark cannot certify a verified ship.

Eval identity

Which suite produced each gate, how it was invoked, and whether it is frozen.
Gate	Suite	Command	Date	Frozen
sql-poc-expanded-synthetic-execution	sql-poc-expanded-synthetic-execution	scripts/run_sql_routed_generate.py + eval-sql/score_sql_public_exact	2026-07-02	locked public64 + synthetic50Identity comes from the dataset manifest's locked-split label; no eval-validity.json records a hashed frozen-eval identity.
sql-public-bmc2-exact-and-router-smoke	sql-public-bmc2-exact-and-router-smoke	— not recordedNo eval command is recorded for this gate.	2026-07-02	— not recordedThe run folder does not record whether the regression suite is frozen.

Per-slice evidence

Primary metric broken down by task slice, so an overall win cannot hide a weak slice.
Slice	Metric	Baseline	Candidate	Delta	Result	n
known_hard_slice_join	synthetic_execution_accuracy	— not recordedThis slice records no baseline score.	0.6875	— not recordedDelta cannot be derived: baseline has no recorded value. No change is implied.	— not recordedThis slice records no gate result.	16
public_bmc2_exact	normalized_exact_match	0.4840	0.5310	0.0470 derived	yes	64
synthetic_sqlite_exact	normalized_exact_match	0.1400	0.8400	0.7000 derived	yes	50
synthetic_sqlite_execution	execution_accuracy	0.1600	0.8600	0.7000 derived	yes	50

Cost and performance

Latency, memory, throughput, and training cost/time. Absent evidence is reported as not measured, never as zero.
Metric	Value	Source
Latency	— not recordedThe candidate eval recorded no latency_ms. It is reported as not measured rather than as zero.	`eval-candidate.json#latency_ms`
RAM / peak RSS	— not recordedThe candidate eval recorded no peak_rss_mb. It is reported as not measured rather than as zero.	`eval-candidate.json#peak_rss_mb`
Decode throughput	— not recordedThe candidate eval recorded no tokens_per_second. It is reported as not measured rather than as zero.	`eval-candidate.json#tokens_per_second`
Training time	— not recordedNo cost.json fragment records this field, so no timing or cost measurement exists for this run.	`cost.json#training_time_seconds`
Training cost	— not recordedNo cost.json fragment records this field, so no timing or cost measurement exists for this run.	`cost.json#training_cost_usd`
Eval time	— not recordedNo cost.json fragment records this field, so no timing or cost measurement exists for this run.	`cost.json#eval_time_seconds`

Eval validity and leakage

Whether the benchmark is a trustworthy ruler: frontier ceiling, frozen-eval identity, and train/eval overlap.
Check	Result	Source
Frontier ceiling	— not recordedNo frontier-ceiling score is recorded for this benchmark. Per docs/factory/eval-protocol.md an unvalidated benchmark cannot certify a verified ship.	`eval-validity.json#frontier.by_suite.sql-poc-expanded-synthetic-execution`
Frozen eval	locked public64 + synthetic50Identity comes from the dataset manifest's locked-split label; no eval-validity.json records a hashed frozen-eval identity.	`dataset.json#processing.heldout_split`
Train/eval overlap	— not recordedNo train/eval overlap check is recorded. Publication cannot certify the held-out set is uncontaminated.	`eval-validity.json#overlap_check.result`

Known eval limitations

Routed report combines two metric families: public b-mc2 exact match and synthetic SQLite execution. Do not collapse this into one public SQL quality score.
Gate `sql-poc-expanded-synthetic-execution` has no frontier-ceiling evidence, so its absolute score is not calibrated against frontier capability.
Gate `sql-public-bmc2-exact-and-router-smoke` has no frontier-ceiling evidence, so its absolute score is not calibrated against frontier capability.

Caveats

Baseline eval note: Synthetic SQLite execution baseline on 50 heldout rows; exact match was 0.140.
Candidate eval note: Routed public-v4 + synthetic-expanded adapters: public b-mc2 exact 0.531 on 64 rows, synthetic exact 0.840 on 50 rows, router smoke 64/50 high-confidence.

Source evidence

Every number above traces to one of these artifacts. Content hashes are recorded where the source file is committed.

run configruns/2026-07-02-sql-routed-qwen06-v1/config.json · local-run-fragment · Local run folder (gitignored); regenerate to inspect.
dataset manifestruns/2026-07-02-sql-routed-qwen06-v1/dataset.json · local-run-fragment · Local run folder (gitignored); regenerate to inspect.
frozen baseline evalruns/2026-07-02-sql-routed-qwen06-v1/eval-baseline.json · local-run-fragment · Local run folder (gitignored); regenerate to inspect.
candidate evalruns/2026-07-02-sql-routed-qwen06-v1/eval-candidate.json · local-run-fragment · Local run folder (gitignored); regenerate to inspect.
decision recordruns/2026-07-02-sql-routed-qwen06-v1/decision.json · local-run-fragment · Local run folder (gitignored); regenerate to inspect.
per-slice metricsruns/2026-07-02-sql-routed-qwen06-v1/slice-metrics.json · local-run-fragment · Local run folder (gitignored); regenerate to inspect.
reproducibility provenanceruns/2026-07-02-sql-routed-qwen06-v1/provenance.json · local-run-fragment · Local run folder (gitignored); regenerate to inspect.
human reportruns/2026-07-02-sql-routed-qwen06-v1/report.md · local-run-fragment · Local run folder (gitignored); regenerate to inspect.
qualitative trace reviewruns/2026-07-02-sql-routed-qwen06-v1/trace_review.md · local-run-fragment · Local run folder (gitignored); regenerate to inspect.
artifact metadataruns/2026-07-02-sql-routed-qwen06-v1/artifact.json · local-run-fragment · Local run folder (gitignored); regenerate to inspect.
training logruns/2026-07-02-sql-routed-qwen06-v1/train.log · local-run-fragment · Local run folder (gitignored); regenerate to inspect.
docs/factory/public-artifacts.mddocs/factory/public-artifacts.md · cited-evidence · sha256 7213abbd01e45a89…
docs/specialists/b1-sql-poc.mddocs/specialists/b1-sql-poc.md · cited-evidence · sha256 a93eb5f0ca73ef2e…

Dataset hashes

SHA-256 of each dataset the run recorded, for independent reproduction.
Dataset	Rows	SHA-256
`evals/sql-poc-expanded/train.jsonl`	108	`b82c44eed469340e967d7c8eb593a73e16ac1b8d3bfac4747bdcfbe879f6fd44`
`evals/sql-poc-expanded/dev.jsonl`	50	`01ad0f5136324507f229e3302e8be53621b7462896db8c60854932df91018c93`
`evals/sql-poc-expanded/preferences.jsonl`	108	`0dc7d074683a97eadabd6d6eb45eb8c84f1a9bf48df9f7024a7382abfbf75ca8`
`evals/sql-public-bmc2-train-v4-joinweighted/train.jsonl`	5567	`749332268f66756625a563c5fb54b6157acc07ef1ab21c39ddab86b1317a318b`
`evals/sql-public-bmc2-train-v4-joinweighted/dev.jsonl`	64	`d5b374d646e348dfb751a321d5701d06cc4f20c6734735c59771215037349f6c`
`evals/sql-public-bmc2-train-v4-joinweighted/preferences.jsonl`	2048	`4a9d12703eb6cd2ca4ff56cb2a79d4fe942b8e41c0ad322a7085c3342c267186`
`evals/sql-routed-mixed-v1/mixed114.jsonl`	114	`c2ba6579df56c3ef2d39827fd54ec8d51c01df1cdf307e1954ffeaf70ab6b183`

How to read the evidence states

measured: Read directly from a source artifact for this candidate.
derived: Computed from other recorded values (for example a delta).
historical: Imported from a legacy record without current canonical provenance. Treat as weaker than a measurement.
skipped: Deliberately not run for this candidate.
not recorded: Evidence should exist but does not. No number is implied.
not applicable: The check does not apply to this candidate.