Attempt Ledger

Attempt Ledger

This ledger records meaningful posttrainllm attempts as worked, failed, regressed, inconclusive, or not yet tried. It is the human-readable companion to run folders and factory reports.

The point is not to look successful. The point is to preserve the experimental state so the next recipe is better than the last one.

Structured index: docs/attempts.json. Check sync with:

bash evals/attempt-ledger-smoke.sh

Read It By Shape, Not By Scrolling

This page is written one attempt at a time but read one question at a time, and the question is almost always “has anything shaped like this been tried?” Do not scroll — query:

python3 scripts/query_attempts.py --method dpo --objective output-format
python3 scripts/query_attempts.py --base flan-t5-small --failures-only
python3 scripts/query_attempts.py --lineage sql-hygiene-dpo-higher-pressure
python3 scripts/query_attempts.py --streaks
python3 scripts/query_attempts.py --coverage

Every model attempt carries kind, methods, bases, objective, data_rows, and varied_from. --lineage walks the varied_from chain so a series of retries reads as one story instead of scattered entries. --streaks reports objectives whose chain ends in consecutive failures – the signal to change axis rather than apply more pressure. A chain that failed three times and then found something that worked does not warn; only trailing failures count.

Coverage is honest rather than complete: shape is filled in only where it is derivable from an attempt’s own recorded evidence. --coverage reports the gaps. Do not backfill shape by guessing — see Historical Coverage Limits.

The lookup is wired into recipe validation, so a recipe with a shape block surfaces its precedents automatically. The workflow around it lives in techniques/method-vs-recipe.md.

Evidence Standard

docs/attempts.json is the machine-readable source for this ledger. Every attempt has evidence sources and a failure-reason confidence label.

Do not upgrade an old attempt from inferred to exact without linking the run report, decision file, artifact metadata, or source doc that proves it.

Status Vocabulary

SQL Specialist Attempts

SQL is the current factory POC and the best-documented attempt family.

Toy SQL SFT

• Recipe: Qwen3-0.6B + rank-4 DoRA/LoRA on 12 rows.
• Evidence: execution 0.167 -> 0.833 on 6-row fixture.
• Status: worked-with-caveat.
• Failure reason: Train/eval overlap made the apparent improvement non-shippable.
• Lesson: The SFT loop mechanics work, but eval splits must be frozen before claiming quality.
• Next action: Use only as a loop sanity check; rely on heldout SQL evals for real claims.

Expanded synthetic SQL SFT

• Recipe: 108 train rows, 50 heldout rows, five SQLite domains.
• Evidence: execution 0.160 -> 0.860, exact 0.140 -> 0.840.
• Status: worked.
• Lesson: Current synthetic SQL adapter; joins remain weaker than single-table rows.
• Next action: Keep as the synthetic incumbent until a candidate beats it on the same heldout gate.

Public b-mc2 v1

• Recipe: 512 public-style SFT rows, rank 8, 300 steps.
• Evidence: exact 0.344 vs T5-small 0.484.
• Status: failed.
• Failure reason: The public-style dataset was too small to beat the T5-small baseline.
• Lesson: Public benchmark adaptation needs more coverage before tuning knobs matter.
• Next action: Scale public-style rows and re-evaluate against the same exact gate.

Public b-mc2 v2

• Recipe: 2048 rows, rank 16, 1200 steps.
• Evidence: exact 0.031.
• Status: failed.
• Failure reason: Increasing rows, rank, and steps together made the recipe worse and confounded the cause.
• Lesson: Do not change data volume, rank, and training length in one uncontrolled jump.
• Next action: Use smaller controlled deltas with fixed eval and comparable training settings.

Public b-mc2 v3

• Recipe: 2048 rows, rank 8, lower LR, 600 steps.
• Evidence: exact 0.422.
• Status: failed.
• Failure reason: Lower LR and fewer steps recovered quality but still missed the public baseline.
• Lesson: The public recipe was directionally better, but not enough for a win.
• Next action: Try targeted data weighting instead of only global hyperparameter changes.

Public b-mc2 v4

• Recipe: Join/group weighted public rows, rank 8, 700 steps.
• Evidence: exact 0.531 vs T5-small 0.484.
• Status: worked-with-caveat.
• Failure reason: The public win did not preserve synthetic SQL execution behavior.
• Lesson: A narrow public exact-match win can still damage the incumbent specialist behavior.
• Next action: Route or compose specialists instead of forcing one adapter to satisfy both gates.

Public v4 synthetic regression check

• Recipe: v4 scored on synthetic execution fixture.
• Evidence: synthetic execution 0.240 vs incumbent 0.860.
• Status: regressed.
• Failure reason: Public-style training overfit schema-linking patterns that did not transfer to synthetic SQLite domains.
• Lesson: Every public benchmark gain needs a breadth/regression gate before packaging.
• Next action: Keep public and synthetic adapters separate unless a candidate passes both gates.

Blended public+synthetic SFT

• Recipe: Public rows + synthetic rows mixed into one adapter.
• Evidence: public 0.297, synthetic execution 0.560.
• Status: failed.
• Failure reason: Naive mixture training caused interference and output-format collapse.
• Lesson: Mixing domains is not automatically regularization; it can reduce both capabilities.
• Next action: Use routing, data-mix search, or explicit slice gates before another blended adapter.

Static multi-LoRA composition

• Recipe: Public + synthetic adapters composed with fixed weights.
• Evidence: best public-pass setting still synthetic-failed.
• Status: failed.
• Failure reason: Fixed adapter weights produced a smooth tradeoff but no setting passed both gates.
• Lesson: Static composition cannot solve domain conflict when the right behavior depends on input type.
• Next action: Prefer input-aware routing over global composition weights.

Routed public + synthetic adapters

• Recipe: Route public schemas to public adapter; synthetic DB rows to synthetic adapter.
• Evidence: public exact 0.531, synthetic execution 0.860.
• Status: worked-with-caveat.
• Failure reason: The routed artifact lacks public execution benchmarking and perf/package gates.
• Lesson: Routing preserves both measured slices, but artifact status still depends on public execution and performance evidence.
• Next action: Add public execution gate, latency/RAM/tok-s measurement, and a package decision.

Hygiene SimPO/DPO

• Recipe: Ref-free SimPO on 108 SQL-only preference pairs.
• Evidence: execution 0.860 -> 0.080, clean-SQL 0.000 -> 0.000.
• Status: failed.
• Failure reason: Ref-free SimPO update collapsed the policy and produced degenerate fenced/prose outputs.
• Lesson: Preference tuning needs reference anchoring, smaller updates, and composed-eval checks.
• Next action: Retry with reference-anchored DPO or much lower LR/step count on the same frozen pairs. Done 2026-07-11 (see next entry).

Hygiene DPO reference-anchored retry (2026-07-11)

• Recipe: Reference-anchored DPO (posttrainllm dpo --loss-type dpo --beta 0.1, LoRA r4 q/v, 50 steps, lr 5e-6) on the same 108 frozen pairs, composed at inference with the SFT DoRA adapter; evaluated on the frozen 50-row dev set.
• Evidence: composed execution 0.860 -> 0.900 (+0.040, exact 0.840 -> 0.860), clean-SQL 0.000 -> 0.000; DPO-adapter-alone execution 0.120 (healthy, not the SimPO retry’s 0.000 fence-spam); DPO step-1 loss 0.6931 ≈ log 2 (correct reference-anchored sanity check). Run: runs/2026-07-11-sql-hygiene-dpo-refanchored-qwen06/.
• Status: regressed? No — retry-training (execution bar passes, hygiene bar fails).
• Failure reason: DPO pressure too low to change output FORMAT — 41/50 raw outputs changed but all keep the Answer:/Explanation: prose wrapper, so clean-SQL stays 0.000 (< 0.80 bar).
• Lesson: Reference-anchored DPO is the correct fix for the SimPO collapse (validated: no collapse, execution preserved/improved, adapter-alone healthy). Output-format hygiene needs stronger preference pressure (more steps / higher beta/lr, still ref-anchored) or an SFT-level format fix. Binary provenance matters: reproduced the 0.860 baseline exactly with a fresh swift-build DEBUG binary before scoring.
• Next action: Higher-pressure reference-anchored DPO… Done 2026-07-11 (next entry).

Hygiene DPO higher-pressure retry (2026-07-11)

• Recipe: Reference-anchored DPO at higher pressure (--beta 0.3, 200 steps, lr 1e-5) on the same 108 frozen pairs; composed eval + DPO-alone.
• Evidence: DPO loss 0.6931 -> 0.0073 (strong preference separation). Composed execution 0.860 -> 0.920 (+0.060, exact 0.880); clean-SQL still 0.000. Composed output keeps Answer:; DPO-alone (exec 0.300) outputs The answer is:. Run: runs/2026-07-11-sql-hygiene-dpo-refanchored-b03-s200-qwen06/.
• Status: failed (for hygiene); decision retry-data.
• Failure reason: Output format is SFT/base-controlled and not reachable by a composed rank-4 q/v preference adapter — DPO perfectly separates the pairs without changing greedy generation’s opening tokens.
• Lesson: Composed DPO cannot fix output-format hygiene here — proven across two pressure regimes (gentle 50-step and aggressive 200-step both leave clean at 0.000 while execution only rises 0.860 → 0.900 → 0.920). Preference tuning is the wrong tool for format. Positive side effect: execution improved every time.
• Diagnosis correction (2026-07-11): the SFT training data is already clean — 108/108 evals/sql-poc-expanded/train.jsonl targets start with a bare SELECT, no wrapper. So the Answer:/The answer is: lead-in is the base model’s (Qwen3-0.6B) prose prior, which the rank-4 DoRA SFT (108 examples) and the rank-4 DPO both fail to suppress. A data rebuild would be a no-op.
• Next action: fix the format at generation strength, not data content — candidates: (a) stronger SFT (higher rank / more epochs / more bare-SELECT examples) to overpower the base prior; (b) inference-time steering (few-shot bare-SELECT exemplars, a stop sequence, or a constrained-generation SELECT prefix). The scorer already extracts the inner SELECT (exec 0.92), so a cheap deterministic output post-process is also a legitimate hygiene fix.

SQL candidate selection

• Recipe: Choose best SQL among candidates before open generation.
• Evidence: tooling and smoke exist; no model run.
• Status: not-tried.
• Lesson: Highest-priority next recipe.
• Next action: Run only after candidate rows are generated without leakage.

Offline rollout / OAPL-style SQL update

• Recipe: Batch rollouts, score offline, one adapter update.
• Evidence: plan renderer exists; no model run.
• Status: not-tried.
• Lesson: Run only after candidate-selection reward surface is clean.
• Next action: Use after candidate selection proves the reward/data loop.

Controlled SQL LoRA rank sweep

• Recipe: Same data, ranks {1,2,4,8}, fixed seed/steps/LR.
• Evidence: not run.
• Status: not-tried.
• Lesson: Needed to separate capacity from data/recipe confounds.
• Next action: Run only after the next target/eval is frozen.

Non-SQL Model / Artifact Attempts

Character Chess 44.53M masked 10k pilot (2026-08-05)

• Recipe: owned 44,527,616-parameter byte model, 10,000 CC0 Lichess-eval general-position rows, completion-only move loss, batch 16, learning rate 3e-4, 2,000 MPS steps.
• Evidence: validation exact 6.16% random -> 10.54% (+4.38 points), test 6.23% -> 10.33% (+4.10); raw and guarded policies each won 0/6 games against random legal play. evals/chess/character-chess-44m-pilot-10k-v1.json.
• Status: failed.
• Failure reason: Completion-only SFT learned a small held-out move preference but missed the frozen +10-point promotion gate and produced no full-game win advantage over random legal play; eight guard interventions converted no wins.
• Lesson: Legal constrained decoding and lower move-prediction loss are not game intelligence; require full-game transfer before scaling a tiny specialist.
• Next action: Do not run the 100k, 1M, or 2M stages under this recipe. Preserve the failed artifact and select a different bounded specialist target.
• Confidence: exact.

Autocorrect FLAN-T5-small tiny-overfit gate (2026-07-25)

• Recipe: adapter-recipe-v1, FLAN-T5-small + LoRA r8 on q/v across all three attention sites, 8 rows / 3,321 bytes, batch 4.
• Evidence: exact match 1.0 at step 50 of 200; loss 1.585 -> 0.030; 0.28 min; 1,135 MiB peak RSS. evals/autocorrect/tiny-overfit-result-v1.json.
• Status: worked-with-caveat.
• Failure reason: The fixture has one unique target, so exact match 1.0 is reachable by memorizing a single sentence; it measures capacity and wiring, not correction.
• Lesson: A memorization gate needs more than one distinct target, or “memorized the data” and “emitted a constant” are the same behaviour. A separate probe on unseen input was needed to tell them apart.
• Next action: Keep as a precondition check only; never quote as a quality result.
• Confidence: exact.

Autocorrect FLAN-T5-small ordinary-loss pilot (2026-07-25)

• Recipe: same recipe, 12 train-split rows (4 development rows monitored, not trained on), scored on the unchanged frozen eval-v1.jsonl.
• Evidence: error reduction +0.0625 -> -0.8125 (delta -0.875); unnecessary edit rate 0.839 against a 0.005 bar; protected spans 0.867 -> 0.800; exact match 0.389 -> 0.444. evals/autocorrect/pilot-result-v1.json.
• Status: regressed.
• Failure reason: The model became a paraphraser rather than a repairer – remeber -> remind you, teh team -> your team, tomorroww -> tomorrow morning – while leaving repourt unfixed. Negative error reduction means output moved further from the clean reference than the noisy input was. Structural slices (casing, name, number, url) stayed clean at 1.0.
• Lesson: Minimum-edit repair is a narrow band between copying everything and rewriting everything, and plain sequence loss on 12 rows crossed it entirely in 50 steps. Separately, the run stopped at step 50 of 300 on a stop rule the base model already violates (clean preservation bar 0.995 vs base 0.667) – a ship bar reused as a training guard, which makes the pilot untrainable past its first eval by construction.
• Next action: No further training under adapter-recipe-v1. A v2 must separate training stop rules from ship bars, add data, and add a meaning-change guard. The edit-aware objective (5.6-5.7) is explicitly rejected by this evidence: it up-weights edit positions, which pushes toward the wall this model already hit.
• Confidence: exact for the measured numbers; the run is truncated evidence (50 of 300 steps) and is not a fair test of the full recipe.

Qwen3-4B file-ops distilled specialist

• Evidence: file-ops hard gate 0.58 -> 1.00; breadth 0.596 -> 0.423.
• Status: worked-with-caveat.
• Failure reason: The specialist improved the narrow file-ops gate but regressed out-of-domain breadth.
• Lesson: The artifact is useful only with routed-only positioning; it should not be sold as a general planner.
• Next action: Keep routed-only warnings prominent and add loader/pull smoke before wiring consumers.
• Confidence: exact.

Qwen3-4B ReST fused breadth recovery variant

• Evidence: depth 100%, breadth 65% vs stock breadth 60%.
• Status: worked-with-caveat.
• Failure reason: Historical runtime and raw trace artifacts were not preserved, so the package cannot claim current latency/RAM/tok-s or a fresh qualitative review.
• Lesson: Existing public weights can be promoted honestly when the measured delta, routing boundary, and missing evidence are explicit.
• Next action: Keep it research-only unless a product-specific target justifies a fresh heavy eval and re-distillation.
• Confidence: exact for artifact/package state; inferred for the rounded historical breadth result.

Qwen3-4B multibackend distilled variant

• Evidence: depth 100%, breadth 31%.
• Status: regressed.
• Failure reason: Negative transfer damaged breadth even though depth remained high.
• Lesson: Multi-backend distillation can over-specialize and needs breadth gates by default.
• Next action: Keep as failed comparison evidence; do not promote without a new breadth-preserving recipe.
• Confidence: exact.

VibeThinker 3B agentic distilled variant

• Evidence: weights preserved; current eval promotion missing.
• Status: inconclusive.
• Failure reason: The weights exist, but the docs do not preserve a current factory eval proving a win.
• Lesson: Preserved weights are not a ship decision.
• Next action: Run the factory eval gate and publish a before/after report before using it as a specialist package.
• Confidence: exact.

Apple on-device Foundation Models action-grounding probe

• Evidence: BFCL agentic 25% full catalog / approximately 0% compact; Pace planner action-grounding 13%.
• Status: failed.
• Failure reason: Apple’s on-device model could not ground actions well enough and its 4096-token context could not hold a real tool catalog.
• Lesson: Apple Foundation Models are useful as a free routing floor and intel source, not as the core capability dependency.
• Next action: Keep Core ML for our own weights as an optional future battery optimization, not a model-dependency bet.
• Confidence: exact.

Pace Planner / Clarify Attempts

The v1-v11 series, split as far as the evidence allows.

Eleven versions were claimed; six have a recorded per-version score (v5, v6, v6.1, v8, v9, v11), v7 was never trained, v10 was never scored, and v1-v4 left no result behind. The entries below split the series only as far as the evidence supports.

Two cross-cutting findings the collapsed entry hid:

• One axis was varied for five straight versions. v5, v6, v8, v9, v10, and v11 all held rank 32 / alpha 64 / 3000 steps / lr 1e-4 / batch 4 constant and changed only the training corpus. Rank, learning rate, step count, and base model were never ablated across the whole series. The single step-count comparison that exists is inside v6.1, in a file that was later deleted.
• The corpus compounded unmeasured. 248 -> 307 -> 404 -> 709 rows, each version appending to the last, so any defect in the v5 base propagated through every successor. One such defect is recorded: the teacher used for v10 multiplication reflexively produced AX.press regardless of input.

Pace planner v1-v4 early versions

• Evidence: datasets, builder scripts, and planned recipes survive; no per-version score was recorded.
• Status: inconclusive.
• Failure reason: No per-version result was kept. The v11 ship gate names this directly: v1 through v10 cycled because there was no fixed bar, no fixed eval, and no immutable record of pass criteria.
• Lesson: An unmeasured version is indistinguishable from a version that never ran. Four consecutive attempts left no score behind.
• Next action: Do not attempt to reconstruct scores for v1-v4; treat the series as starting at v5, the first version with a recorded number.
• Confidence: missing-evidence.

Pace planner v5 hand-crafted gold + teacher paraphrase

• Data: 248 rows.
• Evidence: 17/19 (89%) on fm-fixtures v1 and 6/15 (40.0%) on fm-fixtures-v2.
• Status: worked-with-caveat.
• Failure reason: The 17/19 was measuring format compliance, not capability: a rule-based FakePace endpoint scored 19/19 on the same fixture set.
• Lesson: A high score on a fixture a deterministic endpoint can also ace is a measurement of the harness, not the model.
• Next action: Treat fm-fixtures v1 as retired; only fm-fixtures-v2 numbers are comparable.
• Confidence: exact.

Pace planner v6 label-based output

• Varied from pace-planner-v5: output representation changed from element IDs to labels with a deterministic label-to-ID lookup; hyperparameters held constant.
• Evidence: 14/19 (74%) on fm-fixtures v1 and 3/15 (20.0%) on fm-fixtures-v2, 20pp below v5.
• Status: regressed.
• Failure reason: The label-based SFT destroyed capability, not just output format.
• Lesson: Changing the output representation is a capability change, not a formatting change.
• Next action: Revert to the v5 corpus for further augmentation work.
• Confidence: exact.

Pace planner v6.1 four-adapter repair sweep

• Varied from pace-planner-v6: six adapters over corpus shape and step count: 276 rows/2000 steps, JSON-shaped/1000, fixture-gold/500, then 760 system-prompt-aligned rows at 100 and 300 steps.
• Data: 760 rows.
• Evidence: 10/19, 8/19, 2/19, 0/19, then 14/19 at 100 steps and 17/19 at 300 steps, reaching 19/19 after a serving-side minLength fix; 4/15 (26.7%) on fm-fixtures-v2.
• Status: worked-with-caveat.
• Failure reason: The apparent collapse was two harness bugs, not training: SFT rows omitted the Pace system prompt that serve and eval both included, and a brace-counting JSON extractor was faulty.
• Lesson: A four-way score collapse traced to prompt mismatch between training and serving. Pin the prompt, schema, and grammar triple across train, serve, and eval before believing any comparison.
• Next action: Recover the full sweep with git show d1efcb8^:docs/prds/factory-pace-planner-v6_1.md before repeating any step-count ablation; it is the only one in the series.
• Confidence: exact.

Pace planner v7 tools-in-prompt

• Evidence: tools-in-prompt serve, verb-enum grammar, and data/eval harnesses shipped; the SFT was never run.
• Status: not-tried.
• Lesson: v7 is a numbering gap, not a training version. The eleven-version count includes a version that was never trained.
• Next action: Do not cite v7 as evidence about model capability.
• Confidence: not-applicable.

Pace planner v8 augmented corpus

• Varied from pace-planner-v5: returned to the v5 corpus after the v6 label experiment regressed and added 59 rows of semantic disambiguation, multi-element reasoning, and abstract reference (248 to 307); hyperparameters unchanged.
• Data: 307 rows.
• Evidence: reported 11/15 (73.3%) on fm-fixtures-v2, later re-measured at 5/15 (33.3%) from the same weights and same fixtures.
• Status: inconclusive.
• Failure reason: The headline score did not reproduce. Re-running the same eval against the same baked weights yielded 33.3%, and the gap was root-caused as prompt and schema config drift rather than any model change.
• Lesson: The 73% phantom. A score is only a score if the prompt, schema, and grammar it was measured under are recorded with it.
• Next action: Treat 33.3% as the working v8 baseline and never compare across versions without pinning the harness config.
• Confidence: exact.

Pace planner v9 compose

• Varied from pace-planner-v8: backfilled a fourth schema field bodyText and added about 50 compose and draft rows.
• Evidence: 70% on new compose fixtures with non-compose held at the v8 level; as-shipped re-measurement 9/15 (60.0%); unhappy-path n=130 gave 0/40 ambiguous, 13/60 out-of-scope, 1/30 destructive.
• Status: worked-with-caveat.
• Failure reason: It shipped as the specialist but never became Pace’s runtime planner, and the unhappy-path drill put it at a capacity wall: 0% ambiguous and 3% destructive.
• Lesson: Winning the happy-path fixture says nothing about ambiguous, out-of-scope, or destructive inputs, which is where the product actually needed the planner.
• Next action: Keep as the last shipped specialist; Pace runs qwen3-30b-a3b instead.
• Confidence: exact.

Pace planner v10 parameterized actions

• Varied from pace-planner-v9: output representation changed again, from labels to a parameterized intent and payload schema, with a 404-row corpus built by 10x teacher multiplication.
• Data: 404 rows.
• Evidence: full recipe, action registry, schema, and eval plan are committed; no score was ever recorded and it is not established that training completed.
• Status: inconclusive.
• Failure reason: No v10 score exists anywhere in the repository. The last status recorded is that training was pending, while the v10 corpus went on to become the base of v11.
• Lesson: v10 repeated v6’s move of changing the output representation, and because it was never measured the v6 regression could not be checked for recurrence. Its unmeasured corpus was then inherited by v11.
• Next action: Do not reuse a v10-derived corpus without first measuring the representation change in isolation.
• Confidence: missing-evidence.

Pace planner v11 ship-gate run

• Varied from pace-planner-v10: 404 v10 rows plus 93 seeds plus amplification to 709 rows, a 7-intent grammar, and plain LoRA because the DoRA inference path was broken end to end.
• Data: 709 rows.
• Evidence: failed all six gate dimensions: BFCL pace-12 27.1% against a 40% bar, out-of-scope 0/30 against 80%, ambiguous 0/20 against 50%, destructive 5/10 against 90%, with train loss 0.001.
• Status: failed.
• Failure reason: A 0.6B planner did not reach the fixed bar on any dimension despite the lowest training loss in the series, and the first v11 attempt was voided outright by the DoRA infrastructure failure.
• Lesson: Train loss 0.001 alongside 0% on out-of-scope and ambiguous inputs is the capacity ceiling stated plainly. The one-shot fixed gate was what finally ended the cycle.
• Next action: Do not train v12. The specialist planner track is closed and Pace ships qwen3-30b-a3b.
• Confidence: exact.

Pace clarify-v1 Qwen3-4B LoRA

• Evidence: 38 contrastive rows moved ambig only 0% -> 5% while OOS regressed 80% -> 33% in the retrospective; drilldown records the same attempt as 2%/40%/50% on ambig/oos/destructive.
• Status: regressed.
• Failure reason: Small-corpus LoRA on a competent base caused catastrophic interference across planner dimensions.
• Lesson: Fine-tuning a strong base on a thin behavior slice requires coverage for every dimension that must not regress.
• Next action: Do not fine-tune the planner on narrow clarify rows unless the corpus also covers OOS, destructive, happy path, and breadth gates.
• Confidence: inferred.

Pace two-stage ambiguity detector shim v2

• Evidence: ambig moved 0% -> 20%, while OOS dropped 78% -> 55% and destructive stayed 67%; not shipped on top of Gemma.
• Status: worked-with-caveat.
• Failure reason: The shim improved ambiguity detection for the 4B floor but over-triggered and became unnecessary once Gemma reached the same ambig score with better OOS/destructive behavior.
• Lesson: Rule wrappers can expose the missing behavior, but they still need the same regression gates as model updates.
• Next action: Keep as an option only if shipping the 4B footprint floor; do not stack it on the stronger Gemma path.
• Confidence: exact.

Pace reasoning-model unhappy-path drill

• Evidence: Qwen3-4B-Thinking scored 0%/10%/0% and DeepSeek-R1-Distill-Qwen-7B scored 0%/3%/0% on ambig/oos/destructive under a 300-token cap.
• Status: inconclusive.
• Failure reason: The evaluation budget was consumed by think traces, so the apparent zero scores were not a fair model-quality verdict.
• Lesson: Reasoning models need adequate max tokens and a stop-on-think strategy before interpreting planner scores.
• Next action: Re-run with max_tokens >= 1024 and stop on </think> only if Gemma or the locked planner fails in production.
• Confidence: exact.

Pace Gemma-3-12B unhappy-path drill winner

• Evidence: Gemma-3-12B scored 22% ambig, 82% OOS, 77% destructive and won all three unhappy-path dimensions in the n=130 drill.
• Status: worked-with-caveat.
• Failure reason: Gemma won the drill but ambiguity remained poor in absolute terms and the decision was later superseded by the general planner lock.
• Lesson: A stronger zero-shot base beat the local fine-tunes, but model-selection evidence can be superseded by later broader gates.
• Next action: Keep as historical unhappy-path evidence; prefer current PROJECT_STATUS.md/docs/NEXT.md for active planner decisions.
• Confidence: exact.

Pace Qwen3.5 challenger round

• Evidence: Qwen3.5-9B scored 18%/97%/63%; Qwen3.5-4B scored 15%/55%/70% on ambig/oos/destructive; champion stood.
• Status: worked-with-caveat.
• Failure reason: Qwen3.5 improved some dimensions, especially OOS for 9B, but did not beat the stored champion across the full swap decision.
• Lesson: Single-dimension wins do not justify a planner swap when another required dimension regresses.
• Next action: Use scripts/eval_planner.sh for future challengers and keep swap/no-swap verdicts tied to all required dimensions.
• Confidence: exact.

Browser Product / Demo Attempts

Browser Memory64 large-preset runs

• Evidence: larger browser presets hit Memory64 OOB; matching Node direct-export reproducer passed XL; INITIAL_MEMORY=268435456 fixed up to XL while Behemoth still exercises growth.
• Status: worked-with-caveat.
• Failure reason: The original retries treated the OOB as a model-size problem before the matching reproducer showed a browser pthread plus SharedArrayBuffer memory-growth race.
• Lesson: A reproducer must match the production host and call path before it can explain a browser-only WASM failure.
• Next action: Keep the higher initial memory for supported presets and only claim Behemoth after a SAB-view-aware growth path lands.
• Confidence: exact.

Browser speedup headline curve

• Evidence: single 9.7x headline was replaced by a preset curve: Small 2.6x, Medium 6.8x, Large 9.3x, XL 12.1x.
• Status: worked-with-caveat.
• Failure reason: The original one-number headline made a shape-dependent performance result look like a universal property of the code.
• Lesson: If a performance result depends on preset size, publish the curve rather than the most convenient point.
• Next action: Keep future public performance artifacts tied to config-specific tables with reproduction context.
• Confidence: exact.

Browser default corpus fix

• Evidence: the old default corpus was 863 bytes; a Huge model memorized it with train loss 0.14; default changed to TinyShakespeare at about 1.1 MB.
• Status: worked-with-caveat.
• Failure reason: The demo was trying to prove learning with a corpus so small that memorization looked like training success.
• Lesson: Default data is part of the product claim; a training demo’s default corpus must produce non-memorized samples.
• Next action: Keep corpus defaults in the same parity review as model and optimizer defaults.
• Confidence: exact.

Browser learning-rate default drift

• Evidence: browser default LR was 3e-3 while Python reference used 3e-4; training plateaued around loss 2.45 until the browser default was changed.
• Status: worked-with-caveat.
• Failure reason: Reference/kernel parity tests covered math drift but not hyperparameter default drift.
• Lesson: Reference defaults are part of correctness, not just documentation.
• Next action: Add or keep a config-default parity check whenever browser and reference paths share a training story.
• Confidence: exact.

Browser WebGPU export serialization

• Evidence: a trained WebGPU run was lost because webgpu/gpu_model.ts had no checkpoint serialization; exportState was then implemented and worker plumbing updated.
• Status: worked-with-caveat.
• Failure reason: The training path could produce useful weights that the WebGPU backend could not export, and auto-cleanup destroyed a real trained artifact.
• Lesson: Long-running browser training is not product-complete until the smallest export path is smoke-tested before the long run.
• Next action: Smoke export on a short WebGPU run before any long browser training claim.
• Confidence: exact.

Browser generation streaming and KV cache

• Evidence: tokens/sec counter was added, but browser generation still lacked KV cache and true token streaming; task #72 was logged.
• Status: not-tried.
• Lesson: A typewriter animation over an already-complete response is not true streaming.
• Next action: Add worker protocol support for intermediate tokens and KV-cached browser generation before claiming interactive decode latency.
• Confidence: not-applicable.

Runtime / Browser / Architecture Attempts

WASM backward-scratch reuse

• Evidence: standard 304 -> 305 ms/step, capable 632 -> ~640 ms/step.
• Status: worked-with-caveat.
• Failure reason: Caching backward scratch buffers removed hot-path allocations but produced no measurable speedup because compute, not malloc, was the bottleneck.
• Lesson: Hot-path hygiene is not the same as a performance win; measure the real step time before claiming speedup.
• Next action: Keep the allocation cleanup, but do not prioritize allocator work until profiling shows allocation cost again.
• Confidence: exact.

WASM SIMD training speedup

• Evidence: standard 304 -> 191 ms/step, capable 632 -> 391 ms/step, net 1.6x speedup.
• Status: worked.
• Lesson: Emscripten/LLVM autovectorization was the right browser-training speed lever before hand-written kernels.
• Next action: Keep SIMD enabled by default and use the WASM smoke gate to protect training correctness.
• Confidence: not-applicable.

Browser WebGPU training loop

• Evidence: 24 kernel parity checks, GPU overfit gate loss 5.55 -> 0.002 in 150 steps, and headless-browser e2e passed.
• Status: worked-with-caveat.
• Failure reason: Correctness is proven, but speed is not measured because headless Chromium exposes SwiftShader software WebGPU rather than a real GPU.
• Lesson: GPU API correctness can be CI-proven, but performance claims require a hardware adapter measurement.
• Next action: Collect the documented real-device WebGPU vs WASM-SIMD benchmark before quoting a browser GPU speedup.
• Confidence: exact.

WASM register/cache-blocked matmul

• Evidence: microbench 3-4x, but end-to-end WASM training only 1.03-1.06x after forward blocking.
• Status: worked-with-caveat.
• Failure reason: The isolated microbench overstated impact because the real WASM training workload was threaded, already SIMD-vectorized, and dominated by backward matmuls.
• Lesson: Optimize and report on the real workload, not only on isolated single-kernel microbenches.
• Next action: Keep the bit-exact forward kernel for inference, but judge future perf work with end-to-end training and generation benches.
• Confidence: exact.

WASM backward_dA blocking

• Evidence: fwd+bwd-blocked reached small 90.3, medium 317.6, large 1057.4, xl 1643.6, or about 1.10-1.13x over naive.
• Status: worked-with-caveat.
• Failure reason: The backward_dA optimization improved end-to-end training, but the remaining backward_dB path still limits the total speedup.
• Lesson: Training speed follows the backward-pass bottleneck; forward-only wins are capped until dA and dB are addressed.
• Next action: Only continue matmul blocking if backward_dB or generation benches show enough remaining headroom.
• Confidence: exact.

Speculative-decoding heads Medusa/EAGLE smoke

• Evidence: baseline 232 tok/s; Medusa 50-step 202 tok/s at 21.7%; Medusa 300-step 375 tok/s at 20.8%; EAGLE-2 50-step 200 tok/s at 26.5%, all bit-identical where verified.
• Status: worked-with-caveat.
• Failure reason: The heads trained and verified losslessly, but acceptance stayed far below the paper recipes because this run lacked long training, logit distillation, tree attention, and a full EAGLE transformer draft block.
• Lesson: Speculative heads are a valid local speed path, but the production recipe is data/training heavy rather than a quick sidecar smoke.
• Next action: Revive only with self-generated data, logit-KL distillation, tree verification, and a real acceptance-rate gate.
• Confidence: exact.

MoE dense-compute smoke

• Evidence: Dense MLP 842 K params loss 6.09 -> 1.76 at 55.6 step/s; MoE 4 experts top-2 2.42 M params loss 5.95 -> 1.68 at 29.3 step/s.
• Status: worked-with-caveat.
• Failure reason: The MoE architecture, save/load, sample, eval, and inspect paths work, but the current compute path runs every expert and therefore does not deliver sparse-dispatch savings.
• Lesson: MoE is a capacity feature until sparse gather/scatter or grouped matmul lands; it should not be marketed as a compute-saving win yet.
• Next action: Return to MoE only with MLX scatter_add/grouped matmul support or a scoped custom Metal sparse-dispatch kernel.
• Confidence: exact.

CPU speedup bundle

• Evidence: all four items ON reached median 6.8 step/s vs 5.0 step/s baseline, +36%, with final loss 4.188-4.192 unchanged.
• Status: worked-with-caveat.
• Failure reason: The bundle produced a real host-side speedup, but QoS and compiled cosine LR were individually in the noise, prefetch stayed single-digit, and open CPU items remained.
• Lesson: The big win was fused accumulation inside the compiled trace; individual CPU tweaks need isolation before being credited.
• Next action: Use the bundle as shipped, but keep Rust-FFI BPE, preallocated CPU buffers, and SVD-on-CPU AMX as separate future attempts.
• Confidence: exact.

Cold-start bundle

• Evidence: mmap saved 10-20 ms and 25 MB peak RSS on demo.tinygpt; lazy embedding deferred 262.1 KB; package tests passed 43/43.
• Status: worked-with-caveat.
• Failure reason: The cold-start path improved the measured demo case, but the 250 MB target file was missing, async spinner overhead added about 30 ms, and persistent Metal pipeline caching was infeasible through public MLX-Swift APIs.
• Lesson: mmap and direct byte construction are good scaling levers, but large-model cold-start claims need large-model measurements.
• Next action: Re-measure on a real 50-250 MB checkpoint and keep persistent Metal cache work parked unless MLX-Swift exposes pipeline descriptors.
• Confidence: exact.

Gradient checkpointing Mac training

• Evidence: behemoth B=4 ctx=1024 peak memory dropped 27679 MB -> 17816 MB while step/s moved 0.4 -> 0.3; tiny memory worsened 161 MB -> 189 MB.
• Status: worked-with-caveat.
• Failure reason: Checkpointing recovered large-model memory but hurt small-model memory and speed, and the OOMGuard estimator still reports uncheckpointed activation projections.
• Lesson: Activation recompute is a scale-dependent lever; it should stay opt-in for models where memory, not step time, is the blocker.
• Next action: Update memory estimation for --grad-checkpoint and consider selective layer checkpointing before broad default use.
• Confidence: exact.

KV cache optimization bundle

• Evidence: prompt-cache hot run 1.005 s -> 0.106 s; pre-alloc decode stayed 706 -> 711 tok/s with physical cache flat at 6.3 MB.
• Status: worked-with-caveat.
• Failure reason: Persistent prompt cache delivered the TTFT win, but pre-alloc was kept opt-in, pre-alloc does not compose with KIVI or StreamingLLM, and some cache keying edge cases remain.
• Lesson: Reuse of prefilled KV is the user-visible win; cache storage policy changes need composition gates before becoming defaults.
• Next action: Keep prompt-cache as the main shipped path and revisit pre-alloc defaults only after KIVI/StreamingLLM interactions are designed.
• Confidence: exact.

YOCO cache-halving smoke

• Evidence: huge cache at 206 tokens dropped 5.1 MB -> 2.5 MB, while throughput moved 767 -> 676 tok/s.
• Status: worked-with-caveat.
• Failure reason: YOCO halved decode KV cache memory, but throughput dropped at ctx=256 and HF checkpoint round-trip or production-quality YOCO weights were not proven.
• Lesson: YOCO is a long-context memory lever, not an immediate short-context speedup.
• Next action: Treat YOCO as a long-context experiment until anchor tuning, HF round-trip, and long-context quality gates exist.
• Confidence: exact.

StreamingLLM + KIVI cache compression

• Evidence: StreamingLLM reduced 500-token cache 12.6 MB -> 6.4 MB; int8 KIVI matched greedy baseline 100% and int4 averaged 96% prefix match.
• Status: worked-with-caveat.
• Failure reason: The cache compression worked in smoke tests, but int4 precision still used int8 storage bytes, K scales recompute every step, and no held-out perplexity gate measured quality.
• Lesson: KV compression can preserve greedy decode locally, but storage format and quality gates matter before making long-context claims.
• Next action: Add packed int4 storage or grouped KIVI scales only after a cached-forward perplexity or trajectory-match eval exists.
• Confidence: exact.

Data-side sample packing and BPE-dropout

• Evidence: sample packing flattened CoV 0.582 -> 0.061; BPE-dropout 100-step smoke stayed stable with final loss 7.382 baseline vs 7.597 dropout.
• Status: worked-with-caveat.
• Failure reason: Sample packing’s histogram property was proven, but BPE-dropout was only a stability smoke and did not prove downstream generalization on this corpus.
• Lesson: Data-pipeline regularizers need behavioral evals, not just stable loss curves, before being treated as quality improvements.
• Next action: Use sample-packing where length bias matters, and evaluate BPE-dropout on rare-word or morphology-sensitive gates before promotion.
• Confidence: exact.

MLXFast SDPA and tied-embedding audit

• Evidence: audit found SDPA call sites and tied-embedding build/save/load paths clean; build succeeded and sample smoke reached 603 tok/s on demo and 261 tok/s on mega5.
• Status: worked.
• Lesson: The suspected SDPA mask/contiguity and tied-embedding duplication issues were not present.
• Next action: Keep this as positive verification; only reopen if new SDPA call sites or untied embedding save paths are added.
• Confidence: not-applicable.

Tool-Calling / Eval Infrastructure Attempts

A1 first tool-calling specialist harness

• Evidence: recipe and acceptance harness shipped; training run pending GPU Mac + local BFCL checkout.
• Status: not-tried.
• Lesson: The four-step specialist contract is reusable, but the model claim does not exist until BFCL training/eval runs.
• Next action: Run BFCL baseline, SFT, adapter eval, and +3pp gate once GPU/BFCL prerequisites are available.
• Confidence: not-applicable.

B1 SQL eval infrastructure

• Evidence: posttrainllm eval-sql shipped and smoke-tested; training/generation originally pending GPU.
• Status: worked-with-caveat.
• Failure reason: The scoring half existed before the live SQL factory run, but the original PRD did not include trained adapter evidence.
• Lesson: Execution accuracy over SQLite is the right reusable gate; infra alone is not a specialist result.
• Next action: Continue using eval-sql as the frozen SQL gate and attach run folders to model claims.
• Confidence: exact.

Primary SQL source docs:

• docs/specialists/b1-sql-poc.md
• docs/techniques/sql-technique-backlog.md
• docs/factory/public-artifacts.md
• Local run reports under runs/2026-07-02-* and runs/2026-07-03-sql-hygiene-dpo-qwen06/

Factory / Documentation Attempts

Factory run schema

• Evidence: docs/factory/run-schema.md defines the expected run folder.
• Status: worked-with-caveat.
• Failure reason: The schema exists, but real train/eval commands still need to emit every file automatically.
• Lesson: A schema is necessary but not sufficient; publish checks must enforce complete run folders.
• Next action: Wire live train/eval commands to produce schema-complete run folders automatically.
• Confidence: exact.

SQL factory run renderer

• Evidence: scripts/render_sql_factory_run.py renders canonical SQL report artifacts.
• Status: worked-with-caveat.
• Failure reason: The renderer is a useful bridge but not yet one universal factory command.
• Lesson: Rendered reports made SQL publishable, but the factory still needs command-level automation.
• Next action: Use the renderer until the canonical train/eval/report command can emit the same artifacts directly.
• Confidence: exact.

Public artifact registry

• Evidence: docs/factory/public-artifacts.md tracks artifacts, blockers, release state.
• Status: worked.
• Lesson: Public artifacts and blockers are first-class.
• Next action: Keep every artifact entry updated before a release push.
• Confidence: not-applicable.

Technique registry

• Evidence: docs/techniques/ distinguishes methods from recipes.
• Status: worked-with-caveat.
• Failure reason: The registry exists, but the next SQL recipe still needs validator/release enforcement before training claims.
• Lesson: Method names are not enough; recipes need failure mode, data, eval, slice gate, and stop rule.
• Next action: Require a recipe card before any new post-training run.
• Confidence: exact.

TrainLoop-style tooling

• Evidence: candidate choice, slice metrics, trace review, batch plan, LoRA geometry smokes pass.
• Status: worked-with-caveat.
• Failure reason: The tools exist and pass smokes, but most recipes have not yet been trained with them.
• Lesson: Discipline is valuable only when enforced in every reported run.
• Next action: Do not report the next SQL candidate without slice-metrics.json and trace_review.md.
• Confidence: exact.

Historical Coverage Limits

This ledger is now schema-complete for the structured attempts in docs/attempts.json: each entry has status, evidence, confidence, lesson, next action, and evidence sources. It is not a claim that every structured entry has equal certainty.

Current structured coverage:

This is also not a claim that every scratch experiment in the repo’s older archive has been normalized. Older archive/session docs still contain additional lessons, but they should enter this ledger only when we can attach a status, evidence, confidence, lesson, and next action without inventing history. See history-coverage-audit.md for the current coverage boundary.

Next Ledger Improvements

Required future fields for full historical precision:

• run id
• git commit / binary provenance
• dataset hash
• exact commands
• train/eval cost
• latency/RAM/tok-s
• report URL
• artifact URL