Every experiment. Including the ones that said no.
This is the complete evidence-backed history of posttrainllm. Nothing is queued or quietly “not tried.” Each historical question ends in a measured result or a final disposition, with the lesson preserved for your next fresh experiment.
tooling and smoke exist, but no frozen candidate-selection dataset, baseline, or model run was approved before project closure
Why it stopped
The proposed run never acquired a frozen target/eval contract, and running it retrospectively would start a new experiment rather than complete an existing measured claim.
Learning
Candidate selection remains a plausible future method, but an unfrozen recipe is not unfinished evidence for this project.
Disposition
Closed for this project. Reopen only as a fresh owner-led experiment after the learning phase with new data and a frozen gate.
plan renderer exists, but the required candidate-selection reward surface was never validated and no model run was authorized
Why it stopped
The experiment's explicit prerequisite did not exist, so an OAPL-style update would not have had a trustworthy reward and could not answer its intended question.
Learning
Offline policy updates are downstream of reward validation; skipping the prerequisite is not a useful experiment.
Disposition
Closed for this project. A future fresh experiment must validate its reward surface first.
The unchanged public weights reproduce a strong routed file-operations win but regress held-out breadth by five net paired cases, so they fail the general-successor gate.
Learning
ReST produced a real depth-and-safety specialist, not breadth recovery; routing is the win and general replacement is the failed hypothesis.
Disposition
Retain the package only behind the file-operations route; any broader successor needs breadth-preserving data or optimization on the same frozen ruler.
datasets, builder scripts, and planned recipes survive; no per-version score was recorded
Why it stopped
No per-version result was kept. The v11 ship gate names this directly: v1 through v10 cycled because there was no fixed bar, no fixed eval, and no immutable record of pass criteria.
Learning
An unmeasured version is indistinguishable from a version that never ran. Four consecutive attempts left no score behind.
Disposition
Do not attempt to reconstruct scores for v1-v4; treat the series as starting at v5, the first version with a recorded number.
10/19, 8/19, 2/19, 0/19, then 14/19 at 100 steps and 17/19 at 300 steps, reaching 19/19 after a serving-side minLength fix; 4/15 (26.7%) on fm-fixtures-v2
Why it stopped
The apparent collapse was two harness bugs, not training: SFT rows omitted the Pace system prompt that serve and eval both included, and a brace-counting JSON extractor was faulty.
Learning
A four-way score collapse traced to prompt mismatch between training and serving. Pin the prompt, schema, and grammar triple across train, serve, and eval before believing any comparison.
Disposition
Recover the full sweep with `git show d1efcb8^:docs/prds/factory-pace-planner-v6_1.md` before repeating any step-count ablation; it is the only one in the series.
tools-in-prompt serve, verb-enum grammar, and data/eval harnesses shipped; the SFT was never run
Why it stopped
The version was never trained, and later v8-v11 planner work replaced its proposed prompt/schema direction and supplied the relevant capability evidence.
Learning
v7 is a numbering gap, not a training version. The eleven-version count includes a version that was never trained.
Disposition
Closed as a numbering gap. Do not cite v7 as evidence or revive it outside a new owner-led experiment.
reported 11/15 (73.3%) on fm-fixtures-v2, later re-measured at 5/15 (33.3%) from the same weights and same fixtures
Why it stopped
The headline score did not reproduce. Re-running the same eval against the same baked weights yielded 33.3%, and the gap was root-caused as prompt and schema config drift rather than any model change.
Learning
The 73% phantom. A score is only a score if the prompt, schema, and grammar it was measured under are recorded with it.
Disposition
Treat 33.3% as the working v8 baseline and never compare across versions without pinning the harness config.
70% on new compose fixtures with non-compose held at the v8 level; as-shipped re-measurement 9/15 (60.0%); unhappy-path n=130 gave 0/40 ambiguous, 13/60 out-of-scope, 1/30 destructive
Why it stopped
It shipped as the specialist but never became Pace's runtime planner, and the unhappy-path drill put it at a capacity wall: 0% ambiguous and 3% destructive.
Learning
Winning the happy-path fixture says nothing about ambiguous, out-of-scope, or destructive inputs, which is where the product actually needed the planner.
Disposition
Keep as the last shipped specialist; Pace runs qwen3-30b-a3b instead.
full recipe, action registry, schema, and eval plan are committed; no score was ever recorded and it is not established that training completed
Why it stopped
No v10 score exists anywhere in the repository. The last status recorded is that training was pending, while the v10 corpus went on to become the base of v11.
Learning
v10 repeated v6's move of changing the output representation, and because it was never measured the v6 regression could not be checked for recurrence. Its unmeasured corpus was then inherited by v11.
Disposition
Do not reuse a v10-derived corpus without first measuring the representation change in isolation.
failed all six gate dimensions: BFCL pace-12 27.1% against a 40% bar, out-of-scope 0/30 against 80%, ambiguous 0/20 against 50%, destructive 5/10 against 90%, with train loss 0.001
Why it stopped
A 0.6B planner did not reach the fixed bar on any dimension despite the lowest training loss in the series, and the first v11 attempt was voided outright by the DoRA infrastructure failure.
Learning
Train loss 0.001 alongside 0% on out-of-scope and ambiguous inputs is the capacity ceiling stated plainly. The one-shot fixed gate was what finally ended the cycle.
Disposition
Do not train v12. The specialist planner track is closed and Pace ships qwen3-30b-a3b.
38 contrastive rows moved ambig only `0% -> 5%` while OOS regressed `80% -> 33%` in the retrospective; drilldown records the same attempt as `2%/40%/50%` on ambig/oos/destructive
Why it stopped
Small-corpus LoRA on a competent base caused catastrophic interference across planner dimensions.
Learning
Fine-tuning a strong base on a thin behavior slice requires coverage for every dimension that must not regress.
Disposition
Do not fine-tune the planner on narrow clarify rows unless the corpus also covers OOS, destructive, happy path, and breadth gates.
ambig moved `0% -> 20%`, while OOS dropped `78% -> 55%` and destructive stayed `67%`; not shipped on top of Gemma
Why it stopped
The shim improved ambiguity detection for the 4B floor but over-triggered and became unnecessary once Gemma reached the same ambig score with better OOS/destructive behavior.
Learning
Rule wrappers can expose the missing behavior, but they still need the same regression gates as model updates.
Disposition
Keep as an option only if shipping the 4B footprint floor; do not stack it on the stronger Gemma path.
larger browser presets hit Memory64 OOB; matching Node direct-export reproducer passed XL; `INITIAL_MEMORY=268435456` fixed up to XL while Behemoth still exercises growth
Why it stopped
The original retries treated the OOB as a model-size problem before the matching reproducer showed a browser pthread plus SharedArrayBuffer memory-growth race.
Learning
A reproducer must match the production host and call path before it can explain a browser-only WASM failure.
Disposition
Keep the higher initial memory for supported presets and only claim Behemoth after a SAB-view-aware growth path lands.
the historical curve remains unqualified except at Large, where a 2026-09-02 Apple M5 Pro ABBA receipt measured `10.67x` median WebGPU-over-WASM speedup with `4.72%` maximum paired final-loss drift and zero runtime errors
Why it stopped
The original headline made a shape-dependent result look universal, and the retained tree cannot establish that the historical curve used a real hardware adapter rather than a software fallback.
Learning
Qualify performance one frozen shape and machine at a time; a retained adapter identity plus alternated paired timings can convert one point without laundering the surrounding curve.
Disposition
Keep Large as the measured M5 Pro point; independently reproduce Small, Medium, and XL before promoting the historical curve.
a trained WebGPU run was lost because `webgpu/gpu_model.ts` had no checkpoint serialization; `exportState` was then implemented and worker plumbing updated
Why it stopped
The training path could produce useful weights that the WebGPU backend could not export, and auto-cleanup destroyed a real trained artifact.
Learning
Long-running browser training is not product-complete until the smallest export path is smoke-tested before the long run.
Disposition
Smoke export on a short WebGPU run before any long browser training claim.
Apple M5 Pro / Metal 3 ABBA run measured `10.67x` median Large-preset speedup over WASM with `4.72%` maximum paired final-loss drift and zero runtime errors
Learning
Hardware-qualified browser WebGPU can deliver a double-digit training-step win while preserving the frozen numerical-loss gate.
Disposition
Retain Large as the verified Apple M5 Pro result; Small, Medium, XL, other browsers, and other hardware require independent receipts.
microbench `3-4x`, but end-to-end WASM training only `1.03-1.06x` after forward blocking
Why it stopped
The isolated microbench overstated impact because the real WASM training workload was threaded, already SIMD-vectorized, and dominated by backward matmuls.
Learning
Optimize and report on the real workload, not only on isolated single-kernel microbenches.
Disposition
Keep the bit-exact forward kernel for inference, but judge future perf work with end-to-end training and generation benches.
baseline `232 tok/s`; Medusa 50-step `202 tok/s` at `21.7%`; Medusa 300-step `375 tok/s` at `20.8%`; EAGLE-2 50-step `200 tok/s` at `26.5%`, all bit-identical where verified
Why it stopped
The heads trained and verified losslessly, but acceptance stayed far below the paper recipes because this run lacked long training, logit distillation, tree attention, and a full EAGLE transformer draft block.
Learning
Speculative heads are a valid local speed path, but the production recipe is data/training heavy rather than a quick sidecar smoke.
Disposition
Revive only with self-generated data, logit-KL distillation, tree verification, and a real acceptance-rate gate.
Dense MLP `842 K` params loss `6.09 -> 1.76` at `55.6 step/s`; MoE 4 experts top-2 `2.42 M` params loss `5.95 -> 1.68` at `29.3 step/s`
Why it stopped
The MoE architecture, save/load, sample, eval, and inspect paths work, but the current compute path runs every expert and therefore does not deliver sparse-dispatch savings.
Learning
MoE is a capacity feature until sparse gather/scatter or grouped matmul lands; it should not be marketed as a compute-saving win yet.
Disposition
Return to MoE only with MLX scatter_add/grouped matmul support or a scoped custom Metal sparse-dispatch kernel.
all four items ON reached median `6.8 step/s` vs `5.0 step/s` baseline, `+36%`, with final loss `4.188-4.192` unchanged
Why it stopped
The bundle produced a real host-side speedup, but QoS and compiled cosine LR were individually in the noise, prefetch stayed single-digit, and open CPU items remained.
Learning
The big win was fused accumulation inside the compiled trace; individual CPU tweaks need isolation before being credited.
Disposition
Use the bundle as shipped, but keep Rust-FFI BPE, preallocated CPU buffers, and SVD-on-CPU AMX as separate future attempts.
mmap saved `10-20 ms` and `25 MB` peak RSS on `demo.tinygpt`; lazy embedding deferred `262.1 KB`; package tests passed `43/43`
Why it stopped
The cold-start path improved the measured demo case, but the 250 MB target file was missing, async spinner overhead added about `30 ms`, and persistent Metal pipeline caching was infeasible through public MLX-Swift APIs.
Learning
mmap and direct byte construction are good scaling levers, but large-model cold-start claims need large-model measurements.
Disposition
Re-measure on a real 50-250 MB checkpoint and keep persistent Metal cache work parked unless MLX-Swift exposes pipeline descriptors.
Checkpointing recovered large-model memory but hurt small-model memory and speed, and the OOMGuard estimator still reports uncheckpointed activation projections.
Learning
Activation recompute is a scale-dependent lever; it should stay opt-in for models where memory, not step time, is the blocker.
Disposition
Update memory estimation for `--grad-checkpoint` and consider selective layer checkpointing before broad default use.
prompt-cache hot run `1.005 s -> 0.106 s`; pre-alloc decode stayed `706 -> 711 tok/s` with physical cache flat at `6.3 MB`
Why it stopped
Persistent prompt cache delivered the TTFT win, but pre-alloc was kept opt-in, pre-alloc does not compose with KIVI or StreamingLLM, and some cache keying edge cases remain.
Learning
Reuse of prefilled KV is the user-visible win; cache storage policy changes need composition gates before becoming defaults.
Disposition
Keep prompt-cache as the main shipped path and revisit pre-alloc defaults only after KIVI/StreamingLLM interactions are designed.
StreamingLLM reduced 500-token cache `12.6 MB -> 6.4 MB`; int8 KIVI matched greedy baseline `100%` and int4 averaged `96%` prefix match
Why it stopped
The cache compression worked in smoke tests, but int4 precision still used int8 storage bytes, K scales recompute every step, and no held-out perplexity gate measured quality.
Learning
KV compression can preserve greedy decode locally, but storage format and quality gates matter before making long-context claims.
Disposition
Add packed int4 storage or grouped KIVI scales only after a cached-forward perplexity or trajectory-match eval exists.
audit found SDPA call sites and tied-embedding build/save/load paths clean; build succeeded and sample smoke reached `603 tok/s` on demo and `261 tok/s` on mega5
Learning
The suspected SDPA mask/contiguity and tied-embedding duplication issues were not present.
Disposition
Keep this as positive verification; only reopen if new SDPA call sites or untied embedding save paths are added.
the recipe and acceptance harness shipped; later Qwen3-4B file-ops distillation and BFCL-family evaluations exercised the specialist contract with measured results
Why it stopped
The original A1-specific adapter run was never executed, but its question and workflow were overtaken by later measured tool-calling specialists and should not be recreated for historical bookkeeping.
Learning
Reusable infrastructure can be superseded by stronger downstream evidence even when its first named model run never happened.
Disposition
Closed as superseded. Use the measured file-ops/ReST lineage for evidence and start any new tool-caller as a fresh experiment.
validation exact `6.16% random -> 10.54%` (+4.38 points), test `6.23% -> 10.33%` (+4.10); raw and guarded policies each won 0/6 games against random legal play
Why it stopped
Completion-only SFT learned a small held-out move preference but missed the frozen +10-point promotion gate and produced no full-game win advantage over random legal play; eight guard interventions converted no wins.
Learning
Legal constrained decoding and lower move-prediction loss are not game intelligence; require full-game transfer before scaling a tiny specialist.
Disposition
Do not run the 100k, 1M, or 2M stages under this recipe. Preserve the failed artifact and select a different bounded specialist target.
overall exactness moved 32/94 -> 36/94, while Pace regressed 10/28 -> 6/28, file operations stayed 0/6, and 3 out-of-scope prompts still produced actions
Why it stopped
Oracle task-family routing reduced schema size and latency but did not repair capability or safety; the target Pace slice became worse.
Learning
Catalog restriction is a latency lever, not a capability or safety rescue for this base model.
Disposition
Reject this integration path; require a materially different training hypothesis and a new frozen gate for any future Needle-derived work.
all four arms passed 100% tiny overfit, but public-dev exactness was 25.5%, 25.5%, 26.6%, and 27.7% versus the 34.0% incumbent; every arm produced 7-9 out-of-scope false actions and all 10 destructive bypasses
Why it stopped
The 45M model memorized the tiny gate but neither distractor-aware data nor explicit safety examples transferred to safe 94-case tool selection.
Learning
Tiny overfit proves wiring, not capacity; the interaction treatment recovered only two cases over plain training and could not learn destructive confirmation at this scale.
Disposition
Close 45M recipe search and advance the same frozen selection-and-safety task to the 1.7B class as a fresh experiment.
On the same eight LibriSpeech clips, browser Parakeet v3 scored 0/82 word errors versus native WhisperKit's 7/82, matched all four proper nouns, had zero repetition errors, and decoded 3.51x faster; its 33.84x median short-clip real-time factor missed the frozen 50x bar
Why it stopped
Quality, paired native latency, repetition, and offline-warm gates passed, but fixed dispatch overhead on 2-7 second clips held the preregistered median throughput below 50x; the optional first download is 652.7 MiB.
Learning
Browser WebGPU ASR can beat a native large-model incumbent on both bounded WER and latency while still failing an absolute throughput target; short-clip fixed cost and long-form throughput are different measurements.
Disposition
Close this frozen gate without relabeling it; batching, long-form amortization, Safari, or a larger WER set are fresh experiments.
the ordered volume ladder found its first reproducible boundary at 2,000 words/event or 8,000 submitted words/day
Why it stopped
The boundary occurred in the neutral arm and therefore measures raw context volume or context management, not personal-obligation semantics; full provider/model provenance is incomplete.
Learning
The supported result is a neutral context-volume boundary, not a psychological or family-tension effect.
Disposition
Closed as report-only boundary evidence with the interpretation constrained to raw volume.
100 stable candidates produced 86 admitted positions, but the best recorded frontier screen reached only 75% exact agreement
Why it stopped
The frontier calibration did not approach the required ceiling, external-model coverage was incomplete, and the model alias was mutable, so the suite could not be frozen as a fair ruler.
Learning
A benchmark candidate set is not a benchmark until the frontier ceiling validates the ruler.
Disposition
Closed as candidate-only evidence; design any future chess ruler independently after the learning phase.
pinned Sonnet matched random legal play at 0.995x mean score and pinned Opus reached only 1.058x over three complete pairs before a provider disconnect
Why it stopped
The proposed character interface did not expose a reliable frontier intelligence advantage, so training a 30-50M student would measure ruler noise.
Learning
Validate the teacher and interface before spending compute on a specialist; a failed ruler should stop training.
Disposition
Closed in the current character form; keep the negative benchmark replay public.
the arena retained four chess matches and seven complete 2048 pairs, while every rating remained unqualified
Why it stopped
Chess missed minimum-match and forfeit gates, while 2048 missed minimum-pair or complete-source gates; cross-game ranking would have overstated sparse development evidence.
Learning
A rating calculation is not a public rating until each underlying competition family passes its qualification contract.
Disposition
Closed as a transparent candidate report with no Elo, human-equivalence, or specialist-win claim.
123 development fixtures: student v1 58.5% full pass, reported MPS routing p50 16 ms.
Why it stopped
Teacher labels and limited coverage left single-target click routing weak. Synthetic sealed data and inspected development misses do not qualify a shipping gate.
Learning
Student v1 left single-target clicks weak; label policy and independent evaluation must precede further teacher distillation.
Disposition
Continue issue #200 with a label policy, fresh adjudicated acceptance cases, and per-class regression gates.
123 development fixtures: student v2 53.7% full pass, reported MPS routing p50 18 ms.
Why it stopped
Agreement filtering reduced development full pass rather than repairing label policy. Synthetic sealed data and inspected development misses do not qualify a shipping gate.
Learning
Agreement filtering lowered student v2 full pass to 53.7%; agreement alone does not repair a disputed action policy.
Disposition
Continue issue #200 with a label policy, fresh adjudicated acceptance cases, and per-class regression gates.
123 development fixtures: student v3 65.0% full pass, reported MPS routing p50 14 ms.
Why it stopped
Goal-phrased data improved clicks but full pass remained below the local 71.5% baseline and RESPOND regressed. Synthetic sealed data and inspected development misses do not qualify a shipping gate.
Learning
Goal-phrased training improved student v3 clicks but regressed RESPOND; report per-class gates alongside aggregate full pass.
Disposition
Continue issue #200 with a label policy, fresh adjudicated acceptance cases, and per-class regression gates.
Jeff tools with 32 Pace options: 38.2% full pass, 10.0% ASK_USER, p50 143 ms; 36 repeat guards.
Why it stopped
The unchanged label contract remained weak on step control and clarification.
Learning
Jeff tools with the existing label contract remained weak on clarification and step control; a tool-oriented model is not a qualified policy replacement.
Disposition
Reject the 32-option substitution; preserve a separate upstream-format trial.
96 generated policy cases: rules 96/96, Qwen3.5-4B 4-bit 48/96 with zero errors; Qwen route p50/p95 394.7507/953.5587 ms, model bytes 3061316115.
Why it stopped
Qwen failed all ASK_USER, RESPOND and named-tool cases. The rules-perfect template set establishes regression behavior only, not real-task generalization.
Learning
Preserve raw failures, generated provenance and unknown calibration/hardware measures; a fast or rules-perfect result does not qualify a planner replacement.
Disposition
KEEP the current native planner. The owner withdrew the 400-500 handwritten-case requirement; any future learned replacement needs independent task evidence, not relabeled generated gold.