> Canonical page: https://posttrainllm.com/experiments closed evidence lab · schema v3 # Every experiment. Including the ones that said no. This is the complete evidence-backed history of posttrainllm. Nothing is queued or quietly “not tried.” Each historical question ends in a measured result or a final disposition, with the lesson preserved for your next fresh experiment. 90 accounted attempts 0 unresolved attempts 79 exact-evidence records 14 experiment families outcomes ## The shape of the work Negative results remain first-class evidence. 5 worked 40 worked · caveat 13 failed 6 regressed 9 inconclusive 2 superseded 14 rejected 1 blocked search the lab ## All recorded attempts Showing all 90 Search Family All families apple fm architecture archive model autocorrect browser product chess factory docs file ops game benchmarks offhours pace planner runtime perf sql tool calling Reset 0.833` on 6-row fixture the sft loop mechanics work, but eval splits must be frozen before claiming quality. use only as a loop sanity check; rely on heldout sql evals for real claims. sql-execution sft lora dora qwen3-0.6b" data-astro-cid-jsfgzirp> sql worked · caveat ### [Toy SQL SFT](https://posttrainllm.com/experiments/sql-toy-sft-r4) - sft - lora - dora **Evidence** execution `0.167 -> 0.833` on 6-row fixture **Why it stopped** Train/eval overlap made the apparent improvement non-shippable. **Learning** The SFT loop mechanics work, but eval splits must be frozen before claiming quality. **Disposition** Use only as a loop sanity check; rely on heldout SQL evals for real claims. 0.860`, exact `0.140 -> 0.840` the synthetic sql adapter is the current synthetic incumbent, but joins remain weaker than single-table rows. keep as the synthetic incumbent until a candidate beats it on the same heldout gate. sql-execution sft lora dora qwen3-0.6b" data-astro-cid-jsfgzirp> sql worked ### [Expanded synthetic SQL SFT](https://posttrainllm.com/experiments/sql-expanded-synthetic-sft-r4) - sft - lora - dora **Evidence** execution `0.160 -> 0.860`, exact `0.140 -> 0.840` **Learning** The synthetic SQL adapter is the current synthetic incumbent, but joins remain weaker than single-table rows. **Disposition** Keep as the synthetic incumbent until a candidate beats it on the same heldout gate. sql failed ### [Public b-mc2 v1](https://posttrainllm.com/experiments/sql-public-bmc2-v1) - sft - lora **Evidence** exact `0.344` vs T5-small `0.484` **Why it stopped** The public-style dataset was too small to beat the T5-small baseline. **Learning** Public benchmark adaptation needs more coverage before tuning knobs matter. **Disposition** Scale public-style rows and re-evaluate against the same exact gate. sql failed ### [Public b-mc2 v2](https://posttrainllm.com/experiments/sql-public-bmc2-v2) - sft - lora **Evidence** exact `0.031` **Why it stopped** Increasing rows, rank, and steps together made the recipe worse and confounded the cause. **Learning** Do not change data volume, rank, and training length in one uncontrolled jump. **Disposition** Use smaller controlled deltas with fixed eval and comparable training settings. sql failed ### [Public b-mc2 v3](https://posttrainllm.com/experiments/sql-public-bmc2-v3) - sft - lora **Evidence** exact `0.422` **Why it stopped** Lower LR and fewer steps recovered quality but still missed the public baseline. **Learning** The public recipe was directionally better, but not enough for a win. **Disposition** Try targeted data weighting instead of only global hyperparameter changes. sql worked · caveat ### [Public b-mc2 v4](https://posttrainllm.com/experiments/sql-public-bmc2-v4) - sft - lora **Evidence** exact `0.531` vs T5-small `0.484` **Why it stopped** The public win did not preserve synthetic SQL execution behavior. **Learning** A narrow public exact-match win can still damage the incumbent specialist behavior. **Disposition** Route or compose specialists instead of forcing one adapter to satisfy both gates. sql regressed ### [Public v4 synthetic regression check](https://posttrainllm.com/experiments/sql-public-v4-synthetic-regression) - evaluation **Evidence** synthetic execution `0.240` vs incumbent `0.860` **Why it stopped** Public-style training overfit schema-linking patterns that did not transfer to synthetic SQLite domains. **Learning** Every public benchmark gain needs a breadth/regression gate before packaging. **Disposition** Keep public and synthetic adapters separate unless a candidate passes both gates. sql failed ### [Blended public+synthetic SFT](https://posttrainllm.com/experiments/sql-blended-public-synthetic-v1) - sft - lora **Evidence** public `0.297`, synthetic execution `0.560` **Why it stopped** Naive mixture training caused interference and output-format collapse. **Learning** Mixing domains is not automatically regularization; it can reduce both capabilities. **Disposition** Use routing, data-mix search, or explicit slice gates before another blended adapter. sql failed ### [Static multi-LoRA composition](https://posttrainllm.com/experiments/sql-static-multilora-composition) - multi-lora-composition **Evidence** best public-pass setting still synthetic-failed **Why it stopped** Fixed adapter weights produced a smooth tradeoff but no setting passed both gates. **Learning** Static composition cannot solve domain conflict when the right behavior depends on input type. **Disposition** Prefer input-aware routing over global composition weights. sql worked · caveat ### [Routed public + synthetic adapters](https://posttrainllm.com/experiments/sql-routed-v1) - routing **Evidence** public exact `0.531`, synthetic execution `0.860` **Why it stopped** The routed artifact lacks public execution benchmarking and perf/package gates. **Learning** Routing preserves both measured slices, but artifact status still depends on public execution and performance evidence. **Disposition** Add public execution gate, latency/RAM/tok-s measurement, and a package decision. 0.080`, clean-sql `0.000 -> 0.000` preference tuning needs reference anchoring, smaller updates, and composed-eval checks. retry with reference-anchored dpo or much lower lr/step count on the same frozen pairs. output-format simpo lora qwen3-0.6b" data-astro-cid-jsfgzirp> sql failed ### [Hygiene SimPO/DPO](https://posttrainllm.com/experiments/sql-hygiene-simpo-dpo) - simpo - lora **Evidence** execution `0.860 -> 0.080`, clean-SQL `0.000 -> 0.000` **Why it stopped** Ref-free SimPO update collapsed the policy and produced degenerate fenced/prose outputs. **Learning** Preference tuning needs reference anchoring, smaller updates, and composed-eval checks. **Disposition** Retry with reference-anchored DPO or much lower LR/step count on the same frozen pairs. sql rejected ### [SQL candidate selection](https://posttrainllm.com/experiments/sql-candidate-selection) - candidate-selection **Evidence** tooling and smoke exist, but no frozen candidate-selection dataset, baseline, or model run was approved before project closure **Why it stopped** The proposed run never acquired a frozen target/eval contract, and running it retrospectively would start a new experiment rather than complete an existing measured claim. **Learning** Candidate selection remains a plausible future method, but an unfrozen recipe is not unfinished evidence for this project. **Disposition** Closed for this project. Reopen only as a fresh owner-led experiment after the learning phase with new data and a frozen gate. sql rejected ### [Offline rollout / OAPL-style SQL update](https://posttrainllm.com/experiments/sql-offline-rollout-oapl) - rlvr **Evidence** plan renderer exists, but the required candidate-selection reward surface was never validated and no model run was authorized **Why it stopped** The experiment's explicit prerequisite did not exist, so an OAPL-style update would not have had a trustworthy reward and could not answer its intended question. **Learning** Offline policy updates are downstream of reward validation; skipping the prerequisite is not a useful experiment. **Disposition** Closed for this project. A future fresh experiment must validate its reward surface first. sql rejected ### [Controlled SQL LoRA rank sweep](https://posttrainllm.com/experiments/sql-controlled-lora-rank-sweep) - lora **Evidence** no controlled sweep was run because no next target/eval was frozen after the confounded historical comparisons **Why it stopped** Without one active frozen recipe, a rank sweep would measure a new arbitrary setup rather than resolve the historical confound. **Learning** Hyperparameter sweeps are valid only inside one frozen experiment contract; they are not project-completion chores. **Disposition** Closed for this project. Any future rank study begins as a new experiment with fixed data, seed, steps, learning rate, and eval. 1.00`; breadth `0.596 -> 0.423` the artifact is useful only with routed-only positioning; it should not be sold as a general planner. keep routed-only warnings prominent and add loader/pull smoke before wiring consumers. file-ops-depth distill qwen3-4b" data-astro-cid-jsfgzirp> file ops worked · caveat ### [Qwen3-4B file-ops distilled specialist](https://posttrainllm.com/experiments/fileops-qwen3-4b-distilled) - distill **Evidence** file-ops hard gate `0.58 -> 1.00`; breadth `0.596 -> 0.423` **Why it stopped** The specialist improved the narrow file-ops gate but regressed out-of-domain breadth. **Learning** The artifact is useful only with routed-only positioning; it should not be sold as a general planner. **Disposition** Keep routed-only warnings prominent and add loader/pull smoke before wiring consumers. 12/12 rest` (+25 points), breadth `30/45 stock -> 25/45 rest` (-11.1 points), 10 fewer schema/side-effect events, and 2.42x faster depth wall time rest produced a real depth-and-safety specialist, not breadth recovery; routing is the win and general replacement is the failed hypothesis. retain the package only behind the file-operations route; any broader successor needs breadth-preserving data or optimization on the same frozen ruler. breadth-retention rest distill qwen3-4b" data-astro-cid-jsfgzirp> file ops worked · caveat ### [Qwen3-4B ReST fused breadth recovery variant](https://posttrainllm.com/experiments/fileops-qwen3-4b-rest-fused) - rest - distill **Evidence** fresh frozen run: depth `9/12 stock -> 12/12 ReST` (+25 points), breadth `30/45 stock -> 25/45 ReST` (-11.1 points), 10 fewer schema/side-effect events, and 2.42x faster depth wall time **Why it stopped** The unchanged public weights reproduce a strong routed file-operations win but regress held-out breadth by five net paired cases, so they fail the general-successor gate. **Learning** ReST produced a real depth-and-safety specialist, not breadth recovery; routing is the win and general replacement is the failed hypothesis. **Disposition** Retain the package only behind the file-operations route; any broader successor needs breadth-preserving data or optimization on the same frozen ruler. file ops regressed ### [Qwen3-4B multibackend distilled variant](https://posttrainllm.com/experiments/fileops-qwen3-4b-multibackend-distilled) - distill **Evidence** depth `100%`, breadth `31%` **Why it stopped** Negative transfer damaged breadth even though depth remained high. **Learning** Multi-backend distillation can over-specialize and needs breadth gates by default. **Disposition** Keep as failed comparison evidence; do not promote without a new breadth-preserving recipe. archive model inconclusive ### [VibeThinker 3B agentic distilled variant](https://posttrainllm.com/experiments/vibethinker-3b-agentic-distilled) - distill **Evidence** weights preserved; current eval promotion missing **Why it stopped** The weights exist, but the docs do not preserve a current factory eval proving a win. **Learning** Preserved weights are not a ship decision. **Disposition** Run the factory eval gate and publish a before/after report before using it as a specialist package. apple fm failed ### [Apple on-device Foundation Models action-grounding probe](https://posttrainllm.com/experiments/apple-foundation-models-action-grounding) - evaluation **Evidence** BFCL agentic `25%` full catalog / approximately `0%` compact; Pace planner action-grounding `13%` **Why it stopped** Apple's on-device model could not ground actions well enough and its 4096-token context could not hold a real tool catalog. **Learning** Apple Foundation Models are useful as a free routing floor and intel source, not as the core capability dependency. **Disposition** Keep Core ML for our own weights as an optional future battery optimization, not a model-dependency bet. pace planner inconclusive ### [Pace planner v1-v4 early versions](https://posttrainllm.com/experiments/pace-planner-v1-v4-early) - sft - lora **Evidence** datasets, builder scripts, and planned recipes survive; no per-version score was recorded **Why it stopped** No per-version result was kept. The v11 ship gate names this directly: v1 through v10 cycled because there was no fixed bar, no fixed eval, and no immutable record of pass criteria. **Learning** An unmeasured version is indistinguishable from a version that never ran. Four consecutive attempts left no score behind. **Disposition** Do not attempt to reconstruct scores for v1-v4; treat the series as starting at v5, the first version with a recorded number. pace planner worked · caveat ### [Pace planner v5 hand-crafted gold + teacher paraphrase](https://posttrainllm.com/experiments/pace-planner-v5) - sft - lora **Evidence** 17/19 (89%) on fm-fixtures v1 and 6/15 (40.0%) on fm-fixtures-v2 **Why it stopped** The 17/19 was measuring format compliance, not capability: a rule-based FakePace endpoint scored 19/19 on the same fixture set. **Learning** A high score on a fixture a deterministic endpoint can also ace is a measurement of the harness, not the model. **Disposition** Treat fm-fixtures v1 as retired; only fm-fixtures-v2 numbers are comparable. pace planner regressed ### [Pace planner v6 label-based output](https://posttrainllm.com/experiments/pace-planner-v6) - sft - lora **Evidence** 14/19 (74%) on fm-fixtures v1 and 3/15 (20.0%) on fm-fixtures-v2, 20pp below v5 **Why it stopped** The label-based SFT destroyed capability, not just output format. **Learning** Changing the output representation is a capability change, not a formatting change. **Disposition** Revert to the v5 corpus for further augmentation work. pace planner worked · caveat ### [Pace planner v6.1 four-adapter repair sweep](https://posttrainllm.com/experiments/pace-planner-v6-1) - sft - lora **Evidence** 10/19, 8/19, 2/19, 0/19, then 14/19 at 100 steps and 17/19 at 300 steps, reaching 19/19 after a serving-side minLength fix; 4/15 (26.7%) on fm-fixtures-v2 **Why it stopped** The apparent collapse was two harness bugs, not training: SFT rows omitted the Pace system prompt that serve and eval both included, and a brace-counting JSON extractor was faulty. **Learning** A four-way score collapse traced to prompt mismatch between training and serving. Pin the prompt, schema, and grammar triple across train, serve, and eval before believing any comparison. **Disposition** Recover the full sweep with `git show d1efcb8^:docs/prds/factory-pace-planner-v6_1.md` before repeating any step-count ablation; it is the only one in the series. pace planner superseded ### [Pace planner v7 tools-in-prompt](https://posttrainllm.com/experiments/pace-planner-v7) - sft - lora **Evidence** tools-in-prompt serve, verb-enum grammar, and data/eval harnesses shipped; the SFT was never run **Why it stopped** The version was never trained, and later v8-v11 planner work replaced its proposed prompt/schema direction and supplied the relevant capability evidence. **Learning** v7 is a numbering gap, not a training version. The eleven-version count includes a version that was never trained. **Disposition** Closed as a numbering gap. Do not cite v7 as evidence or revive it outside a new owner-led experiment. pace planner inconclusive ### [Pace planner v8 augmented corpus](https://posttrainllm.com/experiments/pace-planner-v8) - sft - lora **Evidence** reported 11/15 (73.3%) on fm-fixtures-v2, later re-measured at 5/15 (33.3%) from the same weights and same fixtures **Why it stopped** The headline score did not reproduce. Re-running the same eval against the same baked weights yielded 33.3%, and the gap was root-caused as prompt and schema config drift rather than any model change. **Learning** The 73% phantom. A score is only a score if the prompt, schema, and grammar it was measured under are recorded with it. **Disposition** Treat 33.3% as the working v8 baseline and never compare across versions without pinning the harness config. pace planner worked · caveat ### [Pace planner v9 compose](https://posttrainllm.com/experiments/pace-planner-v9) - sft - lora **Evidence** 70% on new compose fixtures with non-compose held at the v8 level; as-shipped re-measurement 9/15 (60.0%); unhappy-path n=130 gave 0/40 ambiguous, 13/60 out-of-scope, 1/30 destructive **Why it stopped** It shipped as the specialist but never became Pace's runtime planner, and the unhappy-path drill put it at a capacity wall: 0% ambiguous and 3% destructive. **Learning** Winning the happy-path fixture says nothing about ambiguous, out-of-scope, or destructive inputs, which is where the product actually needed the planner. **Disposition** Keep as the last shipped specialist; Pace runs qwen3-30b-a3b instead. pace planner inconclusive ### [Pace planner v10 parameterized actions](https://posttrainllm.com/experiments/pace-planner-v10) - sft - lora **Evidence** full recipe, action registry, schema, and eval plan are committed; no score was ever recorded and it is not established that training completed **Why it stopped** No v10 score exists anywhere in the repository. The last status recorded is that training was pending, while the v10 corpus went on to become the base of v11. **Learning** v10 repeated v6's move of changing the output representation, and because it was never measured the v6 regression could not be checked for recurrence. Its unmeasured corpus was then inherited by v11. **Disposition** Do not reuse a v10-derived corpus without first measuring the representation change in isolation. pace planner failed ### [Pace planner v11 ship-gate run](https://posttrainllm.com/experiments/pace-planner-v11) - sft - lora **Evidence** failed all six gate dimensions: BFCL pace-12 27.1% against a 40% bar, out-of-scope 0/30 against 80%, ambiguous 0/20 against 50%, destructive 5/10 against 90%, with train loss 0.001 **Why it stopped** A 0.6B planner did not reach the fixed bar on any dimension despite the lowest training loss in the series, and the first v11 attempt was voided outright by the DoRA infrastructure failure. **Learning** Train loss 0.001 alongside 0% on out-of-scope and ambiguous inputs is the capacity ceiling stated plainly. The one-shot fixed gate was what finally ended the cycle. **Disposition** Do not train v12. The specialist planner track is closed and Pace ships qwen3-30b-a3b. 5%` while oos regressed `80% -> 33%` in the retrospective; drilldown records the same attempt as `2%/40%/50%` on ambig/oos/destructive fine-tuning a strong base on a thin behavior slice requires coverage for every dimension that must not regress. do not fine-tune the planner on narrow clarify rows unless the corpus also covers oos, destructive, happy path, and breadth gates. ambiguity-detection lora qwen3-4b" data-astro-cid-jsfgzirp> pace planner regressed ### [Pace clarify-v1 Qwen3-4B LoRA](https://posttrainllm.com/experiments/pace-clarify-v1-qwen4b-lora) - lora **Evidence** 38 contrastive rows moved ambig only `0% -> 5%` while OOS regressed `80% -> 33%` in the retrospective; drilldown records the same attempt as `2%/40%/50%` on ambig/oos/destructive **Why it stopped** Small-corpus LoRA on a competent base caused catastrophic interference across planner dimensions. **Learning** Fine-tuning a strong base on a thin behavior slice requires coverage for every dimension that must not regress. **Disposition** Do not fine-tune the planner on narrow clarify rows unless the corpus also covers OOS, destructive, happy path, and breadth gates. 20%`, while oos dropped `78% -> 55%` and destructive stayed `67%`; not shipped on top of gemma rule wrappers can expose the missing behavior, but they still need the same regression gates as model updates. keep as an option only if shipping the 4b footprint floor; do not stack it on the stronger gemma path. ambiguity-detection two-stage-shim qwen3-4b" data-astro-cid-jsfgzirp> pace planner worked · caveat ### [Pace two-stage ambiguity detector shim v2](https://posttrainllm.com/experiments/pace-two-stage-ambiguity-shim-v2) - two-stage-shim **Evidence** ambig moved `0% -> 20%`, while OOS dropped `78% -> 55%` and destructive stayed `67%`; not shipped on top of Gemma **Why it stopped** The shim improved ambiguity detection for the 4B floor but over-triggered and became unnecessary once Gemma reached the same ambig score with better OOS/destructive behavior. **Learning** Rule wrappers can expose the missing behavior, but they still need the same regression gates as model updates. **Disposition** Keep as an option only if shipping the 4B footprint floor; do not stack it on the stronger Gemma path. = 1024 and stop on ` ` only if gemma or the locked planner fails in production. unhappy-path-handling evaluation qwen3-4b-thinking deepseek-r1-distill-qwen-7b" data-astro-cid-jsfgzirp> pace planner inconclusive ### [Pace reasoning-model unhappy-path drill](https://posttrainllm.com/experiments/pace-thinking-models-unhappy-path) - evaluation **Evidence** Qwen3-4B-Thinking scored `0%/10%/0%` and DeepSeek-R1-Distill-Qwen-7B scored `0%/3%/0%` on ambig/oos/destructive under a 300-token cap **Why it stopped** The evaluation budget was consumed by think traces, so the apparent zero scores were not a fair model-quality verdict. **Learning** Reasoning models need adequate max tokens and a stop-on-think strategy before interpreting planner scores. **Disposition** Re-run with max_tokens >= 1024 and stop on ` ` only if Gemma or the locked planner fails in production. pace planner worked · caveat ### [Pace Gemma-3-12B unhappy-path drill winner](https://posttrainllm.com/experiments/pace-gemma-3-12b-drill-winner) - evaluation **Evidence** Gemma-3-12B scored `22%` ambig, `82%` OOS, `77%` destructive and won all three unhappy-path dimensions in the n=130 drill **Why it stopped** Gemma won the drill but ambiguity remained poor in absolute terms and the decision was later superseded by the general planner lock. **Learning** A stronger zero-shot base beat the local fine-tunes, but model-selection evidence can be superseded by later broader gates. **Disposition** Keep as historical unhappy-path evidence; prefer current `PROJECT_STATUS.md`/`docs/NEXT.md` for active planner decisions. pace planner worked · caveat ### [Pace Qwen3.5 challenger round](https://posttrainllm.com/experiments/pace-qwen35-challenger-round) - evaluation **Evidence** Qwen3.5-9B scored `18%/97%/63%`; Qwen3.5-4B scored `15%/55%/70%` on ambig/oos/destructive; champion stood **Why it stopped** Qwen3.5 improved some dimensions, especially OOS for 9B, but did not beat the stored champion across the full swap decision. **Learning** Single-dimension wins do not justify a planner swap when another required dimension regresses. **Disposition** Use `scripts/pipelines/eval_planner.sh` for future challengers and keep swap/no-swap verdicts tied to all required dimensions. browser product worked · caveat ### [Browser Memory64 large-preset runs](https://posttrainllm.com/experiments/browser-memory64-large-presets) **Evidence** larger browser presets hit Memory64 OOB; matching Node direct-export reproducer passed XL; `INITIAL_MEMORY=268435456` fixed up to XL while Behemoth still exercises growth **Why it stopped** The original retries treated the OOB as a model-size problem before the matching reproducer showed a browser pthread plus SharedArrayBuffer memory-growth race. **Learning** A reproducer must match the production host and call path before it can explain a browser-only WASM failure. **Disposition** Keep the higher initial memory for supported presets and only claim Behemoth after a SAB-view-aware growth path lands. browser product worked · caveat ### [Browser speedup headline curve](https://posttrainllm.com/experiments/browser-speedup-headline-curve) **Evidence** the historical curve remains unqualified except at Large, where a 2026-09-02 Apple M5 Pro ABBA receipt measured `10.67x` median WebGPU-over-WASM speedup with `4.72%` maximum paired final-loss drift and zero runtime errors **Why it stopped** The original headline made a shape-dependent result look universal, and the retained tree cannot establish that the historical curve used a real hardware adapter rather than a software fallback. **Learning** Qualify performance one frozen shape and machine at a time; a retained adapter identity plus alternated paired timings can convert one point without laundering the surrounding curve. **Disposition** Keep Large as the measured M5 Pro point; independently reproduce Small, Medium, and XL before promoting the historical curve. browser product worked · caveat ### [Browser default corpus fix](https://posttrainllm.com/experiments/browser-default-corpus-fix) **Evidence** the old default corpus was `863 bytes`; a Huge model memorized it with train loss `0.14`; default changed to TinyShakespeare at about `1.1 MB` **Why it stopped** The demo was trying to prove learning with a corpus so small that memorization looked like training success. **Learning** Default data is part of the product claim; a training demo's default corpus must produce non-memorized samples. **Disposition** Keep corpus defaults in the same parity review as model and optimizer defaults. browser product worked · caveat ### [Browser learning-rate default drift](https://posttrainllm.com/experiments/browser-lr-default-drift) **Evidence** browser default LR was `3e-3` while Python reference used `3e-4`; training plateaued around loss `2.45` until the browser default was changed **Why it stopped** Reference/kernel parity tests covered math drift but not hyperparameter default drift. **Learning** Reference defaults are part of correctness, not just documentation. **Disposition** Add or keep a config-default parity check whenever browser and reference paths share a training story. browser product worked · caveat ### [Browser WebGPU export serialization](https://posttrainllm.com/experiments/browser-webgpu-export-serialization) **Evidence** a trained WebGPU run was lost because `webgpu/gpu_model.ts` had no checkpoint serialization; `exportState` was then implemented and worker plumbing updated **Why it stopped** The training path could produce useful weights that the WebGPU backend could not export, and auto-cleanup destroyed a real trained artifact. **Learning** Long-running browser training is not product-complete until the smallest export path is smoke-tested before the long run. **Disposition** Smoke export on a short WebGPU run before any long browser training claim. browser product rejected ### [Browser generation streaming and KV cache](https://posttrainllm.com/experiments/browser-generation-streaming-kv-cache) **Evidence** tokens/sec counter was added, but browser generation still lacked KV cache and true token streaming; task `#72` was logged **Why it stopped** The browser track completed without implementing this feature, and no measured claim depends on pretending the typewriter animation is true streaming. **Learning** A typewriter animation over an already-complete response is not true streaming. **Disposition** Closed for this project with the limitation explicit. A future browser product may reopen true streaming as a new feature. 305 ms/step`, capable `632 -> ~640 ms/step` hot-path hygiene is not the same as a performance win; measure the real step time before claiming speedup. keep the allocation cleanup, but do not prioritize allocator work until profiling shows allocation cost again. " data-astro-cid-jsfgzirp> runtime perf worked · caveat ### [WASM backward-scratch reuse](https://posttrainllm.com/experiments/wasm-backward-scratch-reuse) **Evidence** standard `304 -> 305 ms/step`, capable `632 -> ~640 ms/step` **Why it stopped** Caching backward scratch buffers removed hot-path allocations but produced no measurable speedup because compute, not malloc, was the bottleneck. **Learning** Hot-path hygiene is not the same as a performance win; measure the real step time before claiming speedup. **Disposition** Keep the allocation cleanup, but do not prioritize allocator work until profiling shows allocation cost again. 191 ms/step`, capable `632 -> 391 ms/step`, net `1.6x` speedup emscripten/llvm autovectorization was the right browser-training speed lever before hand-written kernels. keep simd enabled by default and use the wasm smoke gate to protect training correctness. " data-astro-cid-jsfgzirp> runtime perf worked ### [WASM SIMD training speedup](https://posttrainllm.com/experiments/wasm-simd-training-speedup) **Evidence** standard `304 -> 191 ms/step`, capable `632 -> 391 ms/step`, net `1.6x` speedup **Learning** Emscripten/LLVM autovectorization was the right browser-training speed lever before hand-written kernels. **Disposition** Keep SIMD enabled by default and use the WASM smoke gate to protect training correctness. runtime perf worked ### [Browser WebGPU training loop](https://posttrainllm.com/experiments/browser-webgpu-training-loop) **Evidence** Apple M5 Pro / Metal 3 ABBA run measured `10.67x` median Large-preset speedup over WASM with `4.72%` maximum paired final-loss drift and zero runtime errors **Learning** Hardware-qualified browser WebGPU can deliver a double-digit training-step win while preserving the frozen numerical-loss gate. **Disposition** Retain Large as the verified Apple M5 Pro result; Small, Medium, XL, other browsers, and other hardware require independent receipts. runtime perf worked · caveat ### [WASM register/cache-blocked matmul](https://posttrainllm.com/experiments/wasm-register-cache-blocked-matmul) **Evidence** microbench `3-4x`, but end-to-end WASM training only `1.03-1.06x` after forward blocking **Why it stopped** The isolated microbench overstated impact because the real WASM training workload was threaded, already SIMD-vectorized, and dominated by backward matmuls. **Learning** Optimize and report on the real workload, not only on isolated single-kernel microbenches. **Disposition** Keep the bit-exact forward kernel for inference, but judge future perf work with end-to-end training and generation benches. runtime perf worked · caveat ### [WASM backward_dA blocking](https://posttrainllm.com/experiments/wasm-backward-da-blocking) **Evidence** fwd+bwd-blocked reached small `90.3`, medium `317.6`, large `1057.4`, xl `1643.6`, or about `1.10-1.13x` over naive **Why it stopped** The backward_dA optimization improved end-to-end training, but the remaining backward_dB path still limits the total speedup. **Learning** Training speed follows the backward-pass bottleneck; forward-only wins are capped until dA and dB are addressed. **Disposition** Only continue matmul blocking if backward_dB or generation benches show enough remaining headroom. runtime perf worked · caveat ### [Speculative-decoding heads Medusa/EAGLE smoke](https://posttrainllm.com/experiments/speculative-heads-medusa-eagle-smoke) **Evidence** baseline `232 tok/s`; Medusa 50-step `202 tok/s` at `21.7%`; Medusa 300-step `375 tok/s` at `20.8%`; EAGLE-2 50-step `200 tok/s` at `26.5%`, all bit-identical where verified **Why it stopped** The heads trained and verified losslessly, but acceptance stayed far below the paper recipes because this run lacked long training, logit distillation, tree attention, and a full EAGLE transformer draft block. **Learning** Speculative heads are a valid local speed path, but the production recipe is data/training heavy rather than a quick sidecar smoke. **Disposition** Revive only with self-generated data, logit-KL distillation, tree verification, and a real acceptance-rate gate. 1.76` at `55.6 step/s`; moe 4 experts top-2 `2.42 m` params loss `5.95 -> 1.68` at `29.3 step/s` moe is a capacity feature until sparse gather/scatter or grouped matmul lands; it should not be marketed as a compute-saving win yet. return to moe only with mlx scatter_add/grouped matmul support or a scoped custom metal sparse-dispatch kernel. " data-astro-cid-jsfgzirp> architecture worked · caveat ### [MoE dense-compute smoke](https://posttrainllm.com/experiments/moe-dense-compute-smoke) **Evidence** Dense MLP `842 K` params loss `6.09 -> 1.76` at `55.6 step/s`; MoE 4 experts top-2 `2.42 M` params loss `5.95 -> 1.68` at `29.3 step/s` **Why it stopped** The MoE architecture, save/load, sample, eval, and inspect paths work, but the current compute path runs every expert and therefore does not deliver sparse-dispatch savings. **Learning** MoE is a capacity feature until sparse gather/scatter or grouped matmul lands; it should not be marketed as a compute-saving win yet. **Disposition** Return to MoE only with MLX scatter_add/grouped matmul support or a scoped custom Metal sparse-dispatch kernel. runtime perf worked · caveat ### [CPU speedup bundle](https://posttrainllm.com/experiments/cpu-speedup-bundle) **Evidence** all four items ON reached median `6.8 step/s` vs `5.0 step/s` baseline, `+36%`, with final loss `4.188-4.192` unchanged **Why it stopped** The bundle produced a real host-side speedup, but QoS and compiled cosine LR were individually in the noise, prefetch stayed single-digit, and open CPU items remained. **Learning** The big win was fused accumulation inside the compiled trace; individual CPU tweaks need isolation before being credited. **Disposition** Use the bundle as shipped, but keep Rust-FFI BPE, preallocated CPU buffers, and SVD-on-CPU AMX as separate future attempts. runtime perf worked · caveat ### [Cold-start bundle](https://posttrainllm.com/experiments/cold-start-bundle) **Evidence** mmap saved `10-20 ms` and `25 MB` peak RSS on `demo.tinygpt`; lazy embedding deferred `262.1 KB`; package tests passed `43/43` **Why it stopped** The cold-start path improved the measured demo case, but the 250 MB target file was missing, async spinner overhead added about `30 ms`, and persistent Metal pipeline caching was infeasible through public MLX-Swift APIs. **Learning** mmap and direct byte construction are good scaling levers, but large-model cold-start claims need large-model measurements. **Disposition** Re-measure on a real 50-250 MB checkpoint and keep persistent Metal cache work parked unless MLX-Swift exposes pipeline descriptors. 17816 mb` while step/s moved `0.4 -> 0.3`; tiny memory worsened `161 mb -> 189 mb` activation recompute is a scale-dependent lever; it should stay opt-in for models where memory, not step time, is the blocker. update memory estimation for `--grad-checkpoint` and consider selective layer checkpointing before broad default use. " data-astro-cid-jsfgzirp> runtime perf worked · caveat ### [Gradient checkpointing Mac training](https://posttrainllm.com/experiments/gradient-checkpointing-mac-training) **Evidence** behemoth B=4 ctx=1024 peak memory dropped `27679 MB -> 17816 MB` while step/s moved `0.4 -> 0.3`; tiny memory worsened `161 MB -> 189 MB` **Why it stopped** Checkpointing recovered large-model memory but hurt small-model memory and speed, and the OOMGuard estimator still reports uncheckpointed activation projections. **Learning** Activation recompute is a scale-dependent lever; it should stay opt-in for models where memory, not step time, is the blocker. **Disposition** Update memory estimation for `--grad-checkpoint` and consider selective layer checkpointing before broad default use. 0.106 s`; pre-alloc decode stayed `706 -> 711 tok/s` with physical cache flat at `6.3 mb` reuse of prefilled kv is the user-visible win; cache storage policy changes need composition gates before becoming defaults. keep prompt-cache as the main shipped path and revisit pre-alloc defaults only after kivi/streamingllm interactions are designed. " data-astro-cid-jsfgzirp> runtime perf worked · caveat ### [KV cache optimization bundle](https://posttrainllm.com/experiments/kv-cache-optimization-bundle) **Evidence** prompt-cache hot run `1.005 s -> 0.106 s`; pre-alloc decode stayed `706 -> 711 tok/s` with physical cache flat at `6.3 MB` **Why it stopped** Persistent prompt cache delivered the TTFT win, but pre-alloc was kept opt-in, pre-alloc does not compose with KIVI or StreamingLLM, and some cache keying edge cases remain. **Learning** Reuse of prefilled KV is the user-visible win; cache storage policy changes need composition gates before becoming defaults. **Disposition** Keep prompt-cache as the main shipped path and revisit pre-alloc defaults only after KIVI/StreamingLLM interactions are designed. 2.5 mb`, while throughput moved `767 -> 676 tok/s` yoco is a long-context memory lever, not an immediate short-context speedup. treat yoco as a long-context experiment until anchor tuning, hf round-trip, and long-context quality gates exist. " data-astro-cid-jsfgzirp> runtime perf worked · caveat ### [YOCO cache-halving smoke](https://posttrainllm.com/experiments/yoco-cache-halving-smoke) **Evidence** huge cache at 206 tokens dropped `5.1 MB -> 2.5 MB`, while throughput moved `767 -> 676 tok/s` **Why it stopped** YOCO halved decode KV cache memory, but throughput dropped at ctx=256 and HF checkpoint round-trip or production-quality YOCO weights were not proven. **Learning** YOCO is a long-context memory lever, not an immediate short-context speedup. **Disposition** Treat YOCO as a long-context experiment until anchor tuning, HF round-trip, and long-context quality gates exist. 6.4 mb`; int8 kivi matched greedy baseline `100%` and int4 averaged `96%` prefix match kv compression can preserve greedy decode locally, but storage format and quality gates matter before making long-context claims. add packed int4 storage or grouped kivi scales only after a cached-forward perplexity or trajectory-match eval exists. " data-astro-cid-jsfgzirp> runtime perf worked · caveat ### [StreamingLLM + KIVI cache compression](https://posttrainllm.com/experiments/streamingllm-kivi-cache-compression) **Evidence** StreamingLLM reduced 500-token cache `12.6 MB -> 6.4 MB`; int8 KIVI matched greedy baseline `100%` and int4 averaged `96%` prefix match **Why it stopped** The cache compression worked in smoke tests, but int4 precision still used int8 storage bytes, K scales recompute every step, and no held-out perplexity gate measured quality. **Learning** KV compression can preserve greedy decode locally, but storage format and quality gates matter before making long-context claims. **Disposition** Add packed int4 storage or grouped KIVI scales only after a cached-forward perplexity or trajectory-match eval exists. 0.061`; bpe-dropout 100-step smoke stayed stable with final loss `7.382` baseline vs `7.597` dropout data-pipeline regularizers need behavioral evals, not just stable loss curves, before being treated as quality improvements. use sample-packing where length bias matters, and evaluate bpe-dropout on rare-word or morphology-sensitive gates before promotion. " data-astro-cid-jsfgzirp> runtime perf worked · caveat ### [Data-side sample packing and BPE-dropout](https://posttrainllm.com/experiments/data-perf-sample-packing-bpe-dropout) **Evidence** sample packing flattened CoV `0.582 -> 0.061`; BPE-dropout 100-step smoke stayed stable with final loss `7.382` baseline vs `7.597` dropout **Why it stopped** Sample packing's histogram property was proven, but BPE-dropout was only a stability smoke and did not prove downstream generalization on this corpus. **Learning** Data-pipeline regularizers need behavioral evals, not just stable loss curves, before being treated as quality improvements. **Disposition** Use sample-packing where length bias matters, and evaluate BPE-dropout on rare-word or morphology-sensitive gates before promotion. runtime perf worked ### [MLXFast SDPA and tied-embedding audit](https://posttrainllm.com/experiments/mlxfast-sdpa-tied-embedding-audit) **Evidence** audit found SDPA call sites and tied-embedding build/save/load paths clean; build succeeded and sample smoke reached `603 tok/s` on demo and `261 tok/s` on mega5 **Learning** The suspected SDPA mask/contiguity and tied-embedding duplication issues were not present. **Disposition** Keep this as positive verification; only reopen if new SDPA call sites or untied embedding save paths are added. tool calling superseded ### [A1 first tool-calling specialist harness](https://posttrainllm.com/experiments/a1-tool-caller-harness) **Evidence** the recipe and acceptance harness shipped; later Qwen3-4B file-ops distillation and BFCL-family evaluations exercised the specialist contract with measured results **Why it stopped** The original A1-specific adapter run was never executed, but its question and workflow were overtaken by later measured tool-calling specialists and should not be recreated for historical bookkeeping. **Learning** Reusable infrastructure can be superseded by stronger downstream evidence even when its first named model run never happened. **Disposition** Closed as superseded. Use the measured file-ops/ReST lineage for evidence and start any new tool-caller as a fresh experiment. sql worked · caveat ### [B1 SQL eval infrastructure](https://posttrainllm.com/experiments/b1-sql-eval-infra) **Evidence** `posttrainllm eval-sql` shipped and smoke-tested; training/generation originally pending GPU **Why it stopped** The scoring half existed before the live SQL factory run, but the original PRD did not include trained adapter evidence. **Learning** Execution accuracy over SQLite is the right reusable gate; infra alone is not a specialist result. **Disposition** Continue using eval-sql as the frozen SQL gate and attach run folders to model claims. factory docs worked · caveat ### [Factory run schema](https://posttrainllm.com/experiments/factory-run-schema) **Evidence** `docs/factory/run-schema.md` defines the expected run folder **Why it stopped** The schema exists, but real train/eval commands still need to emit every file automatically. **Learning** A schema is necessary but not sufficient; publish checks must enforce complete run folders. **Disposition** Wire live train/eval commands to produce schema-complete run folders automatically. factory docs worked · caveat ### [SQL factory run renderer](https://posttrainllm.com/experiments/sql-factory-run-renderer) **Evidence** `scripts/sql/render_sql_factory_run.py` renders canonical SQL report artifacts **Why it stopped** The renderer is a useful bridge but not yet one universal factory command. **Learning** Rendered reports made SQL publishable, but the factory still needs command-level automation. **Disposition** Use the renderer until the canonical train/eval/report command can emit the same artifacts directly. factory docs worked ### [Public artifact registry](https://posttrainllm.com/experiments/public-artifact-registry) **Evidence** `docs/factory/public-artifacts.md` tracks artifacts, blockers, release state **Learning** Public artifacts and blockers are first-class. **Disposition** Keep every artifact entry updated before a release push. factory docs worked · caveat ### [Technique registry](https://posttrainllm.com/experiments/technique-registry) **Evidence** `docs/techniques/` distinguishes methods from recipes **Why it stopped** The registry exists, but the next SQL recipe still needs validator/release enforcement before training claims. **Learning** Method names are not enough; recipes need failure mode, data, eval, slice gate, and stop rule. **Disposition** Require a recipe card before any new post-training run. factory docs worked · caveat ### [TrainLoop-style tooling](https://posttrainllm.com/experiments/trainloop-style-tooling) **Evidence** candidate choice, slice metrics, trace review, batch plan, LoRA geometry smokes pass **Why it stopped** The tools exist and pass smokes, but most recipes have not yet been trained with them. **Learning** Discipline is valuable only when enforced in every reported run. **Disposition** Do not report the next SQL candidate without `slice-metrics.json` and `trace_review.md`. 0.900` (+0.040, exact `0.840 -> 0.860`) reference-anchored dpo is the correct fix for the simpo collapse ~~higher-pressure reference-anchored dpo...~~ done 2026-07-11 (next entry). output-format dpo lora qwen3-0.6b" data-astro-cid-jsfgzirp> sql failed ### [Hygiene DPO reference-anchored retry (2026-07-11)](https://posttrainllm.com/experiments/sql-hygiene-dpo-refanchored) - dpo - lora **Evidence** composed execution `0.860 -> 0.900` (+0.040, exact `0.840 -> 0.860`) **Why it stopped** DPO pressure too low to change output FORMAT — 41/50 raw **Learning** Reference-anchored DPO is the correct fix for the SimPO collapse **Disposition** ~~Higher-pressure reference-anchored DPO...~~ Done 2026-07-11 (next entry). 0.0073` (strong preference separation). composed composed dpo cannot fix output-format hygiene here fix the format at generation strength, not data content output-format dpo lora qwen3-0.6b" data-astro-cid-jsfgzirp> sql failed ### [Hygiene DPO higher-pressure retry (2026-07-11)](https://posttrainllm.com/experiments/sql-hygiene-dpo-higher-pressure) - dpo - lora **Evidence** DPO loss `0.6931 -> 0.0073` (strong preference separation). Composed **Why it stopped** Output format is SFT/base-controlled and not reachable by a **Learning** Composed DPO cannot fix output-format hygiene here **Disposition** fix the format at generation strength, not data content 0.030` a memorization gate needs more than one distinct target keep as a precondition check only; never quote as a quality result. typo-repair sft lora flan-t5-small" data-astro-cid-jsfgzirp> autocorrect worked · caveat ### [Autocorrect FLAN-T5-small tiny-overfit gate (2026-07-25)](https://posttrainllm.com/experiments/autocorrect-tiny-overfit-gate) - sft - lora **Evidence** exact match `1.0` at step 50 of 200; loss `1.585 -> 0.030` **Why it stopped** The fixture has one unique target, so exact match 1.0 is reachable by memorizing a single sentence **Learning** A memorization gate needs more than one distinct target **Disposition** Keep as a precondition check only; never quote as a quality result. -0.8125` (delta `-0.875`) minimum-edit repair is a narrow band between copying everything and rewriting everything no further training under `adapter-recipe-v1`. typo-repair sft lora flan-t5-small" data-astro-cid-jsfgzirp> autocorrect regressed ### [Autocorrect FLAN-T5-small ordinary-loss pilot (2026-07-25)](https://posttrainllm.com/experiments/autocorrect-ordinary-loss-pilot) - sft - lora **Evidence** error reduction `+0.0625 -> -0.8125` (delta `-0.875`) **Why it stopped** The model became a paraphraser rather than a repairer **Learning** Minimum-edit repair is a narrow band between copying everything and rewriting everything **Disposition** No further training under `adapter-recipe-v1`. 10.54%` (+4.38 points), test `6.23% -> 10.33%` (+4.10); raw and guarded policies each won 0/6 games against random legal play legal constrained decoding and lower move-prediction loss are not game intelligence; require full-game transfer before scaling a tiny specialist. do not run the 100k, 1m, or 2m stages under this recipe. preserve the failed artifact and select a different bounded specialist target. chess-move-selection sft byte-character-chess-44m" data-astro-cid-jsfgzirp> chess failed ### [Character Chess 44.53M masked 10k pilot (2026-08-05)](https://posttrainllm.com/experiments/character-chess-44m-masked-10k-pilot) - sft **Evidence** validation exact `6.16% random -> 10.54%` (+4.38 points), test `6.23% -> 10.33%` (+4.10); raw and guarded policies each won 0/6 games against random legal play **Why it stopped** Completion-only SFT learned a small held-out move preference but missed the frozen +10-point promotion gate and produced no full-game win advantage over random legal play; eight guard interventions converted no wins. **Learning** Legal constrained decoding and lower move-prediction loss are not game intelligence; require full-game transfer before scaling a tiny specialist. **Disposition** Do not run the 100k, 1M, or 2M stages under this recipe. Preserve the failed artifact and select a different bounded specialist target. tool calling failed ### [Needle 2 base public tool-selection gate (2026-08-31)](https://posttrainllm.com/experiments/needle2-base-public-gate) - evaluation **Evidence** 32/94 exact tool selections (34.0%), with 8 out-of-scope false calls and 2 destructive-action calls **Why it stopped** The base artifact was inaccurate on Pace and file operations, produced unsafe false actions, and exposed no useful confidence threshold. **Learning** Tiny packaging and fast decode do not make a tool selector safe enough for action routing. **Disposition** Reject the base artifact for Pace; preserve it only as a tiny-runtime baseline. 36/94, while pace regressed 10/28 -> 6/28, file operations stayed 0/6, and 3 out-of-scope prompts still produced actions catalog restriction is a latency lever, not a capability or safety rescue for this base model. reject this integration path; require a materially different training hypothesis and a new frozen gate for any future needle-derived work. tool-selection routing evaluation needle2" data-astro-cid-jsfgzirp> tool calling regressed ### [Needle 2 task-specific catalog ablation (2026-09-01)](https://posttrainllm.com/experiments/needle2-task-catalog-ablation) - routing - evaluation **Evidence** overall exactness moved 32/94 -> 36/94, while Pace regressed 10/28 -> 6/28, file operations stayed 0/6, and 3 out-of-scope prompts still produced actions **Why it stopped** Oracle task-family routing reduced schema size and latency but did not repair capability or safety; the target Pace slice became worse. **Learning** Catalog restriction is a latency lever, not a capability or safety rescue for this base model. **Disposition** Reject this integration path; require a materially different training hypothesis and a new frozen gate for any future Needle-derived work. tool calling failed ### [Needle 45M distractor-and-safety successor factorial (2026-09-03)](https://posttrainllm.com/experiments/needle2-successor-factorial) - lora - evaluation **Evidence** all four arms passed 100% tiny overfit, but public-dev exactness was 25.5%, 25.5%, 26.6%, and 27.7% versus the 34.0% incumbent; every arm produced 7-9 out-of-scope false actions and all 10 destructive bypasses **Why it stopped** The 45M model memorized the tiny gate but neither distractor-aware data nor explicit safety examples transferred to safe 94-case tool selection. **Learning** Tiny overfit proves wiring, not capacity; the interaction treatment recovered only two cases over plain training and could not learn destructive confirmation at this scale. **Disposition** Close 45M recipe search and advance the same frozen selection-and-safety task to the 1.7B class as a fresh experiment. browser product worked · caveat ### [Parakeet WGSL browser-ASR paired validation (2026-09-02)](https://posttrainllm.com/experiments/parakeet-wgsl-browser-asr-smoke) - evaluation **Evidence** On the same eight LibriSpeech clips, browser Parakeet v3 scored 0/82 word errors versus native WhisperKit's 7/82, matched all four proper nouns, had zero repetition errors, and decoded 3.51x faster; its 33.84x median short-clip real-time factor missed the frozen 50x bar **Why it stopped** Quality, paired native latency, repetition, and offline-warm gates passed, but fixed dispatch overhead on 2-7 second clips held the preregistered median throughput below 50x; the optional first download is 652.7 MiB. **Learning** Browser WebGPU ASR can beat a native large-model incumbent on both bounded WER and latency while still failing an absolute throughput target; short-clip fixed cost and long-form throughput are different measurements. **Disposition** Close this frozen gate without relabeling it; batching, long-form amortization, Safari, or a larger WER set are fresh experiments. offhours inconclusive ### [OffHours Devin stress validation (2026-08-21)](https://posttrainllm.com/experiments/offhours-stress-validation) - evaluation **Evidence** 30/30 workday-conditions and 1,200/1,200 claims completed, but the clean condition scored 95.0% and missed the frozen 98% calibration gate **Why it stopped** The benchmark ruler failed calibration before the stress comparison could support a confirmatory semantic-effect claim. **Learning** A complete run is still inconclusive when the clean frontier calibration misses its preregistered gate. **Disposition** Closed as calibration evidence; do not use it as a causal stress result. offhours worked · caveat ### [OffHours persistent-tension pilot (2026-08-21)](https://posttrainllm.com/experiments/offhours-persistent-tension) - evaluation **Evidence** unresolved tension scored 198/200 versus 197/200 resolved, a -0.5 percentage-point paired error difference with 95% interval -3.0 to +1.5 **Why it stopped** The frozen experiment found no work-quality penalty, while provider prompt-token, quantization, and model-file provenance remained unavailable. **Learning** Unresolved narrative tension did not degrade measured work quality under the exercised conditions; a null result is a valid result. **Disposition** Closed as report-only evidence; do not tune scenarios after the null. offhours worked · caveat ### [OffHours context-saturation boundary (2026-08-21)](https://posttrainllm.com/experiments/offhours-context-saturation) - evaluation **Evidence** the ordered volume ladder found its first reproducible boundary at 2,000 words/event or 8,000 submitted words/day **Why it stopped** The boundary occurred in the neutral arm and therefore measures raw context volume or context management, not personal-obligation semantics; full provider/model provenance is incomplete. **Learning** The supported result is a neutral context-volume boundary, not a psychological or family-tension effect. **Disposition** Closed as report-only boundary evidence with the interpretation constrained to raw volume. game benchmarks rejected ### [Character Chess benchmark candidate audit (2026-08-22)](https://posttrainllm.com/experiments/character-chess-benchmark-candidate) - evaluation **Evidence** 100 stable candidates produced 86 admitted positions, but the best recorded frontier screen reached only 75% exact agreement **Why it stopped** The frontier calibration did not approach the required ceiling, external-model coverage was incomplete, and the model alias was mutable, so the suite could not be frozen as a fair ruler. **Learning** A benchmark candidate set is not a benchmark until the frontier ceiling validates the ruler. **Disposition** Closed as candidate-only evidence; design any future chess ruler independently after the learning phase. game benchmarks rejected ### [Character 2048 frontier benchmark screen (2026-08-04)](https://posttrainllm.com/experiments/character-2048-frontier-screen) - evaluation **Evidence** pinned Sonnet matched random legal play at 0.995x mean score and pinned Opus reached only 1.058x over three complete pairs before a provider disconnect **Why it stopped** The proposed character interface did not expose a reliable frontier intelligence advantage, so training a 30-50M student would measure ruler noise. **Learning** Validate the teacher and interface before spending compute on a specialist; a failed ruler should stop training. **Disposition** Closed in the current character form; keep the negative benchmark replay public. game benchmarks rejected ### [Character game arena candidate report (2026-08-05)](https://posttrainllm.com/experiments/character-game-arena-candidate) **Evidence** the arena retained four chess matches and seven complete 2048 pairs, while every rating remained unqualified **Why it stopped** Chess missed minimum-match and forfeit gates, while 2048 missed minimum-pair or complete-source gates; cross-game ranking would have overstated sparse development evidence. **Learning** A rating calculation is not a public rating until each underlying competition family passes its qualification contract. **Disposition** Closed as a transparent candidate report with no Elo, human-equivalence, or specialist-win claim. pace planner rejected ### [Pace intent router v8 sealed rejection](https://posttrainllm.com/experiments/pace-intent-router-v8-sealed) - sft - routing **Evidence** Synthetic holdout 95.5%; official sealed V1 57.1% (unknown 55.6%), router p50 3.8 ms. **Why it stopped** Synthetic phrasing generalization and confidence failed on independently sealed states. **Learning** A fast classifier can learn the generator rather than the action policy. **Disposition** Retain as a rejected baseline; require policy-adjudicated fresh cases before a successor. pace planner worked · caveat ### [Pace next-step Jev teacher development baseline](https://posttrainllm.com/experiments/pace-next-step-jev-20261004) - evaluation - routing **Evidence** 123 development fixtures: Jev 85.4% full pass (direct and keyless); direct routing p50 385 ms. **Why it stopped** Cloud inference violates Pace runtime privacy, and development fixtures are synthetic and already inspected. **Learning** Use the teacher as offline comparison evidence; runtime integration requires an on-device candidate. **Disposition** Keep PR #201 experiment-only; qualify a separate acceptance set before shipping. pace planner worked · caveat ### [Pace next-step Qwen development baseline](https://posttrainllm.com/experiments/pace-next-step-qwen-20261004) - evaluation - routing **Evidence** 123 development fixtures: Qwen3.5-4B 71.5% full pass, 40% ASK_USER; routing p50 687 ms. **Why it stopped** Clarification and stopping errors remain; a synthetic development score is not hardware acceptance. **Learning** Compare identical simulated loops and argument extractors rather than model-card scores. **Disposition** Pin this development baseline and score the sealed acceptance cases separately. pace planner rejected ### [Pace next-step GLiNER zero-shot rejection](https://posttrainllm.com/experiments/pace-next-step-gliner-20261004) - evaluation - routing **Evidence** 123 development fixtures: GLiNER headed input 0% full pass; prose 17.9%; ASK_USER 0% in both. **Why it stopped** Premature DONE and repeated actions prevented useful task completion. **Learning** Zero-shot label matching does not establish stateful next-step control. **Disposition** Reject this zero-shot path; preserve its recorded negative result. pace planner inconclusive ### [Pace next-step ModernBERT student v1](https://posttrainllm.com/experiments/pace-next-step-student-v1-20261004) - sft - distill - routing **Evidence** 123 development fixtures: student v1 58.5% full pass, reported MPS routing p50 16 ms. **Why it stopped** Teacher labels and limited coverage left single-target click routing weak. Synthetic sealed data and inspected development misses do not qualify a shipping gate. **Learning** Student v1 left single-target clicks weak; label policy and independent evaluation must precede further teacher distillation. **Disposition** Continue issue #200 with a label policy, fresh adjudicated acceptance cases, and per-class regression gates. pace planner inconclusive ### [Pace next-step ModernBERT student v2](https://posttrainllm.com/experiments/pace-next-step-student-v2-20261004) - sft - distill - routing **Evidence** 123 development fixtures: student v2 53.7% full pass, reported MPS routing p50 18 ms. **Why it stopped** Agreement filtering reduced development full pass rather than repairing label policy. Synthetic sealed data and inspected development misses do not qualify a shipping gate. **Learning** Agreement filtering lowered student v2 full pass to 53.7%; agreement alone does not repair a disputed action policy. **Disposition** Continue issue #200 with a label policy, fresh adjudicated acceptance cases, and per-class regression gates. pace planner inconclusive ### [Pace next-step ModernBERT student v3](https://posttrainllm.com/experiments/pace-next-step-student-v3-20261004) - sft - distill - routing **Evidence** 123 development fixtures: student v3 65.0% full pass, reported MPS routing p50 14 ms. **Why it stopped** Goal-phrased data improved clicks but full pass remained below the local 71.5% baseline and RESPOND regressed. Synthetic sealed data and inspected development misses do not qualify a shipping gate. **Learning** Goal-phrased training improved student v3 clicks but regressed RESPOND; report per-class gates alongside aggregate full pass. **Disposition** Continue issue #200 with a label policy, fresh adjudicated acceptance cases, and per-class regression gates. pace planner worked · caveat ### [Pace next-step Qwen reproduced development baseline](https://posttrainllm.com/experiments/pace-next-step-qwen-20261005) - evaluation - routing **Evidence** Fresh 123-case run: 71.5% full pass, 40.0% ASK_USER, routing p50/p95 613/687 ms. **Why it stopped** The prior result reproduced, but synthetic development golds remain disputed. **Learning** Separate tool choice, stateful stopping, input-format compatibility, and acceptance evidence. **Disposition** Keep as the frozen local development comparator; qualify human-adjudicated acceptance separately. pace planner rejected ### [Pace next-step Jeff v1.2 base rejection](https://posttrainllm.com/experiments/pace-next-step-jeff-base-20261005) - evaluation - routing **Evidence** Jeff base: 123 development cases, 33.3% full pass, 6.7% ASK_USER, p50 121 ms; 50 repeat guards. **Why it stopped** Fast routing did not preserve clarification, action selection, or stopping. **Learning** Jeff base latency did not preserve clarification, action choice or stopping; qualify each behavior before choosing a fast router. **Disposition** Reject this zero-shot configuration for Pace; no app wiring. pace planner rejected ### [Pace next-step Jeff v1.2 tools rejection](https://posttrainllm.com/experiments/pace-next-step-jeff-tools-20261005) - evaluation - routing **Evidence** Jeff tools with 32 Pace options: 38.2% full pass, 10.0% ASK_USER, p50 143 ms; 36 repeat guards. **Why it stopped** The unchanged label contract remained weak on step control and clarification. **Learning** Jeff tools with the existing label contract remained weak on clarification and step control; a tool-oriented model is not a qualified policy replacement. **Disposition** Reject the 32-option substitution; preserve a separate upstream-format trial. pace planner rejected ### [Pace next-step Jeff documented tools-format rejection](https://posttrainllm.com/experiments/pace-next-step-jeff-tools-native-20261005) - evaluation - routing **Evidence** Jeff tools documented 31-option format: 34.1% full pass, 33.3% ASK_USER, p50 120 ms; 58 repeat guards. **Why it stopped** Following the published option wording and state format did not repair progress awareness; explicit DONE was unmeasured. **Learning** Jeff documented wording did not repair progress awareness; native format compatibility must be tested separately from DONE and task completion. **Disposition** Reject this format adaptation for the current next-step router. pace planner rejected ### [Pace next-step Jeff frozen synthetic smoke rejection](https://posttrainllm.com/experiments/pace-next-step-jeff-tools-smoke-20261005) - evaluation - routing **Evidence** 416 frozen synthetic states: 36.5% generator agreement, 34.4% teacher agreement, ECE 0.332/0.353, p50/p95 146/169 ms. **Why it stopped** Weakness persisted on ambiguous and progress states; generator and teacher labels are disputed, so this is smoke evidence only. **Learning** The frozen Jeff synthetic smoke retained ambiguous and progress failures; disputed generator and teacher labels cannot establish acceptance. **Disposition** Do not fit fallback thresholds or claim acceptance from this sample. pace planner blocked ### [Pace next-step Apple FM availability blocker](https://posttrainllm.com/experiments/pace-next-step-apple-fm-20261005) - evaluation - routing **Evidence** Standalone FoundationModels bridge Release build passed; runtime modelNotReady, zero evaluated cases and no accuracy score. **Why it stopped** The system model was unavailable at runtime despite a successful isolated build. **Learning** Separate tool choice, stateful stopping, input-format compatibility, and acceptance evidence. **Disposition** Re-evaluate after the system model becomes ready; preserve unknown accuracy. pace planner rejected ### [Pace frozen generated policy Qwen comparison](https://posttrainllm.com/experiments/pace-next-step-qwen-policy-20261008) - evaluation - routing **Evidence** 96 generated policy cases: rules 96/96, Qwen3.5-4B 4-bit 48/96 with zero errors; Qwen route p50/p95 394.7507/953.5587 ms, model bytes 3061316115. **Why it stopped** Qwen failed all ASK_USER, RESPOND and named-tool cases. The rules-perfect template set establishes regression behavior only, not real-task generalization. **Learning** Preserve raw failures, generated provenance and unknown calibration/hardware measures; a fast or rules-perfect result does not qualify a planner replacement. **Disposition** KEEP the current native planner. The owner withdrew the 400-500 handwritten-case requirement; any future learned replacement needs independent task evidence, not relabeled generated gold. No experiments match those filters. Archive closed; lab open. These records are not a hidden backlog. New work starts only when you open a fresh question after learning, freeze its ruler, and create a new recipe and experiment ID.