# Every experiment — posttrainllm > The complete PostTrainLLM experiment archive: every worked, failed, regressed, inconclusive, superseded, and rejected attempt with evidence and a reusable… Canonical: https://posttrainllm.com/experiments This is the complete evidence-backed history of posttrainllm. Nothing is queued or quietly “not tried.” Each historical question ends in a measured result or a final disposition, with the lesson preserved for your next fresh experiment. 128 accounted attempts 0 unresolved attempts 117 exact-evidence records 14 experiment families [Read the narrative ledger →](https://posttrainllm.com/docs/attempt-ledger) [Open reproducible recipes →](https://posttrainllm.com/recipes) [Start the learning lab →](https://posttrainllm.com/learn) ## The shape of the work Negative results remain first-class evidence. - worked: 5 - worked · caveat: 55 - failed: 14 - regressed: 23 - inconclusive: 14 - superseded: 2 - rejected: 14 - blocked: 1 ## All recorded attempts ### [Toy SQL SFT](https://posttrainllm.com/experiments/sql-toy-sft-r4) sql worked · caveat - sft - lora - dora **Evidence:** execution `0.167 -> 0.833` on 6-row fixture **Why it stopped:** Train/eval overlap made the apparent improvement non-shippable. **Learning:** The SFT loop mechanics work, but eval splits must be frozen before claiming quality. **Disposition:** Use only as a loop sanity check; rely on heldout SQL evals for real claims. direct evidence [Proof](https://posttrainllm.com/docs/specialists/b1-sql-poc) [Recipes](https://posttrainllm.com/recipes/sql-lineage) [Learn](https://posttrainllm.com/learn/paths/post-training) ### [Expanded synthetic SQL SFT](https://posttrainllm.com/experiments/sql-expanded-synthetic-sft-r4) sql worked - sft - lora - dora **Evidence:** execution `0.160 -> 0.860`, exact `0.140 -> 0.840` **Learning:** The synthetic SQL adapter is the current synthetic incumbent, but joins remain weaker than single-table rows. **Disposition:** Keep as the synthetic incumbent until a candidate beats it on the same heldout gate. no failure claim [Proof](https://posttrainllm.com/docs/factory/public-artifacts) [Recipes](https://posttrainllm.com/recipes/sql-lineage) [Learn](https://posttrainllm.com/learn/paths/post-training) ### [Public b-mc2 v1](https://posttrainllm.com/experiments/sql-public-bmc2-v1) sql failed - sft - lora **Evidence:** exact `0.344` vs T5-small `0.484` **Why it stopped:** The public-style dataset was too small to beat the T5-small baseline. **Learning:** Public benchmark adaptation needs more coverage before tuning knobs matter. **Disposition:** Scale public-style rows and re-evaluate against the same exact gate. inferred cause [Proof](https://posttrainllm.com/docs/specialists/b1-sql-poc) [Recipes](https://posttrainllm.com/recipes/sql-lineage) [Learn](https://posttrainllm.com/learn/paths/post-training) ### [Public b-mc2 v2](https://posttrainllm.com/experiments/sql-public-bmc2-v2) sql failed - sft - lora **Evidence:** exact `0.031` **Why it stopped:** Increasing rows, rank, and steps together made the recipe worse and confounded the cause. **Learning:** Do not change data volume, rank, and training length in one uncontrolled jump. **Disposition:** Use smaller controlled deltas with fixed eval and comparable training settings. inferred cause [Proof](https://posttrainllm.com/docs/specialists/b1-sql-poc) [Recipes](https://posttrainllm.com/recipes/sql-lineage) [Learn](https://posttrainllm.com/learn/paths/post-training) ### [Public b-mc2 v3](https://posttrainllm.com/experiments/sql-public-bmc2-v3) sql failed - sft - lora **Evidence:** exact `0.422` **Why it stopped:** Lower LR and fewer steps recovered quality but still missed the public baseline. **Learning:** The public recipe was directionally better, but not enough for a win. **Disposition:** Try targeted data weighting instead of only global hyperparameter changes. inferred cause [Proof](https://posttrainllm.com/docs/specialists/b1-sql-poc) [Recipes](https://posttrainllm.com/recipes/sql-lineage) [Learn](https://posttrainllm.com/learn/paths/post-training) ### [Public b-mc2 v4](https://posttrainllm.com/experiments/sql-public-bmc2-v4) sql worked · caveat - sft - lora **Evidence:** exact `0.531` vs T5-small `0.484` **Why it stopped:** The public win did not preserve synthetic SQL execution behavior. **Learning:** A narrow public exact-match win can still damage the incumbent specialist behavior. **Disposition:** Route or compose specialists instead of forcing one adapter to satisfy both gates. direct evidence [Proof](https://posttrainllm.com/docs/specialists/b1-sql-poc) [Recipes](https://posttrainllm.com/recipes/sql-lineage) [Learn](https://posttrainllm.com/learn/paths/post-training) ### [Public v4 synthetic regression check](https://posttrainllm.com/experiments/sql-public-v4-synthetic-regression) sql regressed - evaluation **Evidence:** synthetic execution `0.240` vs incumbent `0.860` **Why it stopped:** Public-style training overfit schema-linking patterns that did not transfer to synthetic SQLite domains. **Learning:** Every public benchmark gain needs a breadth/regression gate before packaging. **Disposition:** Keep public and synthetic adapters separate unless a candidate passes both gates. direct evidence [Proof](https://posttrainllm.com/docs/specialists/b1-sql-poc) [Recipes](https://posttrainllm.com/recipes/sql-lineage) [Learn](https://posttrainllm.com/learn/paths/post-training) ### [Blended public+synthetic SFT](https://posttrainllm.com/experiments/sql-blended-public-synthetic-v1) sql failed - sft - lora **Evidence:** public `0.297`, synthetic execution `0.560` **Why it stopped:** Naive mixture training caused interference and output-format collapse. **Learning:** Mixing domains is not automatically regularization; it can reduce both capabilities. **Disposition:** Use routing, data-mix search, or explicit slice gates before another blended adapter. direct evidence [Proof](https://posttrainllm.com/docs/specialists/b1-sql-poc) [Recipes](https://posttrainllm.com/recipes/sql-lineage) [Learn](https://posttrainllm.com/learn/paths/post-training) ### [Static multi-LoRA composition](https://posttrainllm.com/experiments/sql-static-multilora-composition) sql failed - multi-lora-composition **Evidence:** best public-pass setting still synthetic-failed **Why it stopped:** Fixed adapter weights produced a smooth tradeoff but no setting passed both gates. **Learning:** Static composition cannot solve domain conflict when the right behavior depends on input type. **Disposition:** Prefer input-aware routing over global composition weights. direct evidence [Proof](https://posttrainllm.com/docs/specialists/b1-sql-poc) [Recipes](https://posttrainllm.com/recipes/sql-lineage) [Learn](https://posttrainllm.com/learn/paths/post-training) ### [Routed public + synthetic adapters](https://posttrainllm.com/experiments/sql-routed-v1) sql worked · caveat - routing **Evidence:** public exact `0.531`, synthetic execution `0.860` **Why it stopped:** The routed artifact lacks public execution benchmarking and perf/package gates. **Learning:** Routing preserves both measured slices, but artifact status still depends on public execution and performance evidence. **Disposition:** Add public execution gate, latency/RAM/tok-s measurement, and a package decision. direct evidence [Proof](https://posttrainllm.com/artifacts/qwen06-sql-routed-v1) [Recipes](https://posttrainllm.com/recipes/sql-lineage) [Learn](https://posttrainllm.com/learn/paths/post-training) ### [Hygiene SimPO/DPO](https://posttrainllm.com/experiments/sql-hygiene-simpo-dpo) sql failed - simpo - lora **Evidence:** execution `0.860 -> 0.080`, clean-SQL `0.000 -> 0.000` **Why it stopped:** Ref-free SimPO update collapsed the policy and produced degenerate fenced/prose outputs. **Learning:** Preference tuning needs reference anchoring, smaller updates, and composed-eval checks. **Disposition:** Retry with reference-anchored DPO or much lower LR/step count on the same frozen pairs. direct evidence [Proof](https://posttrainllm.com/docs/attempt-ledger) [Recipes](https://posttrainllm.com/recipes/sql-lineage) [Learn](https://posttrainllm.com/learn/paths/post-training) ### [SQL candidate selection](https://posttrainllm.com/experiments/sql-candidate-selection) sql rejected - candidate-selection **Evidence:** tooling and smoke exist, but no frozen candidate-selection dataset, baseline, or model run was approved before project closure **Why it stopped:** The proposed run never acquired a frozen target/eval contract, and running it retrospectively would start a new experiment rather than complete an existing measured claim. **Learning:** Candidate selection remains a plausible future method, but an unfrozen recipe is not unfinished evidence for this project. **Disposition:** Closed for this project. Reopen only as a fresh owner-led experiment after the learning phase with new data and a frozen gate. direct evidence [Proof](https://posttrainllm.com/docs/techniques/sql-technique-backlog) [Recipes](https://posttrainllm.com/recipes/sql-lineage) [Learn](https://posttrainllm.com/learn/paths/post-training) ### [Offline rollout / OAPL-style SQL update](https://posttrainllm.com/experiments/sql-offline-rollout-oapl) sql rejected - rlvr **Evidence:** plan renderer exists, but the required candidate-selection reward surface was never validated and no model run was authorized **Why it stopped:** The experiment's explicit prerequisite did not exist, so an OAPL-style update would not have had a trustworthy reward and could not answer its intended question. **Learning:** Offline policy updates are downstream of reward validation; skipping the prerequisite is not a useful experiment. **Disposition:** Closed for this project. A future fresh experiment must validate its reward surface first. direct evidence [Proof](https://posttrainllm.com/docs/techniques/sql-technique-backlog) [Recipes](https://posttrainllm.com/recipes/sql-lineage) [Learn](https://posttrainllm.com/learn/paths/post-training) ### [Controlled SQL LoRA rank sweep](https://posttrainllm.com/experiments/sql-controlled-lora-rank-sweep) sql rejected - lora **Evidence:** no controlled sweep was run because no next target/eval was frozen after the confounded historical comparisons **Why it stopped:** Without one active frozen recipe, a rank sweep would measure a new arbitrary setup rather than resolve the historical confound. **Learning:** Hyperparameter sweeps are valid only inside one frozen experiment contract; they are not project-completion chores. **Disposition:** Closed for this project. Any future rank study begins as a new experiment with fixed data, seed, steps, learning rate, and eval. direct evidence [Proof](https://posttrainllm.com/docs/techniques/sql-technique-backlog) [Recipes](https://posttrainllm.com/recipes/sql-lineage) [Learn](https://posttrainllm.com/learn/paths/post-training) ### [Qwen3-4B file-ops distilled specialist](https://posttrainllm.com/experiments/fileops-qwen3-4b-distilled) file ops worked · caveat - distill **Evidence:** file-ops hard gate `0.58 -> 1.00`; breadth `0.596 -> 0.423` **Why it stopped:** The specialist improved the narrow file-ops gate but regressed out-of-domain breadth. **Learning:** The artifact is useful only with routed-only positioning; it should not be sold as a general planner. **Disposition:** Keep routed-only warnings prominent and add loader/pull smoke before wiring consumers. direct evidence [Proof](https://posttrainllm.com/artifacts/qwen3-4b-file-ops-distilled) [Recipes](https://posttrainllm.com/recipes/distillation) [Learn](https://posttrainllm.com/learn/paths/post-training) ### [Qwen3-4B ReST fused breadth recovery variant](https://posttrainllm.com/experiments/fileops-qwen3-4b-rest-fused) file ops worked · caveat - rest - distill **Evidence:** fresh frozen run: depth `9/12 stock -> 12/12 ReST` (+25 points), breadth `30/45 stock -> 25/45 ReST` (-11.1 points), 10 fewer schema/side-effect events, and 2.42x faster depth wall time **Why it stopped:** The unchanged public weights reproduce a strong routed file-operations win but regress held-out breadth by five net paired cases, so they fail the general-successor gate. **Learning:** ReST produced a real depth-and-safety specialist, not breadth recovery; routing is the win and general replacement is the failed hypothesis. **Disposition:** Retain the package only behind the file-operations route; any broader successor needs breadth-preserving data or optimization on the same frozen ruler. direct evidence [Proof](https://posttrainllm.com/artifacts/qwen3-4b-rest-fused) [Recipes](https://posttrainllm.com/recipes/distillation) [Learn](https://posttrainllm.com/learn/paths/post-training) ### [Qwen3-4B multibackend distilled variant](https://posttrainllm.com/experiments/fileops-qwen3-4b-multibackend-distilled) file ops regressed - distill **Evidence:** depth `100%`, breadth `31%` **Why it stopped:** Negative transfer damaged breadth even though depth remained high. **Learning:** Multi-backend distillation can over-specialize and needs breadth gates by default. **Disposition:** Keep as failed comparison evidence; do not promote without a new breadth-preserving recipe. direct evidence [Proof](https://posttrainllm.com/artifacts/qwen3-4b-multibackend-distilled) [Recipes](https://posttrainllm.com/recipes/distillation) [Learn](https://posttrainllm.com/learn/paths/post-training) ### [Frozen gate held-out readback](https://posttrainllm.com/experiments/measurement-gate-audit-20261025) file ops inconclusive - evaluation - routing **Evidence:** ReST `4/4 fileops, 4/4 vehicle`; multibackend `4/4 fileops, 4/4 vehicle`; frontier `8/8`; both emit the same 19 requested calls. Both runner counters report zero schema failures and zero listed unexpected side effects; actuator coverage is incomplete. **Why it stopped:** The eight-task slice does not distinguish the historically retained and rejected candidates, so it cannot validate the comparative selection boundary or overturn the broader rejection. **Learning:** A frontier ceiling is necessary but does not make a simple smoke discriminating; a zero fixed-list side-effect counter also does not prove complete safety. **Disposition:** Preserve the ReST file-operations route and multibackend general-use rejection. Before future acceptance, freeze primary/regression slices representing historical failure classes and review every decoded mutator arguments, multiplicity and order at the execution boundary. direct evidence [Proof](https://posttrainllm.com/docs/attempt-ledger) [Recipes](https://posttrainllm.com/recipes/distillation) [Learn](https://posttrainllm.com/learn/paths/post-training) ### [VibeThinker 3B agentic distilled variant](https://posttrainllm.com/experiments/vibethinker-3b-agentic-distilled) archive model inconclusive - distill **Evidence:** weights preserved; current eval promotion missing **Why it stopped:** The weights exist, but the docs do not preserve a current factory eval proving a win. **Learning:** Preserved weights are not a ship decision. **Disposition:** Run the factory eval gate and publish a before/after report before using it as a specialist package. direct evidence [Proof](https://posttrainllm.com/artifacts/vibethinker-3b-agentic-distilled) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/quantization-and-packaging) ### [Apple on-device Foundation Models action-grounding probe](https://posttrainllm.com/experiments/apple-foundation-models-action-grounding) apple fm failed - evaluation **Evidence:** BFCL agentic `25%` full catalog / approximately `0%` compact; Pace planner action-grounding `13%` **Why it stopped:** Apple's on-device model could not ground actions well enough and its 4096-token context could not hold a real tool catalog. **Learning:** Apple Foundation Models are useful as a free routing floor and intel source, not as the core capability dependency. **Disposition:** Keep Core ML for our own weights as an optional future battery optimization, not a model-dependency bet. direct evidence [Proof](https://posttrainllm.com/docs/learn/apple-on-device-foundation-models) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/browser-and-mac-runtime) ### [Pace planner v1-v4 early versions](https://posttrainllm.com/experiments/pace-planner-v1-v4-early) pace planner inconclusive - sft - lora **Evidence:** datasets, builder scripts, and planned recipes survive; no per-version score was recorded **Why it stopped:** No per-version result was kept. The v11 ship gate names this directly: v1 through v10 cycled because there was no fixed bar, no fixed eval, and no immutable record of pass criteria. **Learning:** An unmeasured version is indistinguishable from a version that never ran. Four consecutive attempts left no score behind. **Disposition:** Do not attempt to reconstruct scores for v1-v4; treat the series as starting at v5, the first version with a recorded number. evidence gap [Proof](https://posttrainllm.com/docs/prds/pace-planner-v11-ship-gate) [Recipes](https://posttrainllm.com/recipes/constrained-generation) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Pace planner v5 hand-crafted gold + teacher paraphrase](https://posttrainllm.com/experiments/pace-planner-v5) pace planner worked · caveat - sft - lora **Evidence:** 17/19 (89%) on fm-fixtures v1 and 6/15 (40.0%) on fm-fixtures-v2 **Why it stopped:** The 17/19 was measuring format compliance, not capability: a rule-based FakePace endpoint scored 19/19 on the same fixture set. **Learning:** A high score on a fixture a deterministic endpoint can also ace is a measurement of the harness, not the model. **Disposition:** Treat fm-fixtures v1 as retired; only fm-fixtures-v2 numbers are comparable. direct evidence [Proof](https://posttrainllm.com/docs/learn/eval-matrix-2026-06-08) [Recipes](https://posttrainllm.com/recipes/constrained-generation) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Pace planner v6 label-based output](https://posttrainllm.com/experiments/pace-planner-v6) pace planner regressed - sft - lora **Evidence:** 14/19 (74%) on fm-fixtures v1 and 3/15 (20.0%) on fm-fixtures-v2, 20pp below v5 **Why it stopped:** The label-based SFT destroyed capability, not just output format. **Learning:** Changing the output representation is a capability change, not a formatting change. **Disposition:** Revert to the v5 corpus for further augmentation work. direct evidence [Proof](https://posttrainllm.com/docs/learn/eval-matrix-2026-06-08) [Recipes](https://posttrainllm.com/recipes/constrained-generation) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Pace planner v6.1 four-adapter repair sweep](https://posttrainllm.com/experiments/pace-planner-v6-1) pace planner worked · caveat - sft - lora **Evidence:** 10/19, 8/19, 2/19, 0/19, then 14/19 at 100 steps and 17/19 at 300 steps, reaching 19/19 after a serving-side minLength fix; 4/15 (26.7%) on fm-fixtures-v2 **Why it stopped:** The apparent collapse was two harness bugs, not training: SFT rows omitted the Pace system prompt that serve and eval both included, and a brace-counting JSON extractor was faulty. **Learning:** A four-way score collapse traced to prompt mismatch between training and serving. Pin the prompt, schema, and grammar triple across train, serve, and eval before believing any comparison. **Disposition:** Recover the full sweep with `git show d1efcb8^:docs/prds/factory-pace-planner-v6_1.md` before repeating any step-count ablation; it is the only one in the series. direct evidence [Proof](https://posttrainllm.com/docs/prds/specialist-pace-planner) [Recipes](https://posttrainllm.com/recipes/constrained-generation) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Pace planner v7 tools-in-prompt](https://posttrainllm.com/experiments/pace-planner-v7) pace planner superseded - sft - lora **Evidence:** tools-in-prompt serve, verb-enum grammar, and data/eval harnesses shipped; the SFT was never run **Why it stopped:** The version was never trained, and later v8-v11 planner work replaced its proposed prompt/schema direction and supplied the relevant capability evidence. **Learning:** v7 is a numbering gap, not a training version. The eleven-version count includes a version that was never trained. **Disposition:** Closed as a numbering gap. Do not cite v7 as evidence or revive it outside a new owner-led experiment. direct evidence [Proof](https://posttrainllm.com/docs/prds/factory-planner-v7-tools-in-prompt) [Recipes](https://posttrainllm.com/recipes/constrained-generation) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Pace planner v8 augmented corpus](https://posttrainllm.com/experiments/pace-planner-v8) pace planner inconclusive - sft - lora **Evidence:** reported 11/15 (73.3%) on fm-fixtures-v2, later re-measured at 5/15 (33.3%) from the same weights and same fixtures **Why it stopped:** The headline score did not reproduce. Re-running the same eval against the same baked weights yielded 33.3%, and the gap was root-caused as prompt and schema config drift rather than any model change. **Learning:** The 73% phantom. A score is only a score if the prompt, schema, and grammar it was measured under are recorded with it. **Disposition:** Treat 33.3% as the working v8 baseline and never compare across versions without pinning the harness config. direct evidence [Proof](https://posttrainllm.com/artifacts/pace-intent-router-v8) [Recipes](https://posttrainllm.com/recipes/constrained-generation) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Pace planner v9 compose](https://posttrainllm.com/experiments/pace-planner-v9) pace planner worked · caveat - sft - lora **Evidence:** 70% on new compose fixtures with non-compose held at the v8 level; as-shipped re-measurement 9/15 (60.0%); unhappy-path n=130 gave 0/40 ambiguous, 13/60 out-of-scope, 1/30 destructive **Why it stopped:** It shipped as the specialist but never became Pace's runtime planner, and the unhappy-path drill put it at a capacity wall: 0% ambiguous and 3% destructive. **Learning:** Winning the happy-path fixture says nothing about ambiguous, out-of-scope, or destructive inputs, which is where the product actually needed the planner. **Disposition:** Keep as the last shipped specialist; Pace runs qwen3-30b-a3b instead. direct evidence [Proof](https://posttrainllm.com/docs/sessions/v11-baselines-2026-06-09) [Recipes](https://posttrainllm.com/recipes/constrained-generation) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Pace planner v10 parameterized actions](https://posttrainllm.com/experiments/pace-planner-v10) pace planner inconclusive - sft - lora **Evidence:** full recipe, action registry, schema, and eval plan are committed; no score was ever recorded and it is not established that training completed **Why it stopped:** No v10 score exists anywhere in the repository. The last status recorded is that training was pending, while the v10 corpus went on to become the base of v11. **Learning:** v10 repeated v6's move of changing the output representation, and because it was never measured the v6 regression could not be checked for recurrence. Its unmeasured corpus was then inherited by v11. **Disposition:** Do not reuse a v10-derived corpus without first measuring the representation change in isolation. evidence gap [Proof](https://posttrainllm.com/docs/attempt-ledger) [Recipes](https://posttrainllm.com/recipes/constrained-generation) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Pace planner v11 ship-gate run](https://posttrainllm.com/experiments/pace-planner-v11) pace planner failed - sft - lora **Evidence:** failed all six gate dimensions: BFCL pace-12 27.1% against a 40% bar, out-of-scope 0/30 against 80%, ambiguous 0/20 against 50%, destructive 5/10 against 90%, with train loss 0.001 **Why it stopped:** A 0.6B planner did not reach the fixed bar on any dimension despite the lowest training loss in the series, and the first v11 attempt was voided outright by the DoRA infrastructure failure. **Learning:** Train loss 0.001 alongside 0% on out-of-scope and ambiguous inputs is the capacity ceiling stated plainly. The one-shot fixed gate was what finally ended the cycle. **Disposition:** Do not train v12. The specialist planner track is closed and Pace ships qwen3-30b-a3b. direct evidence [Proof](https://posttrainllm.com/docs/sessions/v11-baselines-2026-06-09) [Recipes](https://posttrainllm.com/recipes/constrained-generation) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Pace clarify-v1 Qwen3-4B LoRA](https://posttrainllm.com/experiments/pace-clarify-v1-qwen4b-lora) pace planner regressed - lora **Evidence:** 38 contrastive rows moved ambig only `0% -> 5%` while OOS regressed `80% -> 33%` in the retrospective; drilldown records the same attempt as `2%/40%/50%` on ambig/oos/destructive **Why it stopped:** Small-corpus LoRA on a competent base caused catastrophic interference across planner dimensions. **Learning:** Fine-tuning a strong base on a thin behavior slice requires coverage for every dimension that must not regress. **Disposition:** Do not fine-tune the planner on narrow clarify rows unless the corpus also covers OOS, destructive, happy path, and breadth gates. inferred cause [Proof](https://posttrainllm.com/docs/sessions/RETROSPECTIVE) [Recipes](https://posttrainllm.com/recipes/constrained-generation) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Pace two-stage ambiguity detector shim v2](https://posttrainllm.com/experiments/pace-two-stage-ambiguity-shim-v2) pace planner worked · caveat - two-stage-shim **Evidence:** ambig moved `0% -> 20%`, while OOS dropped `78% -> 55%` and destructive stayed `67%`; not shipped on top of Gemma **Why it stopped:** The shim improved ambiguity detection for the 4B floor but over-triggered and became unnecessary once Gemma reached the same ambig score with better OOS/destructive behavior. **Learning:** Rule wrappers can expose the missing behavior, but they still need the same regression gates as model updates. **Disposition:** Keep as an option only if shipping the 4B footprint floor; do not stack it on the stronger Gemma path. direct evidence [Proof](https://posttrainllm.com/docs/sessions/DRILLDOWN) [Recipes](https://posttrainllm.com/recipes/constrained-generation) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Pace reasoning-model unhappy-path drill](https://posttrainllm.com/experiments/pace-thinking-models-unhappy-path) pace planner inconclusive - evaluation **Evidence:** Qwen3-4B-Thinking scored `0%/10%/0%` and DeepSeek-R1-Distill-Qwen-7B scored `0%/3%/0%` on ambig/oos/destructive under a 300-token cap **Why it stopped:** The evaluation budget was consumed by think traces, so the apparent zero scores were not a fair model-quality verdict. **Learning:** Reasoning models need adequate max tokens and a stop-on-think strategy before interpreting planner scores. **Disposition:** Re-run with max_tokens >= 1024 and stop on `` only if Gemma or the locked planner fails in production. direct evidence [Proof](https://posttrainllm.com/docs/sessions/DRILLDOWN) [Recipes](https://posttrainllm.com/recipes/constrained-generation) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Pace Gemma-3-12B unhappy-path drill winner](https://posttrainllm.com/experiments/pace-gemma-3-12b-drill-winner) pace planner worked · caveat - evaluation **Evidence:** Gemma-3-12B scored `22%` ambig, `82%` OOS, `77%` destructive and won all three unhappy-path dimensions in the n=130 drill **Why it stopped:** Gemma won the drill but ambiguity remained poor in absolute terms and the decision was later superseded by the general planner lock. **Learning:** A stronger zero-shot base beat the local fine-tunes, but model-selection evidence can be superseded by later broader gates. **Disposition:** Keep as historical unhappy-path evidence; prefer current `PROJECT_STATUS.md`/`docs/NEXT.md` for active planner decisions. direct evidence [Proof](https://posttrainllm.com/docs/sessions/DRILLDOWN) [Recipes](https://posttrainllm.com/recipes/constrained-generation) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Pace Qwen3.5 challenger round](https://posttrainllm.com/experiments/pace-qwen35-challenger-round) pace planner worked · caveat - evaluation **Evidence:** Qwen3.5-9B scored `18%/97%/63%`; Qwen3.5-4B scored `15%/55%/70%` on ambig/oos/destructive; champion stood **Why it stopped:** Qwen3.5 improved some dimensions, especially OOS for 9B, but did not beat the stored champion across the full swap decision. **Learning:** Single-dimension wins do not justify a planner swap when another required dimension regresses. **Disposition:** Use `scripts/pipelines/eval_planner.sh` for future challengers and keep swap/no-swap verdicts tied to all required dimensions. direct evidence [Proof](https://posttrainllm.com/docs/sessions/DRILLDOWN) [Recipes](https://posttrainllm.com/recipes/constrained-generation) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Browser Memory64 large-preset runs](https://posttrainllm.com/experiments/browser-memory64-large-presets) browser product worked · caveat **Evidence:** larger browser presets hit Memory64 OOB; matching Node direct-export reproducer passed XL; `INITIAL_MEMORY=268435456` fixed up to XL while Behemoth still exercises growth **Why it stopped:** The original retries treated the OOB as a model-size problem before the matching reproducer showed a browser pthread plus SharedArrayBuffer memory-growth race. **Learning:** A reproducer must match the production host and call path before it can explain a browser-only WASM failure. **Disposition:** Keep the higher initial memory for supported presets and only claim Behemoth after a SAB-view-aware growth path lands. direct evidence [Proof](https://posttrainllm.com/artifacts/memory64-browser-behemoth) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/browser-and-mac-runtime) ### [Browser speedup headline curve](https://posttrainllm.com/experiments/browser-speedup-headline-curve) browser product worked · caveat **Evidence:** the historical curve remains unqualified except at Large, where a 2026-09-02 Apple M5 Pro ABBA receipt measured `10.67x` median WebGPU-over-WASM speedup with `4.72%` maximum paired final-loss drift and zero runtime errors **Why it stopped:** The original headline made a shape-dependent result look universal, and the retained tree cannot establish that the historical curve used a real hardware adapter rather than a software fallback. **Learning:** Qualify performance one frozen shape and machine at a time; a retained adapter identity plus alternated paired timings can convert one point without laundering the surrounding curve. **Disposition:** Keep Large as the measured M5 Pro point; independently reproduce Small, Medium, and XL before promoting the historical curve. direct evidence [Proof](https://posttrainllm.com/artifacts/browser-webgpu-speedup) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/browser-and-mac-runtime) ### [Browser default corpus fix](https://posttrainllm.com/experiments/browser-default-corpus-fix) browser product worked · caveat **Evidence:** the old default corpus was `863 bytes`; a Huge model memorized it with train loss `0.14`; default changed to TinyShakespeare at about `1.1 MB` **Why it stopped:** The demo was trying to prove learning with a corpus so small that memorization looked like training success. **Learning:** Default data is part of the product claim; a training demo's default corpus must produce non-memorized samples. **Disposition:** Keep corpus defaults in the same parity review as model and optimizer defaults. direct evidence [Proof](https://posttrainllm.com/docs/archive/lessons) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/browser-and-mac-runtime) ### [Browser learning-rate default drift](https://posttrainllm.com/experiments/browser-lr-default-drift) browser product worked · caveat **Evidence:** browser default LR was `3e-3` while Python reference used `3e-4`; training plateaued around loss `2.45` until the browser default was changed **Why it stopped:** Reference/kernel parity tests covered math drift but not hyperparameter default drift. **Learning:** Reference defaults are part of correctness, not just documentation. **Disposition:** Add or keep a config-default parity check whenever browser and reference paths share a training story. direct evidence [Proof](https://posttrainllm.com/docs/archive/lessons) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/browser-and-mac-runtime) ### [Browser WebGPU export serialization](https://posttrainllm.com/experiments/browser-webgpu-export-serialization) browser product worked · caveat **Evidence:** a trained WebGPU run was lost because `webgpu/gpu_model.ts` had no checkpoint serialization; `exportState` was then implemented and worker plumbing updated **Why it stopped:** The training path could produce useful weights that the WebGPU backend could not export, and auto-cleanup destroyed a real trained artifact. **Learning:** Long-running browser training is not product-complete until the smallest export path is smoke-tested before the long run. **Disposition:** Smoke export on a short WebGPU run before any long browser training claim. direct evidence [Proof](https://posttrainllm.com/docs/sessions/qa_log) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/browser-and-mac-runtime) ### [Browser generation streaming and KV cache](https://posttrainllm.com/experiments/browser-generation-streaming-kv-cache) browser product rejected **Evidence:** tokens/sec counter was added, but browser generation still lacked KV cache and true token streaming; task `#72` was logged **Why it stopped:** The browser track completed without implementing this feature, and no measured claim depends on pretending the typewriter animation is true streaming. **Learning:** A typewriter animation over an already-complete response is not true streaming. **Disposition:** Closed for this project with the limitation explicit. A future browser product may reopen true streaming as a new feature. direct evidence [Proof](https://posttrainllm.com/docs/sessions/qa_log) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/browser-and-mac-runtime) ### [WASM backward-scratch reuse](https://posttrainllm.com/experiments/wasm-backward-scratch-reuse) runtime perf worked · caveat **Evidence:** standard `304 -> 305 ms/step`, capable `632 -> ~640 ms/step` **Why it stopped:** Caching backward scratch buffers removed hot-path allocations but produced no measurable speedup because compute, not malloc, was the bottleneck. **Learning:** Hot-path hygiene is not the same as a performance win; measure the real step time before claiming speedup. **Disposition:** Keep the allocation cleanup, but do not prioritize allocator work until profiling shows allocation cost again. direct evidence [Proof](https://posttrainllm.com/docs/performance/performance) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [WASM SIMD training speedup](https://posttrainllm.com/experiments/wasm-simd-training-speedup) runtime perf worked **Evidence:** standard `304 -> 191 ms/step`, capable `632 -> 391 ms/step`, net `1.6x` speedup **Learning:** Emscripten/LLVM autovectorization was the right browser-training speed lever before hand-written kernels. **Disposition:** Keep SIMD enabled by default and use the WASM smoke gate to protect training correctness. no failure claim [Proof](https://posttrainllm.com/docs/performance/performance) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Browser WebGPU training loop](https://posttrainllm.com/experiments/browser-webgpu-training-loop) runtime perf worked **Evidence:** Apple M5 Pro / Metal 3 ABBA run measured `10.67x` median Large-preset speedup over WASM with `4.72%` maximum paired final-loss drift and zero runtime errors **Learning:** Hardware-qualified browser WebGPU can deliver a double-digit training-step win while preserving the frozen numerical-loss gate. **Disposition:** Retain Large as the verified Apple M5 Pro result; Small, Medium, XL, other browsers, and other hardware require independent receipts. no failure claim [Proof](https://posttrainllm.com/docs/attempt-ledger) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [WASM register/cache-blocked matmul](https://posttrainllm.com/experiments/wasm-register-cache-blocked-matmul) runtime perf worked · caveat **Evidence:** microbench `3-4x`, but end-to-end WASM training only `1.03-1.06x` after forward blocking **Why it stopped:** The isolated microbench overstated impact because the real WASM training workload was threaded, already SIMD-vectorized, and dominated by backward matmuls. **Learning:** Optimize and report on the real workload, not only on isolated single-kernel microbenches. **Disposition:** Keep the bit-exact forward kernel for inference, but judge future perf work with end-to-end training and generation benches. direct evidence [Proof](https://posttrainllm.com/docs/performance/performance) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [WASM backward_dA blocking](https://posttrainllm.com/experiments/wasm-backward-da-blocking) runtime perf worked · caveat **Evidence:** fwd+bwd-blocked reached small `90.3`, medium `317.6`, large `1057.4`, xl `1643.6`, or about `1.10-1.13x` over naive **Why it stopped:** The backward_dA optimization improved end-to-end training, but the remaining backward_dB path still limits the total speedup. **Learning:** Training speed follows the backward-pass bottleneck; forward-only wins are capped until dA and dB are addressed. **Disposition:** Only continue matmul blocking if backward_dB or generation benches show enough remaining headroom. direct evidence [Proof](https://posttrainllm.com/docs/performance/performance) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Speculative-decoding heads Medusa/EAGLE smoke](https://posttrainllm.com/experiments/speculative-heads-medusa-eagle-smoke) runtime perf worked · caveat **Evidence:** baseline `232 tok/s`; Medusa 50-step `202 tok/s` at `21.7%`; Medusa 300-step `375 tok/s` at `20.8%`; EAGLE-2 50-step `200 tok/s` at `26.5%`, all bit-identical where verified **Why it stopped:** The heads trained and verified losslessly, but acceptance stayed far below the paper recipes because this run lacked long training, logit distillation, tree attention, and a full EAGLE transformer draft block. **Learning:** Speculative heads are a valid local speed path, but the production recipe is data/training heavy rather than a quick sidecar smoke. **Disposition:** Revive only with self-generated data, logit-KL distillation, tree verification, and a real acceptance-rate gate. direct evidence [Proof](https://posttrainllm.com/docs/techniques/speculative_heads) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [MoE dense-compute smoke](https://posttrainllm.com/experiments/moe-dense-compute-smoke) architecture worked · caveat **Evidence:** Dense MLP `842 K` params loss `6.09 -> 1.76` at `55.6 step/s`; MoE 4 experts top-2 `2.42 M` params loss `5.95 -> 1.68` at `29.3 step/s` **Why it stopped:** The MoE architecture, save/load, sample, eval, and inspect paths work, but the current compute path runs every expert and therefore does not deliver sparse-dispatch savings. **Learning:** MoE is a capacity feature until sparse gather/scatter or grouped matmul lands; it should not be marketed as a compute-saving win yet. **Disposition:** Return to MoE only with MLX scatter_add/grouped matmul support or a scoped custom Metal sparse-dispatch kernel. direct evidence [Proof](https://posttrainllm.com/docs/techniques/moe) [Recipes](https://posttrainllm.com/recipes/moe) [Learn](https://posttrainllm.com/learn/paths/architecture-and-kernels) ### [CPU speedup bundle](https://posttrainllm.com/experiments/cpu-speedup-bundle) runtime perf worked · caveat **Evidence:** all four items ON reached median `6.8 step/s` vs `5.0 step/s` baseline, `+36%`, with final loss `4.188-4.192` unchanged **Why it stopped:** The bundle produced a real host-side speedup, but QoS and compiled cosine LR were individually in the noise, prefetch stayed single-digit, and open CPU items remained. **Learning:** The big win was fused accumulation inside the compiled trace; individual CPU tweaks need isolation before being credited. **Disposition:** Use the bundle as shipped, but keep Rust-FFI BPE, preallocated CPU buffers, and SVD-on-CPU AMX as separate future attempts. direct evidence [Proof](https://posttrainllm.com/docs/performance/cpu_speedup_results) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Cold-start bundle](https://posttrainllm.com/experiments/cold-start-bundle) runtime perf worked · caveat **Evidence:** mmap saved `10-20 ms` and `25 MB` peak RSS on `demo.tinygpt`; lazy embedding deferred `262.1 KB`; package tests passed `43/43` **Why it stopped:** The cold-start path improved the measured demo case, but the 250 MB target file was missing, async spinner overhead added about `30 ms`, and persistent Metal pipeline caching was infeasible through public MLX-Swift APIs. **Learning:** mmap and direct byte construction are good scaling levers, but large-model cold-start claims need large-model measurements. **Disposition:** Re-measure on a real 50-250 MB checkpoint and keep persistent Metal cache work parked unless MLX-Swift exposes pipeline descriptors. direct evidence [Proof](https://posttrainllm.com/docs/performance/cold_start_results) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Gradient checkpointing Mac training](https://posttrainllm.com/experiments/gradient-checkpointing-mac-training) runtime perf worked · caveat **Evidence:** behemoth B=4 ctx=1024 peak memory dropped `27679 MB -> 17816 MB` while step/s moved `0.4 -> 0.3`; tiny memory worsened `161 MB -> 189 MB` **Why it stopped:** Checkpointing recovered large-model memory but hurt small-model memory and speed, and the OOMGuard estimator still reports uncheckpointed activation projections. **Learning:** Activation recompute is a scale-dependent lever; it should stay opt-in for models where memory, not step time, is the blocker. **Disposition:** Update memory estimation for `--grad-checkpoint` and consider selective layer checkpointing before broad default use. direct evidence [Proof](https://posttrainllm.com/docs/performance/gradient_checkpointing_results) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [KV cache optimization bundle](https://posttrainllm.com/experiments/kv-cache-optimization-bundle) runtime perf worked · caveat **Evidence:** prompt-cache hot run `1.005 s -> 0.106 s`; pre-alloc decode stayed `706 -> 711 tok/s` with physical cache flat at `6.3 MB` **Why it stopped:** Persistent prompt cache delivered the TTFT win, but pre-alloc was kept opt-in, pre-alloc does not compose with KIVI or StreamingLLM, and some cache keying edge cases remain. **Learning:** Reuse of prefilled KV is the user-visible win; cache storage policy changes need composition gates before becoming defaults. **Disposition:** Keep prompt-cache as the main shipped path and revisit pre-alloc defaults only after KIVI/StreamingLLM interactions are designed. direct evidence [Proof](https://posttrainllm.com/docs/performance/kv_cache_optimization) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [YOCO cache-halving smoke](https://posttrainllm.com/experiments/yoco-cache-halving-smoke) runtime perf worked · caveat **Evidence:** huge cache at 206 tokens dropped `5.1 MB -> 2.5 MB`, while throughput moved `767 -> 676 tok/s` **Why it stopped:** YOCO halved decode KV cache memory, but throughput dropped at ctx=256 and HF checkpoint round-trip or production-quality YOCO weights were not proven. **Learning:** YOCO is a long-context memory lever, not an immediate short-context speedup. **Disposition:** Treat YOCO as a long-context experiment until anchor tuning, HF round-trip, and long-context quality gates exist. direct evidence [Proof](https://posttrainllm.com/docs/performance/yoco_results) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [StreamingLLM + KIVI cache compression](https://posttrainllm.com/experiments/streamingllm-kivi-cache-compression) runtime perf worked · caveat **Evidence:** StreamingLLM reduced 500-token cache `12.6 MB -> 6.4 MB`; int8 KIVI matched greedy baseline `100%` and int4 averaged `96%` prefix match **Why it stopped:** The cache compression worked in smoke tests, but int4 precision still used int8 storage bytes, K scales recompute every step, and no held-out perplexity gate measured quality. **Learning:** KV compression can preserve greedy decode locally, but storage format and quality gates matter before making long-context claims. **Disposition:** Add packed int4 storage or grouped KIVI scales only after a cached-forward perplexity or trajectory-match eval exists. direct evidence [Proof](https://posttrainllm.com/docs/techniques/streaming_llm_kivi) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Data-side sample packing and BPE-dropout](https://posttrainllm.com/experiments/data-perf-sample-packing-bpe-dropout) runtime perf worked · caveat **Evidence:** sample packing flattened CoV `0.582 -> 0.061`; BPE-dropout 100-step smoke stayed stable with final loss `7.382` baseline vs `7.597` dropout **Why it stopped:** Sample packing's histogram property was proven, but BPE-dropout was only a stability smoke and did not prove downstream generalization on this corpus. **Learning:** Data-pipeline regularizers need behavioral evals, not just stable loss curves, before being treated as quality improvements. **Disposition:** Use sample-packing where length bias matters, and evaluate BPE-dropout on rare-word or morphology-sensitive gates before promotion. direct evidence [Proof](https://posttrainllm.com/docs/performance/data_perf) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [MLXFast SDPA and tied-embedding audit](https://posttrainllm.com/experiments/mlxfast-sdpa-tied-embedding-audit) runtime perf worked **Evidence:** audit found SDPA call sites and tied-embedding build/save/load paths clean; build succeeded and sample smoke reached `603 tok/s` on demo and `261 tok/s` on mega5 **Learning:** The suspected SDPA mask/contiguity and tied-embedding duplication issues were not present. **Disposition:** Keep this as positive verification; only reopen if new SDPA call sites or untied embedding save paths are added. no failure claim [Proof](https://posttrainllm.com/docs/performance/perf_audit_mlxfast_tied) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [A1 first tool-calling specialist harness](https://posttrainllm.com/experiments/a1-tool-caller-harness) tool calling superseded **Evidence:** the recipe and acceptance harness shipped; later Qwen3-4B file-ops distillation and BFCL-family evaluations exercised the specialist contract with measured results **Why it stopped:** The original A1-specific adapter run was never executed, but its question and workflow were overtaken by later measured tool-calling specialists and should not be recreated for historical bookkeeping. **Learning:** Reusable infrastructure can be superseded by stronger downstream evidence even when its first named model run never happened. **Disposition:** Closed as superseded. Use the measured file-ops/ReST lineage for evidence and start any new tool-caller as a fresh experiment. direct evidence [Proof](https://posttrainllm.com/docs/specialists/a1-tool-caller) [Recipes](https://posttrainllm.com/recipes/distillation) [Learn](https://posttrainllm.com/learn/paths/post-training) ### [B1 SQL eval infrastructure](https://posttrainllm.com/experiments/b1-sql-eval-infra) sql worked · caveat **Evidence:** `posttrainllm eval-sql` shipped and smoke-tested; training/generation originally pending GPU **Why it stopped:** The scoring half existed before the live SQL factory run, but the original PRD did not include trained adapter evidence. **Learning:** Execution accuracy over SQLite is the right reusable gate; infra alone is not a specialist result. **Disposition:** Continue using eval-sql as the frozen SQL gate and attach run folders to model claims. direct evidence [Proof](https://posttrainllm.com/docs/specialists/b1-sql) [Recipes](https://posttrainllm.com/recipes/sql-lineage) [Learn](https://posttrainllm.com/learn/paths/post-training) ### [Factory run schema](https://posttrainllm.com/experiments/factory-run-schema) factory docs worked · caveat **Evidence:** `docs/factory/run-schema.md` defines the expected run folder **Why it stopped:** The schema exists, but real train/eval commands still need to emit every file automatically. **Learning:** A schema is necessary but not sufficient; publish checks must enforce complete run folders. **Disposition:** Wire live train/eval commands to produce schema-complete run folders automatically. direct evidence [Proof](https://posttrainllm.com/artifacts/factory-run-schema-v1) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [SQL factory run renderer](https://posttrainllm.com/experiments/sql-factory-run-renderer) factory docs worked · caveat **Evidence:** `scripts/sql/render_sql_factory_run.py` renders canonical SQL report artifacts **Why it stopped:** The renderer is a useful bridge but not yet one universal factory command. **Learning:** Rendered reports made SQL publishable, but the factory still needs command-level automation. **Disposition:** Use the renderer until the canonical train/eval/report command can emit the same artifacts directly. direct evidence [Proof](https://posttrainllm.com/docs/attempt-ledger) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Public artifact registry](https://posttrainllm.com/experiments/public-artifact-registry) factory docs worked **Evidence:** `docs/factory/public-artifacts.md` tracks artifacts, blockers, release state **Learning:** Public artifacts and blockers are first-class. **Disposition:** Keep every artifact entry updated before a release push. no failure claim [Proof](https://posttrainllm.com/docs/factory/public-artifacts) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Technique registry](https://posttrainllm.com/experiments/technique-registry) factory docs worked · caveat **Evidence:** `docs/techniques/` distinguishes methods from recipes **Why it stopped:** The registry exists, but the next SQL recipe still needs validator/release enforcement before training claims. **Learning:** Method names are not enough; recipes need failure mode, data, eval, slice gate, and stop rule. **Disposition:** Require a recipe card before any new post-training run. direct evidence [Proof](https://posttrainllm.com/docs/techniques/README) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [TrainLoop-style tooling](https://posttrainllm.com/experiments/trainloop-style-tooling) factory docs worked · caveat **Evidence:** candidate choice, slice metrics, trace review, batch plan, LoRA geometry smokes pass **Why it stopped:** The tools exist and pass smokes, but most recipes have not yet been trained with them. **Learning:** Discipline is valuable only when enforced in every reported run. **Disposition:** Do not report the next SQL candidate without `slice-metrics.json` and `trace_review.md`. direct evidence [Proof](https://posttrainllm.com/docs/NEXT) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Hygiene DPO reference-anchored retry (2026-07-11)](https://posttrainllm.com/experiments/sql-hygiene-dpo-refanchored) sql failed - dpo - lora **Evidence:** composed execution `0.860 -> 0.900` (+0.040, exact `0.840 -> 0.860`) **Why it stopped:** DPO pressure too low to change output FORMAT — 41/50 raw **Learning:** Reference-anchored DPO is the correct fix for the SimPO collapse **Disposition:** ~~Higher-pressure reference-anchored DPO...~~ Done 2026-07-11 (next entry). direct evidence [Proof](https://posttrainllm.com/docs/NEXT) [Recipes](https://posttrainllm.com/recipes/sql-lineage) [Learn](https://posttrainllm.com/learn/paths/post-training) ### [Hygiene DPO higher-pressure retry (2026-07-11)](https://posttrainllm.com/experiments/sql-hygiene-dpo-higher-pressure) sql failed - dpo - lora **Evidence:** DPO loss `0.6931 -> 0.0073` (strong preference separation). Composed **Why it stopped:** Output format is SFT/base-controlled and not reachable by a **Learning:** Composed DPO cannot fix output-format hygiene here **Disposition:** fix the format at generation strength, not data content direct evidence [Proof](https://posttrainllm.com/docs/NEXT) [Recipes](https://posttrainllm.com/recipes/sql-lineage) [Learn](https://posttrainllm.com/learn/paths/post-training) ### [Autocorrect FLAN-T5-small tiny-overfit gate (2026-07-25)](https://posttrainllm.com/experiments/autocorrect-tiny-overfit-gate) autocorrect worked · caveat - sft - lora **Evidence:** exact match `1.0` at step 50 of 200; loss `1.585 -> 0.030` **Why it stopped:** The fixture has one unique target, so exact match 1.0 is reachable by memorizing a single sentence **Learning:** A memorization gate needs more than one distinct target **Disposition:** Keep as a precondition check only; never quote as a quality result. direct evidence [Proof](https://posttrainllm.com/docs/attempt-ledger) [Recipes](https://posttrainllm.com/recipes/lora) [Learn](https://posttrainllm.com/learn/paths/post-training) ### [Autocorrect FLAN-T5-small ordinary-loss pilot (2026-07-25)](https://posttrainllm.com/experiments/autocorrect-ordinary-loss-pilot) autocorrect regressed - sft - lora **Evidence:** error reduction `+0.0625 -> -0.8125` (delta `-0.875`) **Why it stopped:** The model became a paraphraser rather than a repairer **Learning:** Minimum-edit repair is a narrow band between copying everything and rewriting everything **Disposition:** No further training under `adapter-recipe-v1`. direct evidence [Proof](https://posttrainllm.com/docs/attempt-ledger) [Recipes](https://posttrainllm.com/recipes/lora) [Learn](https://posttrainllm.com/learn/paths/post-training) ### [Character Chess 44.53M masked 10k pilot (2026-08-05)](https://posttrainllm.com/experiments/character-chess-44m-masked-10k-pilot) chess failed - sft **Evidence:** validation exact `6.16% random -> 10.54%` (+4.38 points), test `6.23% -> 10.33%` (+4.10); raw and guarded policies each won 0/6 games against random legal play **Why it stopped:** Completion-only SFT learned a small held-out move preference but missed the frozen +10-point promotion gate and produced no full-game win advantage over random legal play; eight guard interventions converted no wins. **Learning:** Legal constrained decoding and lower move-prediction loss are not game intelligence; require full-game transfer before scaling a tiny specialist. **Disposition:** Do not run the 100k, 1M, or 2M stages under this recipe. Preserve the failed artifact and select a different bounded specialist target. direct evidence [Proof](https://posttrainllm.com/docs/attempt-ledger) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Needle 2 base public tool-selection gate (2026-08-31)](https://posttrainllm.com/experiments/needle2-base-public-gate) tool calling failed - evaluation **Evidence:** 32/94 exact tool selections (34.0%), with 8 out-of-scope false calls and 2 destructive-action calls **Why it stopped:** The base artifact was inaccurate on Pace and file operations, produced unsafe false actions, and exposed no useful confidence threshold. **Learning:** Tiny packaging and fast decode do not make a tool selector safe enough for action routing. **Disposition:** Reject the base artifact for Pace; preserve it only as a tiny-runtime baseline. direct evidence [Proof](https://posttrainllm.com/artifacts/needle2-tool-selection) [Recipes](https://posttrainllm.com/recipes/needle2-evaluation) [Learn](https://posttrainllm.com/learn/paths/post-training) ### [Needle 2 task-specific catalog ablation (2026-09-01)](https://posttrainllm.com/experiments/needle2-task-catalog-ablation) tool calling regressed - routing - evaluation **Evidence:** overall exactness moved 32/94 -> 36/94, while Pace regressed 10/28 -> 6/28, file operations stayed 0/6, and 3 out-of-scope prompts still produced actions **Why it stopped:** Oracle task-family routing reduced schema size and latency but did not repair capability or safety; the target Pace slice became worse. **Learning:** Catalog restriction is a latency lever, not a capability or safety rescue for this base model. **Disposition:** Reject this integration path; require a materially different training hypothesis and a new frozen gate for any future Needle-derived work. direct evidence [Proof](https://posttrainllm.com/artifacts/needle2-tool-selection) [Recipes](https://posttrainllm.com/recipes/needle2-evaluation) [Learn](https://posttrainllm.com/learn/paths/post-training) ### [Needle 45M distractor-and-safety successor factorial (2026-09-03)](https://posttrainllm.com/experiments/needle2-successor-factorial) tool calling failed - lora - evaluation **Evidence:** all four arms passed 100% tiny overfit, but public-dev exactness was 25.5%, 25.5%, 26.6%, and 27.7% versus the 34.0% incumbent; every arm produced 7-9 out-of-scope false actions and all 10 destructive bypasses **Why it stopped:** The 45M model memorized the tiny gate but neither distractor-aware data nor explicit safety examples transferred to safe 94-case tool selection. **Learning:** Tiny overfit proves wiring, not capacity; the interaction treatment recovered only two cases over plain training and could not learn destructive confirmation at this scale. **Disposition:** Close 45M recipe search and advance the same frozen selection-and-safety task to the 1.7B class as a fresh experiment. direct evidence [Proof](https://posttrainllm.com/docs/attempt-ledger) [Recipes](https://posttrainllm.com/recipes/distillation) [Learn](https://posttrainllm.com/learn/paths/post-training) ### [Parakeet WGSL browser-ASR paired validation (2026-09-02)](https://posttrainllm.com/experiments/parakeet-wgsl-browser-asr-smoke) browser product worked · caveat - evaluation **Evidence:** On the same eight LibriSpeech clips, browser Parakeet v3 scored 0/82 word errors versus native WhisperKit's 7/82, matched all four proper nouns, had zero repetition errors, and decoded 3.51x faster; its 33.84x median short-clip real-time factor missed the frozen 50x bar **Why it stopped:** Quality, paired native latency, repetition, and offline-warm gates passed, but fixed dispatch overhead on 2-7 second clips held the preregistered median throughput below 50x; the optional first download is 652.7 MiB. **Learning:** Browser WebGPU ASR can beat a native large-model incumbent on both bounded WER and latency while still failing an absolute throughput target; short-clip fixed cost and long-form throughput are different measurements. **Disposition:** Close this frozen gate without relabeling it; batching, long-form amortization, Safari, or a larger WER set are fresh experiments. direct evidence [Proof](https://posttrainllm.com/artifacts/parakeet-wgsl-browser-asr) [Recipes](https://posttrainllm.com/recipes/parakeet-browser-asr) [Learn](https://posttrainllm.com/learn/paths/browser-and-mac-runtime) ### [OffHours Devin stress validation (2026-08-21)](https://posttrainllm.com/experiments/offhours-stress-validation) offhours inconclusive - evaluation **Evidence:** 30/30 workday-conditions and 1,200/1,200 claims completed, but the clean condition scored 95.0% and missed the frozen 98% calibration gate **Why it stopped:** The benchmark ruler failed calibration before the stress comparison could support a confirmatory semantic-effect claim. **Learning:** A complete run is still inconclusive when the clean frontier calibration misses its preregistered gate. **Disposition:** Closed as calibration evidence; do not use it as a causal stress result. direct evidence [Proof](https://posttrainllm.com/artifacts/offhours-context-interference) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [OffHours persistent-tension pilot (2026-08-21)](https://posttrainllm.com/experiments/offhours-persistent-tension) offhours worked · caveat - evaluation **Evidence:** unresolved tension scored 198/200 versus 197/200 resolved, a -0.5 percentage-point paired error difference with 95% interval -3.0 to +1.5 **Why it stopped:** The frozen experiment found no work-quality penalty, while provider prompt-token, quantization, and model-file provenance remained unavailable. **Learning:** Unresolved narrative tension did not degrade measured work quality under the exercised conditions; a null result is a valid result. **Disposition:** Closed as report-only evidence; do not tune scenarios after the null. direct evidence [Proof](https://posttrainllm.com/artifacts/offhours-context-interference) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [OffHours context-saturation boundary (2026-08-21)](https://posttrainllm.com/experiments/offhours-context-saturation) offhours worked · caveat - evaluation **Evidence:** the ordered volume ladder found its first reproducible boundary at 2,000 words/event or 8,000 submitted words/day **Why it stopped:** The boundary occurred in the neutral arm and therefore measures raw context volume or context management, not personal-obligation semantics; full provider/model provenance is incomplete. **Learning:** The supported result is a neutral context-volume boundary, not a psychological or family-tension effect. **Disposition:** Closed as report-only boundary evidence with the interpretation constrained to raw volume. direct evidence [Proof](https://posttrainllm.com/artifacts/offhours-context-interference) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Character Chess benchmark candidate audit (2026-08-22)](https://posttrainllm.com/experiments/character-chess-benchmark-candidate) game benchmarks rejected - evaluation **Evidence:** 100 stable candidates produced 86 admitted positions, but the best recorded frontier screen reached only 75% exact agreement **Why it stopped:** The frontier calibration did not approach the required ceiling, external-model coverage was incomplete, and the model alias was mutable, so the suite could not be frozen as a fair ruler. **Learning:** A benchmark candidate set is not a benchmark until the frontier ceiling validates the ruler. **Disposition:** Closed as candidate-only evidence; design any future chess ruler independently after the learning phase. direct evidence [Proof](https://posttrainllm.com/benchmarks/chess) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Character 2048 frontier benchmark screen (2026-08-04)](https://posttrainllm.com/experiments/character-2048-frontier-screen) game benchmarks rejected - evaluation **Evidence:** pinned Sonnet matched random legal play at 0.995x mean score and pinned Opus reached only 1.058x over three complete pairs before a provider disconnect **Why it stopped:** The proposed character interface did not expose a reliable frontier intelligence advantage, so training a 30-50M student would measure ruler noise. **Learning:** Validate the teacher and interface before spending compute on a specialist; a failed ruler should stop training. **Disposition:** Closed in the current character form; keep the negative benchmark replay public. direct evidence [Proof](https://posttrainllm.com/benchmarks/game-2048) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Character game arena candidate report (2026-08-05)](https://posttrainllm.com/experiments/character-game-arena-candidate) game benchmarks rejected **Evidence:** the arena retained four chess matches and seven complete 2048 pairs, while every rating remained unqualified **Why it stopped:** Chess missed minimum-match and forfeit gates, while 2048 missed minimum-pair or complete-source gates; cross-game ranking would have overstated sparse development evidence. **Learning:** A rating calculation is not a public rating until each underlying competition family passes its qualification contract. **Disposition:** Closed as a transparent candidate report with no Elo, human-equivalence, or specialist-win claim. direct evidence [Proof](https://posttrainllm.com/benchmarks/arena) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Pace intent router v8 sealed rejection](https://posttrainllm.com/experiments/pace-intent-router-v8-sealed) pace planner rejected - sft - routing **Evidence:** Synthetic holdout 95.5%; official sealed V1 57.1% (unknown 55.6%), router p50 3.8 ms. **Why it stopped:** Synthetic phrasing generalization and confidence failed on independently sealed states. **Learning:** A fast classifier can learn the generator rather than the action policy. **Disposition:** Retain as a rejected baseline; require policy-adjudicated fresh cases before a successor. direct evidence [Proof](https://posttrainllm.com/docs/attempt-ledger) [Recipes](https://posttrainllm.com/recipes/constrained-generation) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Pace next-step Jev teacher development baseline](https://posttrainllm.com/experiments/pace-next-step-jev-20261004) pace planner worked · caveat - evaluation - routing **Evidence:** 123 development fixtures: Jev 85.4% full pass (direct and keyless); direct routing p50 385 ms. **Why it stopped:** Cloud inference violates Pace runtime privacy, and development fixtures are synthetic and already inspected. **Learning:** Use the teacher as offline comparison evidence; runtime integration requires an on-device candidate. **Disposition:** Keep PR #201 experiment-only; qualify a separate acceptance set before shipping. direct evidence [Proof](https://posttrainllm.com/docs/attempt-ledger) [Recipes](https://posttrainllm.com/recipes/constrained-generation) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Pace next-step Qwen development baseline](https://posttrainllm.com/experiments/pace-next-step-qwen-20261004) pace planner worked · caveat - evaluation - routing **Evidence:** 123 development fixtures: Qwen3.5-4B 71.5% full pass, 40% ASK_USER; routing p50 687 ms. **Why it stopped:** Clarification and stopping errors remain; a synthetic development score is not hardware acceptance. **Learning:** Compare identical simulated loops and argument extractors rather than model-card scores. **Disposition:** Pin this development baseline and score the sealed acceptance cases separately. direct evidence [Proof](https://posttrainllm.com/docs/attempt-ledger) [Recipes](https://posttrainllm.com/recipes/constrained-generation) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Pace next-step GLiNER zero-shot rejection](https://posttrainllm.com/experiments/pace-next-step-gliner-20261004) pace planner rejected - evaluation - routing **Evidence:** 123 development fixtures: GLiNER headed input 0% full pass; prose 17.9%; ASK_USER 0% in both. **Why it stopped:** Premature DONE and repeated actions prevented useful task completion. **Learning:** Zero-shot label matching does not establish stateful next-step control. **Disposition:** Reject this zero-shot path; preserve its recorded negative result. direct evidence [Proof](https://posttrainllm.com/docs/attempt-ledger) [Recipes](https://posttrainllm.com/recipes/constrained-generation) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Pace next-step ModernBERT student v1](https://posttrainllm.com/experiments/pace-next-step-student-v1-20261004) pace planner inconclusive - sft - distill - routing **Evidence:** 123 development fixtures: student v1 58.5% full pass, reported MPS routing p50 16 ms. **Why it stopped:** Teacher labels and limited coverage left single-target click routing weak. Synthetic sealed data and inspected development misses do not qualify a shipping gate. **Learning:** Student v1 left single-target clicks weak; label policy and independent evaluation must precede further teacher distillation. **Disposition:** Continue issue #200 with a label policy, fresh adjudicated acceptance cases, and per-class regression gates. direct evidence [Proof](https://posttrainllm.com/docs/attempt-ledger) [Recipes](https://posttrainllm.com/recipes/constrained-generation) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Pace next-step ModernBERT student v2](https://posttrainllm.com/experiments/pace-next-step-student-v2-20261004) pace planner inconclusive - sft - distill - routing **Evidence:** 123 development fixtures: student v2 53.7% full pass, reported MPS routing p50 18 ms. **Why it stopped:** Agreement filtering reduced development full pass rather than repairing label policy. Synthetic sealed data and inspected development misses do not qualify a shipping gate. **Learning:** Agreement filtering lowered student v2 full pass to 53.7%; agreement alone does not repair a disputed action policy. **Disposition:** Continue issue #200 with a label policy, fresh adjudicated acceptance cases, and per-class regression gates. direct evidence [Proof](https://posttrainllm.com/docs/attempt-ledger) [Recipes](https://posttrainllm.com/recipes/constrained-generation) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Pace next-step ModernBERT student v3](https://posttrainllm.com/experiments/pace-next-step-student-v3-20261004) pace planner inconclusive - sft - distill - routing **Evidence:** 123 development fixtures: student v3 65.0% full pass, reported MPS routing p50 14 ms. **Why it stopped:** Goal-phrased data improved clicks but full pass remained below the local 71.5% baseline and RESPOND regressed. Synthetic sealed data and inspected development misses do not qualify a shipping gate. **Learning:** Goal-phrased training improved student v3 clicks but regressed RESPOND; report per-class gates alongside aggregate full pass. **Disposition:** Continue issue #200 with a label policy, fresh adjudicated acceptance cases, and per-class regression gates. direct evidence [Proof](https://posttrainllm.com/docs/attempt-ledger) [Recipes](https://posttrainllm.com/recipes/constrained-generation) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Pace next-step Qwen reproduced development baseline](https://posttrainllm.com/experiments/pace-next-step-qwen-20261005) pace planner worked · caveat - evaluation - routing **Evidence:** Fresh 123-case run: 71.5% full pass, 40.0% ASK_USER, routing p50/p95 613/687 ms. **Why it stopped:** The prior result reproduced, but synthetic development golds remain disputed. **Learning:** Separate tool choice, stateful stopping, input-format compatibility, and acceptance evidence. **Disposition:** Keep as the frozen local development comparator; qualify human-adjudicated acceptance separately. direct evidence [Proof](https://posttrainllm.com/docs/attempt-ledger) [Recipes](https://posttrainllm.com/recipes/constrained-generation) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Pace next-step Jeff v1.2 base rejection](https://posttrainllm.com/experiments/pace-next-step-jeff-base-20261005) pace planner rejected - evaluation - routing **Evidence:** Jeff base: 123 development cases, 33.3% full pass, 6.7% ASK_USER, p50 121 ms; 50 repeat guards. **Why it stopped:** Fast routing did not preserve clarification, action selection, or stopping. **Learning:** Jeff base latency did not preserve clarification, action choice or stopping; qualify each behavior before choosing a fast router. **Disposition:** Reject this zero-shot configuration for Pace; no app wiring. direct evidence [Proof](https://posttrainllm.com/docs/attempt-ledger) [Recipes](https://posttrainllm.com/recipes/constrained-generation) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Pace next-step Jeff v1.2 tools rejection](https://posttrainllm.com/experiments/pace-next-step-jeff-tools-20261005) pace planner rejected - evaluation - routing **Evidence:** Jeff tools with 32 Pace options: 38.2% full pass, 10.0% ASK_USER, p50 143 ms; 36 repeat guards. **Why it stopped:** The unchanged label contract remained weak on step control and clarification. **Learning:** Jeff tools with the existing label contract remained weak on clarification and step control; a tool-oriented model is not a qualified policy replacement. **Disposition:** Reject the 32-option substitution; preserve a separate upstream-format trial. direct evidence [Proof](https://posttrainllm.com/docs/attempt-ledger) [Recipes](https://posttrainllm.com/recipes/constrained-generation) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Pace next-step Jeff documented tools-format rejection](https://posttrainllm.com/experiments/pace-next-step-jeff-tools-native-20261005) pace planner rejected - evaluation - routing **Evidence:** Jeff tools documented 31-option format: 34.1% full pass, 33.3% ASK_USER, p50 120 ms; 58 repeat guards. **Why it stopped:** Following the published option wording and state format did not repair progress awareness; explicit DONE was unmeasured. **Learning:** Jeff documented wording did not repair progress awareness; native format compatibility must be tested separately from DONE and task completion. **Disposition:** Reject this format adaptation for the current next-step router. direct evidence [Proof](https://posttrainllm.com/docs/attempt-ledger) [Recipes](https://posttrainllm.com/recipes/constrained-generation) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Pace next-step Jeff frozen synthetic smoke rejection](https://posttrainllm.com/experiments/pace-next-step-jeff-tools-smoke-20261005) pace planner rejected - evaluation - routing **Evidence:** 416 frozen synthetic states: 36.5% generator agreement, 34.4% teacher agreement, ECE 0.332/0.353, p50/p95 146/169 ms. **Why it stopped:** Weakness persisted on ambiguous and progress states; generator and teacher labels are disputed, so this is smoke evidence only. **Learning:** The frozen Jeff synthetic smoke retained ambiguous and progress failures; disputed generator and teacher labels cannot establish acceptance. **Disposition:** Do not fit fallback thresholds or claim acceptance from this sample. direct evidence [Proof](https://posttrainllm.com/docs/attempt-ledger) [Recipes](https://posttrainllm.com/recipes/constrained-generation) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Pace next-step Apple FM availability blocker](https://posttrainllm.com/experiments/pace-next-step-apple-fm-20261005) pace planner blocked - evaluation - routing **Evidence:** Standalone FoundationModels bridge Release build passed; runtime modelNotReady, zero evaluated cases and no accuracy score. **Why it stopped:** The system model was unavailable at runtime despite a successful isolated build. **Learning:** Separate tool choice, stateful stopping, input-format compatibility, and acceptance evidence. **Disposition:** Re-evaluate after the system model becomes ready; preserve unknown accuracy. direct evidence [Proof](https://posttrainllm.com/docs/attempt-ledger) [Recipes](https://posttrainllm.com/recipes/constrained-generation) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Pace frozen generated policy Qwen comparison](https://posttrainllm.com/experiments/pace-next-step-qwen-policy-20261008) pace planner rejected - evaluation - routing **Evidence:** 96 generated policy cases: rules 96/96, Qwen3.5-4B 4-bit 48/96 with zero errors; Qwen route p50/p95 394.7507/953.5587 ms, model bytes 3061316115. **Why it stopped:** Qwen failed all ASK_USER, RESPOND and named-tool cases. The rules-perfect template set establishes regression behavior only, not real-task generalization. **Learning:** Preserve raw failures, generated provenance and unknown calibration/hardware measures; a fast or rules-perfect result does not qualify a planner replacement. **Disposition:** KEEP the current native planner. The owner withdrew the 400-500 handwritten-case requirement; any future learned replacement needs independent task evidence, not relabeled generated gold. direct evidence [Proof](https://posttrainllm.com/docs/attempt-ledger) [Recipes](https://posttrainllm.com/recipes/constrained-generation) [Learn](https://posttrainllm.com/learn/paths/evaluation-and-factory) ### [Open Kernel broad split-K router](https://posttrainllm.com/experiments/openkernel-router-broad-20261023) runtime perf regressed **Evidence:** 2026-10-23 H200 check afe0a26a817f473c: 0.8443x aggregate, reported 95% interval 0.8316-0.8570x; correctness passed with no flags. **Why it stopped:** Applying the small-tile split-K path at larger token counts caused measured per-shape regressions and an aggregate below baseline. **Learning:** A fused kernel needs a measured dispatch range; correctness alone does not qualify a performance change. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel large-token router tile](https://posttrainllm.com/experiments/openkernel-router-large-20261023) runtime perf inconclusive **Evidence:** 2026-10-23 H200 check 9cd67921edfc40b7: 1.0088x aggregate, reported 95% interval 0.9936-1.0239x; correctness passed with no flags. **Why it stopped:** The reported aggregate interval includes 1x despite improvements at individual large-token shapes. **Learning:** Keep per-shape improvements separate from demonstrated aggregate gains. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel narrow split-K router](https://posttrainllm.com/experiments/openkernel-router-narrow-20261023) runtime perf worked · caveat **Evidence:** 2026-10-23 H200 scored run 85ab373d8dcc44f7: 1.0534x aggregate, reported 95% interval 1.0376-1.0692x; correctness passed with no flags. Earlier BK64 scored run 5b1f1c6523524705 measured 1.0309x; tile variants have overlapping aggregate intervals. **Why it stopped:** The demonstrated gain is limited to the organizer H200 workload; a larger tile advantage over the earlier scored variant is not statistically established by the aggregate intervals. **Learning:** Preserve tiny-token and large-token baseline paths, and judge tile choices using both target-shape timing and aggregate uncertainty. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel fused attention preparation](https://posttrainllm.com/experiments/openkernel-attention-20261023) runtime perf worked · caveat **Evidence:** 2026-10-23 H200 scored run f141c8d265234a99: 1.1357x aggregate, reported 95% interval 1.1187-1.1528x; correctness passed with no flags. **Why it stopped:** This measures attention preparation plus the existing attention implementation on one organizer workload, not application throughput or Mac performance. **Learning:** Fusion must preserve BF16 rounding, the exact RoPE table, cache slot writes and non-unit scale fallbacks. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel global-expert MoE block sizing](https://posttrainllm.com/experiments/openkernel-moe-20261023) runtime perf worked · caveat **Evidence:** 2026-10-23 H200 scored run 84fb1b656d3c44e2: 1.0645x aggregate, reported 95% interval 1.0485-1.0804x; correctness passed with no flags. **Why it stopped:** The gain is specific to the supplied expert-parallel routing distribution and organizer H200 workload. **Learning:** Use the global expert count for routing density; do not confuse that heuristic with local expert IDs used by alignment. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel combined decoder layer](https://posttrainllm.com/experiments/openkernel-decoder-layer-20261023) runtime perf worked · caveat **Evidence:** 2026-10-23 H200 scored run 0a072555103b4e2d: 1.0254x aggregate, reported 95% interval 1.0100-1.0408x; correctness passed with no flags. Its preceding check 5e444821e58e404a measured 1.0181x overall but regressed at 32 and 128 tokens. **Why it stopped:** Component gains did not compose uniformly: the full-layer check regressed at 32 and 128 tokens despite a positive aggregate. **Learning:** Measure attention and MoE together while preserving the full layer router precision and stream ordering; component speedups do not multiply into a layer claim. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel decoder layer with conservative MoE dispatch](https://posttrainllm.com/experiments/openkernel-decoder-layer-narrow-20261023) runtime perf worked · caveat **Evidence:** 2026-10-23 H200 scored run bc25711588004579: 1.0351x aggregate, reported 95% interval 1.0196-1.0506x; correctness passed with no flags. **Why it stopped:** The cutoff preserves baseline MoE tiling below 256 tokens based on measured layer regressions; results establish only this organizer H200 workload. **Learning:** Use combined-layer evidence to narrow component optimizations, and rerun correctness and timing after restoring fallbacks. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel split-K attention FP8 load compilation](https://posttrainllm.com/experiments/openkernel-attention-fp8-load-20261023) runtime perf failed **Evidence:** 2026-10-23 H200 check 035753a0aef44f2f failed five correctness cases during Triton compilation: integer zero fallback values could not be cast to FP8 E4M3 loads; no timing score was produced. **Why it stopped:** Masked FP8 loads used an integer other=0 value that the judge Triton compiler could not cast to fp8e4nv. **Learning:** Python syntax and lint checks do not validate Triton backend types; masked FP8 loads need a compatible floating-point fill value. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel single-token dense GEMV](https://posttrainllm.com/experiments/openkernel-dense-gemv-20261023) runtime perf worked · caveat **Evidence:** 2026-10-23 H200 scored run 1095cf9b43bf49a0 passed with no flags: 1.0673x aggregate, reported 95% interval 1.0513-1.0834x. All four targeted one-token projections measured 1.13-1.20x versus SGLang, repeating the preceding check gains. **Why it stopped:** The aggregate interval overlaps the previous dense candidate; targeted projection gains do not establish the exact aggregate improvement or gains outside one-token dispatch. **Learning:** A single streaming GEMV can remove split-reduction overhead for one-token projections while retaining existing dispatch for other shapes. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel fused small-batch expert norm](https://posttrainllm.com/experiments/openkernel-moe-fused-norm-20261023) runtime perf worked · caveat **Evidence:** 2026-10-23 H200 scored run 20b3199ecbd847f2 passed with no flags: 1.2236x aggregate, reported 95% interval 1.2053-1.2420x. Small batches measured 1.22-1.49x versus SGLang; the 32-token case measured 0.988x. The preceding check measured 1.2201x aggregate. **Why it stopped:** The existing 32-token path retains a small measured regression, and the gain applies to the organizer workload rather than application throughput. **Learning:** Profile wrapper dispatch as well as kernel arithmetic: the existing fused norm kernel removed several small-batch framework launches while preserving FP32 reduction and invalid-expert masking. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel broad paged split-K attention](https://posttrainllm.com/experiments/openkernel-attention-splitk-broad-20261023) runtime perf regressed **Evidence:** 2026-10-23 H200 check 34c48a28a66d46d3 passed correctness with no flags: 1.0564x aggregate, reported 95% interval 1.0405-1.0722x, below the selected 1.1357x attention candidate. Full-context verification at batches 16 and 32 measured 0.553x and 0.537x versus SGLang. **Why it stopped:** Applying the same split-K kernel to eight-token long-context verification nearly doubled latency; larger sliding verification also lost against the existing fused-preparation candidate. **Learning:** A custom attention kernel can help decoding while losing verification; compare each dispatch regime with the current incumbent as well as the official baseline. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel decoder layer with fused small-batch expert norm](https://posttrainllm.com/experiments/openkernel-layer-fused-norm-20261023) runtime perf worked · caveat **Evidence:** 2026-10-23 H200 scored run 1c749497f0bf4c89 passed correctness with no flags: 1.1070x aggregate, reported 95% interval 1.0903-1.1236x, with all 12 benchmark shapes faster than SGLang. The preceding check measured 1.1086x. **Why it stopped:** This qualifies the MoE norm fusion inside the organizer decoder-layer workload; it does not establish serving throughput or compose the separate new GEMV and split-K attention kernels. **Learning:** A wrapper-level fusion can carry into the full layer, but retain the layer router precision, stream ordering and measured MoE tile fallback. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel narrow paged split-K attention](https://posttrainllm.com/experiments/openkernel-attention-splitk-narrow-20261023) runtime perf worked · caveat **Evidence:** 2026-10-23 H200 scored run 3439c74e04594576 passed correctness with no flags: 1.2804x aggregate, reported 95% interval 1.2612-1.2996x. All 12 benchmark shapes measured faster than SGLang; single-request sliding decoding reached 2.05x. **Why it stopped:** The custom kernel is selected only for one-token decoding and single-request sliding verification on supported configurations; other shapes retain FA3, and full-layer composition has not been measured. **Learning:** Use measured dispatch boundaries to retain custom attention wins and avoid verification regressions; preserve live cache metadata, legal keys and FP32 softmax state. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel decoder-layer GEMV and attention composition](https://posttrainllm.com/experiments/openkernel-layer-compose-20261024) runtime perf worked · caveat **Evidence:** 2026-10-23 H200 check 2928071145b44c58 measured 1.1364x; exact-source scored run e5f50469aadf4638 passed fresh correctness with no flags at 1.1366x, reported 95% interval 1.1196-1.1537x. All twelve scored shapes beat SGLang. **Why it stopped:** The repeated composition gain is qualified and promoted, but 1.1366x remains far from the overall top-ten target. It includes the earlier BF16-dot attention, not the newer joint-head experiment. **Learning:** Compose qualified components while preserving FP32 router input, stream ordering, cache semantics and normalization boundaries. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel native FP8 QK and PV attention](https://posttrainllm.com/experiments/openkernel-attention-native-fp8-20261024) runtime perf regressed **Evidence:** 2026-10-23 H200 check 8c3311ccc5cd4b3c passed correctness with no flags: 1.2223x aggregate, reported 95% interval 1.2040-1.2458x. This is an off-board check, not a scored submission. **Why it stopped:** Aggregate 1.2223x is below selected 1.2804x; full decode at batch 16 and 32 took 230.6 and 443.3 microseconds, versus selected measurements near 189.7 and 350.9. Both use the same keys; native FP8 dots did not improve these regimes. **Learning:** Lower operand precision does not guarantee lower latency; layout conversion and dependent gathers are hypotheses until compiler or hardware evidence establishes their cost. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel small-batch dense split-K and native tiny router](https://posttrainllm.com/experiments/openkernel-dense-splitk-20261024) runtime perf regressed **Evidence:** 2026-10-23 H200 check a2d8bfdabc884c1f passed correctness with no flags: 1.0485x aggregate, reported 95% interval 1.0328-1.0643x. This is an off-board check, not a scored submission. **Why it stopped:** Aggregate 1.0485x is below selected 1.0673x. Pooler at 8 and 16 tokens improved about 10 percent over SGLang, but 32-token dense paths and the tiny-router replacement regressed. **Learning:** Separate public-shape regimes before composing improvements; more splits add a reduction launch and may lose despite sufficient parallel work. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel single-token direct packed NVFP4 SIMT GEMV](https://posttrainllm.com/experiments/openkernel-moe-direct-gemv-20261024) runtime perf regressed **Evidence:** 2026-10-23 H200 check facf2729e11e442c passed correctness with no flags: 0.9291x aggregate, reported 95% interval 0.9152-0.9519x. This is an off-board check, not a scored submission. **Why it stopped:** One-token latency was 271.789 microseconds versus 77.408 for SGLang and 51.41 for the selected source; the two custom GEMVs consumed 97.3 percent of the diagnostic profile. **Learning:** Removing route alignment is insufficient when serial unpack, scaling and repeated reductions dominate; register spilling or occupancy was not measured. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel raw grouped NVFP4 tensor-core experts](https://posttrainllm.com/experiments/openkernel-moe-grouped-20261024) runtime perf regressed **Evidence:** 2026-10-23 H200 check 875ff735dfa84734 passed correctness with no flags: 0.2087x aggregate, reported 95% interval 0.2056-0.2118x. This is an off-board check, not a scored submission. **Why it stopped:** Aggregate 0.2087x is far below selected 1.2236x. At 16 tokens the candidate took 1402.630 microseconds versus 191.744 for SGLang and 157.29 for the selected source; the two custom GEMMs account for 99 percent of the diagnostic profile. **Learning:** Correct tensor-core math does not establish an efficient implementation. The measured regression is inside GEMMs; unpacking cost, resource pressure and compiler layout remain unverified explanations. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel joint-head FP8 paged attention](https://posttrainllm.com/experiments/openkernel-attention-joint-heads-20261024) runtime perf worked · caveat **Evidence:** 2026-10-23 H200 check 717f23591a6f4ff8 passed correctness with no flags: 1.3664x aggregate, reported 95% interval 1.3459-1.3869x. This is an off-board check, not a scored submission. Exact-source score 939615aea8b24931 subsequently passed all sixteen checks with no flags: 1.3723038378x, reported interval 1.3517192802-1.3995453283x; selected. **Why it stopped:** Selected attention is measured on organizer H200 workloads only. Full-layer composition is separately validated by final run beb82b4224554523; no serving or Mac gain is claimed. **Learning:** Loading both contiguous KV heads in one CTA improves this measured decode regime. This result does not establish a general FP8 advantage: the separate-head FP8 variant regressed, and verification still uses FA3. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel grouped NVFP4 experts with bit decoding](https://posttrainllm.com/experiments/openkernel-moe-grouped-bitdecode-20261024) runtime perf regressed **Evidence:** 2026-10-23 H200 check 63bd8567d57d4199 passed all five fresh correctness cases with no flags: 0.2905x aggregate, reported 95% interval 0.2861-0.2948x. This is an off-board check, not a scored submission. **Why it stopped:** Aggregate 0.2905x is below selected 1.2236x. At 16 tokens the candidate took 964.156 microseconds versus 189.056 for SGLang and 157.29 for the selected source. Multi-token diagnostic profiles place 98-99 percent of GPU time in the two grouped GEMMs. **Learning:** Cheaper bit decoding improves the earlier grouped implementation at 0.2087x but leaves every custom multi-token case slower than SGLang. The one-token result uses the unchanged Marlin fallback. Register pressure, spills and instruction costs remain unmeasured. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel single-token NVFP4 bit decoding and vector accumulation](https://posttrainllm.com/experiments/openkernel-moe-direct-fast-20261024) runtime perf regressed **Evidence:** 2026-10-23 H200 check ea95ebe0676248be passed all five fresh correctness cases with no flags: 1.0400x aggregate, reported 95% interval 1.0244-1.0556x. This is an off-board check, not a scored submission. **Why it stopped:** Aggregate 1.0400x is below selected 1.2236x. The custom one-token path took 137.590 microseconds versus 77.680 for SGLang and the selected source's prior 51.414 measurement. Its two custom GEMVs consume 93.4 percent of diagnostic GPU time. **Learning:** Bit decoding and one final reduction per output improve the earlier direct GEMV's 271.789-microsecond result, but still lose to Marlin. Unchanged multi-token fallback timings do not establish a new optimization. Spills, occupancy and instruction costs remain unmeasured. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel additional full-context attention splits](https://posttrainllm.com/experiments/openkernel-attention-more-splits-20261024) runtime perf inconclusive **Evidence:** 2026-10-23 H200 check 6a1c90b00a0c46a5 passed all sixteen fresh correctness cases with no flags: 1.2968x aggregate, reported 95% interval 1.2774-1.3221x. All twelve benchmark cases beat SGLang. This is an off-board check, not a scored submission. **Why it stopped:** The aggregate is only 1.28 percent above selected attention 1.2804x, and their reported intervals overlap. Full-context decode at batches 16 and 32 took 179.835 and 341.780 microseconds versus the joint-head check's earlier 140.938 and 266.767 measurements. This check does not establish a repeatable improvement over the selected source. **Learning:** Doubling the full-context split target preserves correctness but does not establish a meaningful aggregate gain. Diagnostic profiles place 97.1 and 98.6 percent of those full-context decode cases in the partial-attention kernel. The underlying hardware bottleneck remains unmeasured. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel whole-row FP32 tiny-router reduction](https://posttrainllm.com/experiments/openkernel-dense-router-whole-row-20261024) runtime perf worked · caveat **Evidence:** 2026-10-23 H200 check 45dde2ec9cfa4fbe passed all nine fresh correctness cases with no flags: 1.0915x aggregate, reported 95% interval 1.0751-1.1079x. The one-token router took 3.939 microseconds versus SGLang 10.352 and selected source's earlier 10.089. This is an off-board check, not a scored submission. **Why it stopped:** The aggregate point estimate is 2.26 percent above selected dense 1.0673x, but reported intervals overlap and the source remains unscored. The eight-token router took 11.218 microseconds versus the selected source's earlier 10.997, so extending the reduction to eight tokens did not establish a gain. Owner policy parks this source without scoring or promotion. **Learning:** One whole-row FP32 reduction removes the cast launch for the one-token router and shows a substantial measured case gain. The diagnostic profile contains one 2.4-microsecond router kernel at one token and one 10.9-microsecond kernel at eight tokens. A case gain does not establish an aggregate or full-layer improvement. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel whole-row single-token dense and router reductions](https://posttrainllm.com/experiments/openkernel-dense-whole-row-20261024) runtime perf worked · caveat **Evidence:** 2026-10-23 H200 check 350b4c2c87ef4312 passed all nine fresh correctness cases with no flags: 1.1131x aggregate, reported 95% interval 1.0964-1.1298x. The point estimate is 4.29 percent above selected dense 1.0673x, whose reported interval is 1.0513-1.0834x. These intervals do not overlap. This is an off-board check, not a scored submission. **Why it stopped:** The standalone candidate was not scored. Its whole-row reductions were composed into final layer beb82b4224554523, which passed at 1.2503x; that combined result does not isolate their contribution. **Learning:** Whole-row FP32 reductions improve the measured long single-token projections while retaining the short O-projection path. QKV, pooler, unpooler and router measured 10.619, 16.461, 3.348 and 4.242 microseconds versus selected prior 12.569, 17.890, 5.867 and 10.089. Their separate profiles each contain one whole-row kernel; the pooler kernel takes 16.1 microseconds. Hardware bandwidth and occupancy were not measured. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel attention merge/norm/gate fusion](https://posttrainllm.com/experiments/openkernel-attention-merge-20261024) runtime perf regressed **Evidence:** 2026-10-24 H200 off-board check a848d0b7c8794535: passed 16 fresh correctness cases with no flags; 1.3067727260x, reported 95% interval 1.2871711351-1.3263743169x. **Why it stopped:** Below the selected joint-head attention; epilogue fusion alone did not win. **Learning:** Below the selected joint-head attention; epilogue fusion alone did not win. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel fixed-query FA3 scheduling](https://posttrainllm.com/experiments/openkernel-attention-fixedq-20261024) runtime perf regressed **Evidence:** 2026-10-24 H200 off-board check 5a7a3f5158e94210: passed 16 fresh correctness cases with no flags; 1.2911857494x, reported 95% interval 1.2674624602-1.3164755845x. **Why it stopped:** Below selected joint-head attention; preserve the existing FA3 fallback. **Learning:** Below selected joint-head attention; preserve the existing FA3 fallback. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel grouped BF16-decoded NVFP4 experts](https://posttrainllm.com/experiments/openkernel-moe-bf16-decode-20261024) runtime perf regressed **Evidence:** 2026-10-24 H200 off-board check 59b8264c0c8d4e59: passed 5 fresh correctness cases with no flags; 0.2986659409x, reported 95% interval 0.2941859518-0.3031459300x. **Why it stopped:** 96 registers and zero spills do not explain the large regression; grouped GEMMs still dominate. **Learning:** 96 registers and zero spills do not explain the large regression; grouped GEMMs still dominate. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel single-token direct tensor-core experts](https://posttrainllm.com/experiments/openkernel-moe-direct-dot-20261024) runtime perf regressed **Evidence:** 2026-10-24 H200 off-board check cac9617e49ed43eb: passed 5 fresh correctness cases with no flags; 0.9062609933x, reported 95% interval 0.8926670784-0.9198549082x. **Why it stopped:** The direct tensor-core path did not beat the selected Marlin implementation. **Learning:** The direct tensor-core path did not beat the selected Marlin implementation. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel layer local expert-ID mapping](https://posttrainllm.com/experiments/openkernel-layer-local-ids-20261024) runtime perf inconclusive **Evidence:** 2026-10-24 H200 off-board check 622cee5dfcfd4f1e: passed 13 fresh correctness cases with no flags; 1.1552940589x, reported 95% interval 1.1379646480-1.1726234698x. **Why it stopped:** Point estimate improved but reported interval overlaps the previous scored layer. **Learning:** Point estimate improved but reported interval overlaps the previous scored layer. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel layer M128 expert tile](https://posttrainllm.com/experiments/openkernel-layer-m128-20261024) runtime perf regressed **Evidence:** 2026-10-24 H200 off-board check 048fe1129b344184: passed 13 fresh correctness cases with no flags; 1.1346135282x, reported 95% interval 1.1175943253-1.1516327311x. **Why it stopped:** The global-density M128 tile change did not improve the aggregate. **Learning:** The global-density M128 tile change did not improve the aggregate. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel layer normalization and ID-map fusion](https://posttrainllm.com/experiments/openkernel-layer-norm-ids-20261024) runtime perf worked · caveat **Evidence:** 2026-10-24 H200 off-board check ec00131830f54653: passed 13 fresh correctness cases with no flags; 1.1791209960x, reported 95% interval 1.1614341810-1.1968078109x. **Why it stopped:** Best prior layer check; not scored separately. Included in the final composition. **Learning:** Best prior layer check; not scored separately. Included in the final composition. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel joint-head attention layer integration](https://posttrainllm.com/experiments/openkernel-layer-joint-20261024) runtime perf inconclusive **Evidence:** 2026-10-24 H200 off-board check 0c5c8d83f1fa489f: passed 13 fresh correctness cases with no flags; 1.1627907377x, reported 95% interval 1.1453488766-1.1802325988x. **Why it stopped:** Only 2.30 percent over the scored layer with overlapping intervals; failed the prior promotion gate. **Learning:** Only 2.30 percent over the scored layer with overlapping intervals; failed the prior promotion gate. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel wider attention key tile](https://posttrainllm.com/experiments/openkernel-attention-bn256-20261024) runtime perf regressed **Evidence:** 2026-10-24 H200 off-board check 30ed1e95b2984d51: passed 16 fresh correctness cases with no flags; 1.2292738162x, reported 95% interval 1.2108347089-1.2477129234x. **Why it stopped:** Large full-decode kernels used 255 registers and 96 spills; wider tiles regressed. **Learning:** Large full-decode kernels used 255 registers and 96 spills; wider tiles regressed. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel shared-load verification attention](https://posttrainllm.com/experiments/openkernel-attention-joint-verify-20261024) runtime perf regressed **Evidence:** 2026-10-24 H200 off-board check 583215fc82cb43db: passed 16 fresh correctness cases with no flags; 0.8432330590x, reported 95% interval 0.8305845631-0.8591197270x. **Why it stopped:** 254-255 registers with no spills; shared loading did not make this mapping faster. **Learning:** 254-255 registers with no spills; shared loading did not make this mapping faster. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel lossless single-token dense pooler](https://posttrainllm.com/experiments/openkernel-dense-lossless-20261024) runtime perf regressed **Evidence:** 2026-10-24 H200 off-board check c490ffa4d6aa4d0e: passed 9 fresh correctness cases with no flags; 1.0309774607x, reported 95% interval 1.0155127988-1.0464421226x. **Why it stopped:** Below selected dense despite a lossless weight/layout treatment. **Learning:** Below selected dense despite a lossless weight/layout treatment. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel large-tile dense pooler](https://posttrainllm.com/experiments/openkernel-dense-large-pooler-20261024) runtime perf regressed **Evidence:** 2026-10-24 H200 off-board check d4d13353fe054eb6: passed 9 fresh correctness cases with no flags; 1.0465768799x, reported 95% interval 1.0308782267-1.0622755331x. **Why it stopped:** Larger pooler tiles did not beat selected dense. **Learning:** Larger pooler tiles did not beat selected dense. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) ### [Open Kernel final valid-local expert decoder composition](https://posttrainllm.com/experiments/openkernel-final-composition-20261024) runtime perf worked · caveat **Evidence:** 2026-10-24 H200 scored run beb82b4224554523: passed 13 fresh correctness cases with no flags; 1.2503460947x, reported 95% interval 1.2315909033-1.2691012861x. Final check b5f1d79d24524705 measured 1.2480231870x and met the single-score gate. Source SHA-256 48655c1bef954efc8c894169eacc504c1338a44a102e18a62bd5a4d1f329e894. **Why it stopped:** Final layer is 10.01 percent faster by aggregate speedup ratio than the previous scored layer, but rank 283/749 missed the top-100 target. Component contributions are not isolated. **Learning:** Final layer is 10.01 percent faster by aggregate speedup ratio than the previous scored layer, but rank 283/749 missed the top-100 target. Component contributions are not isolated. **Disposition:** Experiment closed after scored run beb82b4224554523 finished at rank 283/749, outside the owner top-100 target. Retain the research artifact; no further jobs. direct evidence [Proof](https://posttrainllm.com/docs/learn/openkernel) [Recipes](https://posttrainllm.com/recipes) [Learn](https://posttrainllm.com/learn/paths/runtime-and-agents) **Archive closed; lab open.** These records are not a hidden backlog. New work starts only when you open a fresh question after learning, freeze its ruler, and create a new recipe and experiment ID.