Every experiment. Including the ones that said no.
This is the complete evidence-backed history of posttrainllm. Nothing is
queued or quietly “not tried.” Each historical question ends in a
measured result or a final disposition, with the lesson preserved for
your next fresh experiment.
75accounted attempts
0unresolved attempts
64exact-evidence records
14experiment families
outcomes
The shape of the work
Negative results remain first-class evidence.
search the lab
All recorded attempts
Showing all 75
sql worked · caveat
Toy SQL SFT
sft
lora
dora
Evidence
execution `0.167 -> 0.833` on 6-row fixture
Why it stopped
Train/eval overlap made the apparent improvement non-shippable.
Learning
The SFT loop mechanics work, but eval splits must be frozen before claiming quality.
Disposition
Use only as a loop sanity check; rely on heldout SQL evals for real claims.
Ref-free SimPO update collapsed the policy and produced degenerate fenced/prose outputs.
Learning
Preference tuning needs reference anchoring, smaller updates, and composed-eval checks.
Disposition
Retry with reference-anchored DPO or much lower LR/step count on the same frozen pairs.
sql rejected
SQL candidate selection
candidate-selection
Evidence
tooling and smoke exist, but no frozen candidate-selection dataset, baseline, or model run was approved before project closure
Why it stopped
The proposed run never acquired a frozen target/eval contract, and running it retrospectively would start a new experiment rather than complete an existing measured claim.
Learning
Candidate selection remains a plausible future method, but an unfrozen recipe is not unfinished evidence for this project.
Disposition
Closed for this project. Reopen only as a fresh owner-led experiment after the learning phase with new data and a frozen gate.
sql rejected
Offline rollout / OAPL-style SQL update
rlvr
Evidence
plan renderer exists, but the required candidate-selection reward surface was never validated and no model run was authorized
Why it stopped
The experiment's explicit prerequisite did not exist, so an OAPL-style update would not have had a trustworthy reward and could not answer its intended question.
Learning
Offline policy updates are downstream of reward validation; skipping the prerequisite is not a useful experiment.
Disposition
Closed for this project. A future fresh experiment must validate its reward surface first.
sql rejected
Controlled SQL LoRA rank sweep
lora
Evidence
no controlled sweep was run because no next target/eval was frozen after the confounded historical comparisons
Why it stopped
Without one active frozen recipe, a rank sweep would measure a new arbitrary setup rather than resolve the historical confound.
Learning
Hyperparameter sweeps are valid only inside one frozen experiment contract; they are not project-completion chores.
Disposition
Closed for this project. Any future rank study begins as a new experiment with fixed data, seed, steps, learning rate, and eval.
file ops worked · caveat
Qwen3-4B file-ops distilled specialist
distill
Evidence
file-ops hard gate `0.58 -> 1.00`; breadth `0.596 -> 0.423`
Why it stopped
The specialist improved the narrow file-ops gate but regressed out-of-domain breadth.
Learning
The artifact is useful only with routed-only positioning; it should not be sold as a general planner.
Disposition
Keep routed-only warnings prominent and add loader/pull smoke before wiring consumers.
file ops worked · caveat
Qwen3-4B ReST fused breadth recovery variant
rest
distill
Evidence
depth `100%`, breadth `65%` vs stock breadth `60%`
Why it stopped
Historical runtime and raw trace artifacts were not preserved, so the package cannot claim current latency/RAM/tok-s or a fresh qualitative review.
Learning
Existing public weights can be promoted honestly when the measured delta, routing boundary, and missing evidence are explicit.
Disposition
Keep it research-only unless a product-specific target justifies a fresh heavy eval and re-distillation.
file ops regressed
Qwen3-4B multibackend distilled variant
distill
Evidence
depth `100%`, breadth `31%`
Why it stopped
Negative transfer damaged breadth even though depth remained high.
Learning
Multi-backend distillation can over-specialize and needs breadth gates by default.
Disposition
Keep as failed comparison evidence; do not promote without a new breadth-preserving recipe.
archive model inconclusive
VibeThinker 3B agentic distilled variant
distill
Evidence
weights preserved; current eval promotion missing
Why it stopped
The weights exist, but the docs do not preserve a current factory eval proving a win.
Learning
Preserved weights are not a ship decision.
Disposition
Run the factory eval gate and publish a before/after report before using it as a specialist package.
apple fm failed
Apple on-device Foundation Models action-grounding probe
evaluation
Evidence
BFCL agentic `25%` full catalog / approximately `0%` compact; Pace planner action-grounding `13%`
Why it stopped
Apple's on-device model could not ground actions well enough and its 4096-token context could not hold a real tool catalog.
Learning
Apple Foundation Models are useful as a free routing floor and intel source, not as the core capability dependency.
Disposition
Keep Core ML for our own weights as an optional future battery optimization, not a model-dependency bet.
pace planner inconclusive
Pace planner v1-v4 early versions
sft
lora
Evidence
datasets, builder scripts, and planned recipes survive; no per-version score was recorded
Why it stopped
No per-version result was kept. The v11 ship gate names this directly: v1 through v10 cycled because there was no fixed bar, no fixed eval, and no immutable record of pass criteria.
Learning
An unmeasured version is indistinguishable from a version that never ran. Four consecutive attempts left no score behind.
Disposition
Do not attempt to reconstruct scores for v1-v4; treat the series as starting at v5, the first version with a recorded number.
pace planner worked · caveat
Pace planner v5 hand-crafted gold + teacher paraphrase
sft
lora
Evidence
17/19 (89%) on fm-fixtures v1 and 6/15 (40.0%) on fm-fixtures-v2
Why it stopped
The 17/19 was measuring format compliance, not capability: a rule-based FakePace endpoint scored 19/19 on the same fixture set.
Learning
A high score on a fixture a deterministic endpoint can also ace is a measurement of the harness, not the model.
Disposition
Treat fm-fixtures v1 as retired; only fm-fixtures-v2 numbers are comparable.
pace planner regressed
Pace planner v6 label-based output
sft
lora
Evidence
14/19 (74%) on fm-fixtures v1 and 3/15 (20.0%) on fm-fixtures-v2, 20pp below v5
Why it stopped
The label-based SFT destroyed capability, not just output format.
Learning
Changing the output representation is a capability change, not a formatting change.
Disposition
Revert to the v5 corpus for further augmentation work.
pace planner worked · caveat
Pace planner v6.1 four-adapter repair sweep
sft
lora
Evidence
10/19, 8/19, 2/19, 0/19, then 14/19 at 100 steps and 17/19 at 300 steps, reaching 19/19 after a serving-side minLength fix; 4/15 (26.7%) on fm-fixtures-v2
Why it stopped
The apparent collapse was two harness bugs, not training: SFT rows omitted the Pace system prompt that serve and eval both included, and a brace-counting JSON extractor was faulty.
Learning
A four-way score collapse traced to prompt mismatch between training and serving. Pin the prompt, schema, and grammar triple across train, serve, and eval before believing any comparison.
Disposition
Recover the full sweep with `git show d1efcb8^:docs/prds/factory-pace-planner-v6_1.md` before repeating any step-count ablation; it is the only one in the series.
pace planner superseded
Pace planner v7 tools-in-prompt
sft
lora
Evidence
tools-in-prompt serve, verb-enum grammar, and data/eval harnesses shipped; the SFT was never run
Why it stopped
The version was never trained, and later v8-v11 planner work replaced its proposed prompt/schema direction and supplied the relevant capability evidence.
Learning
v7 is a numbering gap, not a training version. The eleven-version count includes a version that was never trained.
Disposition
Closed as a numbering gap. Do not cite v7 as evidence or revive it outside a new owner-led experiment.
pace planner inconclusive
Pace planner v8 augmented corpus
sft
lora
Evidence
reported 11/15 (73.3%) on fm-fixtures-v2, later re-measured at 5/15 (33.3%) from the same weights and same fixtures
Why it stopped
The headline score did not reproduce. Re-running the same eval against the same baked weights yielded 33.3%, and the gap was root-caused as prompt and schema config drift rather than any model change.
Learning
The 73% phantom. A score is only a score if the prompt, schema, and grammar it was measured under are recorded with it.
Disposition
Treat 33.3% as the working v8 baseline and never compare across versions without pinning the harness config.
pace planner worked · caveat
Pace planner v9 compose
sft
lora
Evidence
70% on new compose fixtures with non-compose held at the v8 level; as-shipped re-measurement 9/15 (60.0%); unhappy-path n=130 gave 0/40 ambiguous, 13/60 out-of-scope, 1/30 destructive
Why it stopped
It shipped as the specialist but never became Pace's runtime planner, and the unhappy-path drill put it at a capacity wall: 0% ambiguous and 3% destructive.
Learning
Winning the happy-path fixture says nothing about ambiguous, out-of-scope, or destructive inputs, which is where the product actually needed the planner.
Disposition
Keep as the last shipped specialist; Pace runs qwen3-30b-a3b instead.
pace planner inconclusive
Pace planner v10 parameterized actions
sft
lora
Evidence
full recipe, action registry, schema, and eval plan are committed; no score was ever recorded and it is not established that training completed
Why it stopped
No v10 score exists anywhere in the repository. The last status recorded is that training was pending, while the v10 corpus went on to become the base of v11.
Learning
v10 repeated v6's move of changing the output representation, and because it was never measured the v6 regression could not be checked for recurrence. Its unmeasured corpus was then inherited by v11.
Disposition
Do not reuse a v10-derived corpus without first measuring the representation change in isolation.
pace planner failed
Pace planner v11 ship-gate run
sft
lora
Evidence
failed all six gate dimensions: BFCL pace-12 27.1% against a 40% bar, out-of-scope 0/30 against 80%, ambiguous 0/20 against 50%, destructive 5/10 against 90%, with train loss 0.001
Why it stopped
A 0.6B planner did not reach the fixed bar on any dimension despite the lowest training loss in the series, and the first v11 attempt was voided outright by the DoRA infrastructure failure.
Learning
Train loss 0.001 alongside 0% on out-of-scope and ambiguous inputs is the capacity ceiling stated plainly. The one-shot fixed gate was what finally ended the cycle.
Disposition
Do not train v12. The specialist planner track is closed and Pace ships qwen3-30b-a3b.
pace planner regressed
Pace clarify-v1 Qwen3-4B LoRA
lora
Evidence
38 contrastive rows moved ambig only `0% -> 5%` while OOS regressed `80% -> 33%` in the retrospective; drilldown records the same attempt as `2%/40%/50%` on ambig/oos/destructive
Why it stopped
Small-corpus LoRA on a competent base caused catastrophic interference across planner dimensions.
Learning
Fine-tuning a strong base on a thin behavior slice requires coverage for every dimension that must not regress.
Disposition
Do not fine-tune the planner on narrow clarify rows unless the corpus also covers OOS, destructive, happy path, and breadth gates.
pace planner worked · caveat
Pace two-stage ambiguity detector shim v2
two-stage-shim
Evidence
ambig moved `0% -> 20%`, while OOS dropped `78% -> 55%` and destructive stayed `67%`; not shipped on top of Gemma
Why it stopped
The shim improved ambiguity detection for the 4B floor but over-triggered and became unnecessary once Gemma reached the same ambig score with better OOS/destructive behavior.
Learning
Rule wrappers can expose the missing behavior, but they still need the same regression gates as model updates.
Disposition
Keep as an option only if shipping the 4B footprint floor; do not stack it on the stronger Gemma path.
pace planner inconclusive
Pace reasoning-model unhappy-path drill
evaluation
Evidence
Qwen3-4B-Thinking scored `0%/10%/0%` and DeepSeek-R1-Distill-Qwen-7B scored `0%/3%/0%` on ambig/oos/destructive under a 300-token cap
Why it stopped
The evaluation budget was consumed by think traces, so the apparent zero scores were not a fair model-quality verdict.
Learning
Reasoning models need adequate max tokens and a stop-on-think strategy before interpreting planner scores.
Disposition
Re-run with max_tokens >= 1024 and stop on `</think>` only if Gemma or the locked planner fails in production.
pace planner worked · caveat
Pace Gemma-3-12B unhappy-path drill winner
evaluation
Evidence
Gemma-3-12B scored `22%` ambig, `82%` OOS, `77%` destructive and won all three unhappy-path dimensions in the n=130 drill
Why it stopped
Gemma won the drill but ambiguity remained poor in absolute terms and the decision was later superseded by the general planner lock.
Learning
A stronger zero-shot base beat the local fine-tunes, but model-selection evidence can be superseded by later broader gates.
Disposition
Keep as historical unhappy-path evidence; prefer current `PROJECT_STATUS.md`/`docs/NEXT.md` for active planner decisions.
pace planner worked · caveat
Pace Qwen3.5 challenger round
evaluation
Evidence
Qwen3.5-9B scored `18%/97%/63%`; Qwen3.5-4B scored `15%/55%/70%` on ambig/oos/destructive; champion stood
Why it stopped
Qwen3.5 improved some dimensions, especially OOS for 9B, but did not beat the stored champion across the full swap decision.
Learning
Single-dimension wins do not justify a planner swap when another required dimension regresses.
Disposition
Use `scripts/pipelines/eval_planner.sh` for future challengers and keep swap/no-swap verdicts tied to all required dimensions.
browser product worked · caveat
Browser Memory64 large-preset runs
Evidence
larger browser presets hit Memory64 OOB; matching Node direct-export reproducer passed XL; `INITIAL_MEMORY=268435456` fixed up to XL while Behemoth still exercises growth
Why it stopped
The original retries treated the OOB as a model-size problem before the matching reproducer showed a browser pthread plus SharedArrayBuffer memory-growth race.
Learning
A reproducer must match the production host and call path before it can explain a browser-only WASM failure.
Disposition
Keep the higher initial memory for supported presets and only claim Behemoth after a SAB-view-aware growth path lands.
browser product worked · caveat
Browser speedup headline curve
Evidence
single `9.7x` headline was replaced by a preset curve: Small `2.6x`, Medium `6.8x`, Large `9.3x`, XL `12.1x`
Why it stopped
The original one-number headline made a shape-dependent performance result look like a universal property of the code.
Learning
If a performance result depends on preset size, publish the curve rather than the most convenient point.
Disposition
Keep future public performance artifacts tied to config-specific tables with reproduction context.
browser product worked · caveat
Browser default corpus fix
Evidence
the old default corpus was `863 bytes`; a Huge model memorized it with train loss `0.14`; default changed to TinyShakespeare at about `1.1 MB`
Why it stopped
The demo was trying to prove learning with a corpus so small that memorization looked like training success.
Learning
Default data is part of the product claim; a training demo's default corpus must produce non-memorized samples.
Disposition
Keep corpus defaults in the same parity review as model and optimizer defaults.
browser product worked · caveat
Browser learning-rate default drift
Evidence
browser default LR was `3e-3` while Python reference used `3e-4`; training plateaued around loss `2.45` until the browser default was changed
Why it stopped
Reference/kernel parity tests covered math drift but not hyperparameter default drift.
Learning
Reference defaults are part of correctness, not just documentation.
Disposition
Add or keep a config-default parity check whenever browser and reference paths share a training story.
browser product worked · caveat
Browser WebGPU export serialization
Evidence
a trained WebGPU run was lost because `webgpu/gpu_model.ts` had no checkpoint serialization; `exportState` was then implemented and worker plumbing updated
Why it stopped
The training path could produce useful weights that the WebGPU backend could not export, and auto-cleanup destroyed a real trained artifact.
Learning
Long-running browser training is not product-complete until the smallest export path is smoke-tested before the long run.
Disposition
Smoke export on a short WebGPU run before any long browser training claim.
browser product rejected
Browser generation streaming and KV cache
Evidence
tokens/sec counter was added, but browser generation still lacked KV cache and true token streaming; task `#72` was logged
Why it stopped
The browser track completed without implementing this feature, and no measured claim depends on pretending the typewriter animation is true streaming.
Learning
A typewriter animation over an already-complete response is not true streaming.
Disposition
Closed for this project with the limitation explicit. A future browser product may reopen true streaming as a new feature.
runtime perf worked · caveat
WASM backward-scratch reuse
Evidence
standard `304 -> 305 ms/step`, capable `632 -> ~640 ms/step`
Why it stopped
Caching backward scratch buffers removed hot-path allocations but produced no measurable speedup because compute, not malloc, was the bottleneck.
Learning
Hot-path hygiene is not the same as a performance win; measure the real step time before claiming speedup.
Disposition
Keep the allocation cleanup, but do not prioritize allocator work until profiling shows allocation cost again.
runtime perf worked
WASM SIMD training speedup
Evidence
standard `304 -> 191 ms/step`, capable `632 -> 391 ms/step`, net `1.6x` speedup
Learning
Emscripten/LLVM autovectorization was the right browser-training speed lever before hand-written kernels.
Disposition
Keep SIMD enabled by default and use the WASM smoke gate to protect training correctness.
runtime perf worked · caveat
Browser WebGPU training loop
Evidence
24 kernel parity checks, GPU overfit gate loss `5.55 -> 0.002` in 150 steps, and headless-browser e2e passed
Why it stopped
Correctness is proven, but speed is not measured because headless Chromium exposes SwiftShader software WebGPU rather than a real GPU.
Learning
GPU API correctness can be CI-proven, but performance claims require a hardware adapter measurement.
Disposition
Collect the documented real-device WebGPU vs WASM-SIMD benchmark before quoting a browser GPU speedup.
runtime perf worked · caveat
WASM register/cache-blocked matmul
Evidence
microbench `3-4x`, but end-to-end WASM training only `1.03-1.06x` after forward blocking
Why it stopped
The isolated microbench overstated impact because the real WASM training workload was threaded, already SIMD-vectorized, and dominated by backward matmuls.
Learning
Optimize and report on the real workload, not only on isolated single-kernel microbenches.
Disposition
Keep the bit-exact forward kernel for inference, but judge future perf work with end-to-end training and generation benches.
runtime perf worked · caveat
WASM backward_dA blocking
Evidence
fwd+bwd-blocked reached small `90.3`, medium `317.6`, large `1057.4`, xl `1643.6`, or about `1.10-1.13x` over naive
Why it stopped
The backward_dA optimization improved end-to-end training, but the remaining backward_dB path still limits the total speedup.
Learning
Training speed follows the backward-pass bottleneck; forward-only wins are capped until dA and dB are addressed.
Disposition
Only continue matmul blocking if backward_dB or generation benches show enough remaining headroom.
runtime perf worked · caveat
Speculative-decoding heads Medusa/EAGLE smoke
Evidence
baseline `232 tok/s`; Medusa 50-step `202 tok/s` at `21.7%`; Medusa 300-step `375 tok/s` at `20.8%`; EAGLE-2 50-step `200 tok/s` at `26.5%`, all bit-identical where verified
Why it stopped
The heads trained and verified losslessly, but acceptance stayed far below the paper recipes because this run lacked long training, logit distillation, tree attention, and a full EAGLE transformer draft block.
Learning
Speculative heads are a valid local speed path, but the production recipe is data/training heavy rather than a quick sidecar smoke.
Disposition
Revive only with self-generated data, logit-KL distillation, tree verification, and a real acceptance-rate gate.
architecture worked · caveat
MoE dense-compute smoke
Evidence
Dense MLP `842 K` params loss `6.09 -> 1.76` at `55.6 step/s`; MoE 4 experts top-2 `2.42 M` params loss `5.95 -> 1.68` at `29.3 step/s`
Why it stopped
The MoE architecture, save/load, sample, eval, and inspect paths work, but the current compute path runs every expert and therefore does not deliver sparse-dispatch savings.
Learning
MoE is a capacity feature until sparse gather/scatter or grouped matmul lands; it should not be marketed as a compute-saving win yet.
Disposition
Return to MoE only with MLX scatter_add/grouped matmul support or a scoped custom Metal sparse-dispatch kernel.
runtime perf worked · caveat
CPU speedup bundle
Evidence
all four items ON reached median `6.8 step/s` vs `5.0 step/s` baseline, `+36%`, with final loss `4.188-4.192` unchanged
Why it stopped
The bundle produced a real host-side speedup, but QoS and compiled cosine LR were individually in the noise, prefetch stayed single-digit, and open CPU items remained.
Learning
The big win was fused accumulation inside the compiled trace; individual CPU tweaks need isolation before being credited.
Disposition
Use the bundle as shipped, but keep Rust-FFI BPE, preallocated CPU buffers, and SVD-on-CPU AMX as separate future attempts.
runtime perf worked · caveat
Cold-start bundle
Evidence
mmap saved `10-20 ms` and `25 MB` peak RSS on `demo.tinygpt`; lazy embedding deferred `262.1 KB`; package tests passed `43/43`
Why it stopped
The cold-start path improved the measured demo case, but the 250 MB target file was missing, async spinner overhead added about `30 ms`, and persistent Metal pipeline caching was infeasible through public MLX-Swift APIs.
Learning
mmap and direct byte construction are good scaling levers, but large-model cold-start claims need large-model measurements.
Disposition
Re-measure on a real 50-250 MB checkpoint and keep persistent Metal cache work parked unless MLX-Swift exposes pipeline descriptors.
Checkpointing recovered large-model memory but hurt small-model memory and speed, and the OOMGuard estimator still reports uncheckpointed activation projections.
Learning
Activation recompute is a scale-dependent lever; it should stay opt-in for models where memory, not step time, is the blocker.
Disposition
Update memory estimation for `--grad-checkpoint` and consider selective layer checkpointing before broad default use.
runtime perf worked · caveat
KV cache optimization bundle
Evidence
prompt-cache hot run `1.005 s -> 0.106 s`; pre-alloc decode stayed `706 -> 711 tok/s` with physical cache flat at `6.3 MB`
Why it stopped
Persistent prompt cache delivered the TTFT win, but pre-alloc was kept opt-in, pre-alloc does not compose with KIVI or StreamingLLM, and some cache keying edge cases remain.
Learning
Reuse of prefilled KV is the user-visible win; cache storage policy changes need composition gates before becoming defaults.
Disposition
Keep prompt-cache as the main shipped path and revisit pre-alloc defaults only after KIVI/StreamingLLM interactions are designed.
runtime perf worked · caveat
YOCO cache-halving smoke
Evidence
huge cache at 206 tokens dropped `5.1 MB -> 2.5 MB`, while throughput moved `767 -> 676 tok/s`
Why it stopped
YOCO halved decode KV cache memory, but throughput dropped at ctx=256 and HF checkpoint round-trip or production-quality YOCO weights were not proven.
Learning
YOCO is a long-context memory lever, not an immediate short-context speedup.
Disposition
Treat YOCO as a long-context experiment until anchor tuning, HF round-trip, and long-context quality gates exist.
runtime perf worked · caveat
StreamingLLM + KIVI cache compression
Evidence
StreamingLLM reduced 500-token cache `12.6 MB -> 6.4 MB`; int8 KIVI matched greedy baseline `100%` and int4 averaged `96%` prefix match
Why it stopped
The cache compression worked in smoke tests, but int4 precision still used int8 storage bytes, K scales recompute every step, and no held-out perplexity gate measured quality.
Learning
KV compression can preserve greedy decode locally, but storage format and quality gates matter before making long-context claims.
Disposition
Add packed int4 storage or grouped KIVI scales only after a cached-forward perplexity or trajectory-match eval exists.
runtime perf worked · caveat
Data-side sample packing and BPE-dropout
Evidence
sample packing flattened CoV `0.582 -> 0.061`; BPE-dropout 100-step smoke stayed stable with final loss `7.382` baseline vs `7.597` dropout
Why it stopped
Sample packing's histogram property was proven, but BPE-dropout was only a stability smoke and did not prove downstream generalization on this corpus.
Learning
Data-pipeline regularizers need behavioral evals, not just stable loss curves, before being treated as quality improvements.
Disposition
Use sample-packing where length bias matters, and evaluate BPE-dropout on rare-word or morphology-sensitive gates before promotion.
runtime perf worked
MLXFast SDPA and tied-embedding audit
Evidence
audit found SDPA call sites and tied-embedding build/save/load paths clean; build succeeded and sample smoke reached `603 tok/s` on demo and `261 tok/s` on mega5
Learning
The suspected SDPA mask/contiguity and tied-embedding duplication issues were not present.
Disposition
Keep this as positive verification; only reopen if new SDPA call sites or untied embedding save paths are added.
tool calling superseded
A1 first tool-calling specialist harness
Evidence
the recipe and acceptance harness shipped; later Qwen3-4B file-ops distillation and BFCL-family evaluations exercised the specialist contract with measured results
Why it stopped
The original A1-specific adapter run was never executed, but its question and workflow were overtaken by later measured tool-calling specialists and should not be recreated for historical bookkeeping.
Learning
Reusable infrastructure can be superseded by stronger downstream evidence even when its first named model run never happened.
Disposition
Closed as superseded. Use the measured file-ops/ReST lineage for evidence and start any new tool-caller as a fresh experiment.
sql worked · caveat
B1 SQL eval infrastructure
Evidence
`posttrainllm eval-sql` shipped and smoke-tested; training/generation originally pending GPU
Why it stopped
The scoring half existed before the live SQL factory run, but the original PRD did not include trained adapter evidence.
Learning
Execution accuracy over SQLite is the right reusable gate; infra alone is not a specialist result.
Disposition
Continue using eval-sql as the frozen SQL gate and attach run folders to model claims.
factory docs worked · caveat
Factory run schema
Evidence
`docs/factory/run-schema.md` defines the expected run folder
Why it stopped
The schema exists, but real train/eval commands still need to emit every file automatically.
Learning
A schema is necessary but not sufficient; publish checks must enforce complete run folders.
Disposition
Wire live train/eval commands to produce schema-complete run folders automatically.
The model became a paraphraser rather than a repairer
Learning
Minimum-edit repair is a narrow band between copying everything and rewriting everything
Disposition
No further training under `adapter-recipe-v1`.
chess failed
Character Chess 44.53M masked 10k pilot (2026-08-05)
sft
Evidence
validation exact `6.16% random -> 10.54%` (+4.38 points), test `6.23% -> 10.33%` (+4.10); raw and guarded policies each won 0/6 games against random legal play
Why it stopped
Completion-only SFT learned a small held-out move preference but missed the frozen +10-point promotion gate and produced no full-game win advantage over random legal play; eight guard interventions converted no wins.
Learning
Legal constrained decoding and lower move-prediction loss are not game intelligence; require full-game transfer before scaling a tiny specialist.
Disposition
Do not run the 100k, 1M, or 2M stages under this recipe. Preserve the failed artifact and select a different bounded specialist target.
tool calling failed
Needle 2 base public tool-selection gate (2026-08-31)
evaluation
Evidence
32/94 exact tool selections (34.0%), with 8 out-of-scope false calls and 2 destructive-action calls
Why it stopped
The base artifact was inaccurate on Pace and file operations, produced unsafe false actions, and exposed no useful confidence threshold.
Learning
Tiny packaging and fast decode do not make a tool selector safe enough for action routing.
Disposition
Reject the base artifact for Pace; preserve it only as a tiny-runtime baseline.
overall exactness moved 32/94 -> 36/94, while Pace regressed 10/28 -> 6/28, file operations stayed 0/6, and 3 out-of-scope prompts still produced actions
Why it stopped
Oracle task-family routing reduced schema size and latency but did not repair capability or safety; the target Pace slice became worse.
Learning
Catalog restriction is a latency lever, not a capability or safety rescue for this base model.
Disposition
Reject this integration path; require a materially different training hypothesis and a new frozen gate for any future Needle-derived work.
browser product worked · caveat
Parakeet WGSL browser-ASR smoke (2026-09-01)
evaluation
Evidence
17m 10s of audio transcribed in 7.225s after cold load and 7.68-7.80s warm (132-143x real-time); warm transcript was byte-stable
Why it stopped
Runtime speed and repeatability passed, but no reference transcript or WER scorer established accuracy and the optional first download is 386.5 MiB.
Learning
Browser WebGPU ASR is technically viable on Apple Silicon, but speed cannot substitute for a controlled accuracy and UX gate.
Disposition
Preserve as a validated proof; compare against WhisperKit only as a fresh owner-led experiment after the learning phase.
offhours inconclusive
OffHours Devin stress validation (2026-08-21)
evaluation
Evidence
30/30 workday-conditions and 1,200/1,200 claims completed, but the clean condition scored 95.0% and missed the frozen 98% calibration gate
Why it stopped
The benchmark ruler failed calibration before the stress comparison could support a confirmatory semantic-effect claim.
Learning
A complete run is still inconclusive when the clean frontier calibration misses its preregistered gate.
Disposition
Closed as calibration evidence; do not use it as a causal stress result.
offhours worked · caveat
OffHours persistent-tension pilot (2026-08-21)
evaluation
Evidence
unresolved tension scored 198/200 versus 197/200 resolved, a -0.5 percentage-point paired error difference with 95% interval -3.0 to +1.5
Why it stopped
The frozen experiment found no work-quality penalty, while provider prompt-token, quantization, and model-file provenance remained unavailable.
Learning
Unresolved narrative tension did not degrade measured work quality under the exercised conditions; a null result is a valid result.
Disposition
Closed as report-only evidence; do not tune scenarios after the null.
offhours worked · caveat
OffHours context-saturation boundary (2026-08-21)
evaluation
Evidence
the ordered volume ladder found its first reproducible boundary at 2,000 words/event or 8,000 submitted words/day
Why it stopped
The boundary occurred in the neutral arm and therefore measures raw context volume or context management, not personal-obligation semantics; full provider/model provenance is incomplete.
Learning
The supported result is a neutral context-volume boundary, not a psychological or family-tension effect.
Disposition
Closed as report-only boundary evidence with the interpretation constrained to raw volume.
game benchmarks rejected
Character Chess benchmark candidate audit (2026-08-22)
evaluation
Evidence
100 stable candidates produced 86 admitted positions, but the best recorded frontier screen reached only 75% exact agreement
Why it stopped
The frontier calibration did not approach the required ceiling, external-model coverage was incomplete, and the model alias was mutable, so the suite could not be frozen as a fair ruler.
Learning
A benchmark candidate set is not a benchmark until the frontier ceiling validates the ruler.
Disposition
Closed as candidate-only evidence; design any future chess ruler independently after the learning phase.
game benchmarks rejected
Character 2048 frontier benchmark screen (2026-08-04)
evaluation
Evidence
pinned Sonnet matched random legal play at 0.995x mean score and pinned Opus reached only 1.058x over three complete pairs before a provider disconnect
Why it stopped
The proposed character interface did not expose a reliable frontier intelligence advantage, so training a 30-50M student would measure ruler noise.
Learning
Validate the teacher and interface before spending compute on a specialist; a failed ruler should stop training.
Disposition
Closed in the current character form; keep the negative benchmark replay public.
game benchmarks rejected
Character game arena candidate report (2026-08-05)
Evidence
the arena retained four chess matches and seven complete 2048 pairs, while every rating remained unqualified
Why it stopped
Chess missed minimum-match and forfeit gates, while 2048 missed minimum-pair or complete-source gates; cross-game ranking would have overstated sparse development evidence.
Learning
A rating calculation is not a public rating until each underlying competition family passes its qualification contract.
Disposition
Closed as a transparent candidate report with no Elo, human-equivalence, or specialist-win claim.