> Canonical page: https://posttrainllm.com/devlog

[← posttrainllm](https://posttrainllm.com/) · [changelog](https://posttrainllm.com/changelog) · [roadmap](https://github.com/PostTrainLLM/posttrainllm/issues) · devlog · [docs](https://posttrainllm.com/docs/)

# Devlog — how the lab got built

 Decisions made, measurements taken, things that didn't work — the whole build, from a
 from-scratch GPT in a browser tab to a Mac-local specialist factory. The dialogue from
 the AI-pairing sessions is condensed; the numbers are verbatim from runs on this codebase.

 Two threads run through it: the browser / WebGPU learning track (the
 kernel work below) and the Mac factory (post-training, evals,
 packaging — the newer entries). Every "measured" value is run-on-this-machine, not
 extrapolation. Companion logs: the
 [attempt ledger](https://posttrainllm.com/docs/attempt-ledger/) (every experiment + decision), the
 [learning log](https://posttrainllm.com/docs/learn/curriculum/), and the
 [recipes](https://posttrainllm.com/recipes).

## The site you're reading — a dark research-lab redesign shipped

 The site had drifted into "clean editorial SaaS" — cohesive but AI-looking, and split
 across a bright web and a dark app. This pass unified everything into one
 instrument : a single dark design system (crafted deep-ink palette with a faint
 grain, teal as the live signal, Bricolage Grotesque for display), the loss-curve logo as
 the universal mark, a manifesto, machine-transparent claims, and structured-data SEO.

 Lesson. A design token system earns its keep at redesign time: flipping
 system.css from light to dark reskinned ~100 doc pages, the artifacts pages,
 and the homepage at once, because every surface read its color from the same variables.
 The disconnection wasn't taste — it was two palettes ( #8db4ff blue on the
 web, #48e5c2 teal in the app). One accent fixed it.

## The factory loop closed — and the first ship-decision was "retry" measured

 The north star stopped being "another surface" and became one canonical run:
 target → data → post-train → eval → package → report , ending in a documented
 ship/reject/retry decision. On 2026-07-04 it executed end-to-end for the first time on a
 real candidate — the frozen qwen06-sql-hygiene-dpo target — and produced a
 schema-valid run folder that factory-run validate accepts.

Every run now assembles a folder — config · dataset · eval-baseline ·
 eval-candidate · decision · report — via scripts/factory/assemble_factory_run.py ,
 which computes the eval delta and hashes the real dataset into provenance. The one proof
 still missing from the whole project is a run whose decision is ship .

## The DPO saga — a policy collapse, then a base-model prior honest miss

 The SQL specialist generated correct queries wrapped in prose ("Answer: SELECT …;
 Explanation: …"). The goal: strip the wrapper without losing execution accuracy. Three runs,
 each teaching the next:

```
2026-07-04 ref-free SimPO exec 0.860 → 0.080 COLLAPSE
2026-07-11 reference-anchored DPO exec 0.860 → 0.900 no collapse
2026-07-11 higher-pressure DPO exec 0.860 → 0.920 loss → 0.007
```

 Reference anchoring cured the collapse (the DPO sanity check — step-1 loss ≈ log 2 —
 confirmed it), and execution actually improved . But clean-SQL rate stayed
 0.000 across every pressure regime. The diagnosis, corrected in the open:
 the 108 SFT targets were already bare SELECT — the wrapper is a base-model prose
 prior a rank-4 adapter can't override. So it's a generation-strength problem
 (constrained decoding), not a preference-tuning one. A clean negative result: composed DPO
 is ruled out for output hygiene.

 Lesson. Record the binary provenance. The 2026-06-25 release build scored
 the same predictions at 0.000 where the debug build scored 0.860 — it composed multi-LoRA
 differently. Every eval now reproduces the frozen baseline with the exact binary before
 trusting a candidate number.

## Frontier parity at 4B — the distillation that worked measured win

 The clearest win in the repo: a Qwen3-4B distilled on multi-turn file-ops trajectories
 reached 100% on the BFCL hard gate, up from 58%. The honest cost is on the
 same page: out-of-domain breadth dropped 59.6% → 42.3% — real catastrophic
 forgetting. So it ships as a routed specialist, never a general planner. A number
 without its regression would have been marketing.

## The eval that was broken honest miss

 Early on, a fine-tune looked like it beat its baseline — until the baseline turned out to be
 non-reproducible and the v1 fixture set was measuring the wrong thing. The whole factory's
 eval discipline came out of that miss: frozen baselines stamped before training, exact-match
 vs execution-accuracy treated as different metrics, contamination checks, slice metrics, and
 gates that exit non-zero on regression. Bad evals create fake progress; that's the most
 expensive bug in the project.

## The Mac runtime — MLX-Swift, OpenAI-compatible, on one machine shipped

 The factory is a native Swift/MLX CLI — 100+ subcommands, one binary. It serves an
 OpenAI- and Ollama-compatible endpoint on one socket (Continue.dev and any OpenAI client
 plug in unchanged), runs an agent loop with FSM-constrained JSON, and packages to MLX,
 safetensors, and CoreML. Building it surfaced a real toolchain wall: MLX's Metal shaders
 need the full Xcode Metal Toolchain, not CommandLineTools — a scoped DEVELOPER_DIR
 override builds it without a global system change.

## Learning from y = mx + b up shipped

 Running alongside the build is a ground-up curriculum: ten modules from "a model is a
 parameterized function" to a self-improving factory, each with a toy exercise, a repo anchor,
 and a mastery gate. A coverage map ties every subsystem in the codebase to a learning anchor
 so nothing is a black box. The point isn't to finish a course — it's to know when a
 post-training result is real. Read it as the [learning log](https://posttrainllm.com/docs/learn/curriculum/).

 Earlier chapters — the browser / WebGPU learning track.
 Where it started: a GPT-2-shaped model from scratch in Python → C++/WASM → WebGPU, and the
 speed work that followed. The interesting part is which optimizations worked, which
 didn't, and why .

## Memory64 — breaking the 4 GB tab ceiling measured win

 Before this work, the browser playground couldn't allocate a model bigger than
 ~250M parameters in fp32. V8 caps each tab's WebAssembly heap near 4 GB using
 32-bit pointers; weights + AdamW optimizer state (≈12 bytes per parameter) hits
 that wall at exactly that size.

 WebAssembly's -sMEMORY64=1 + -sWASM_BIGINT flags switch
 the module to 64-bit pointers, lifting the cap into the tens of GB on Chromium
 133+ and Firefox 134+. The build script
 ( wasm/build_wasm64.sh ) produces a separate
 posttrainllm64.{js,wasm} module — same C++ source, just compiled with the
 new flags. Runtime feature-detection picks the right module.

 Measured. Allocated a 473M-parameter model end-to-end:

```
handle: 80312
params: 473,244,160
alloc time: 3,703 ms
1 train step: loss 5.78 in 82.2 s (initial loss for random init, sane)
freed cleanly
```

 The same allocation hard-OOMs the 32-bit module. The Behemoth preset in
 the playground deliberately surfaces this — pick it and the "Memory64 ✓" capability
 pill lights up, and a pre-flight check blocks the run on browsers that don't
 support it (telling you which browsers do).

 Lesson learned: the Memory64 descriptor spelling changed
 mid-flight in the WebAssembly proposal. Newer Chromium uses
 address: "i64" ; older Chromium (still bundled with Playwright as of
 late 2026) uses index: "i64" . The loader probes both. Without that
 fallback, browsers that did support the feature would silently load the
 32-bit module.

## Matmul kernel sweep — what worked, what didn't measured win

 Most of training time is matmul. So most of the speed work was matmul. The bench
 button on the playground runs a side-by-side sweep across kernel variants at
 realistic sizes (256³ → 2048³, inputs uploaded outside the timed loop so we measure
 dispatch cost, not packing). The data anchors every speed claim here.

### What worked, in order

 1. Workgroup-shared tiling (Goto/VandeGeijn 16×16): the canonical
 first optimization. Load a 16×16 tile of A and B into shared memory cooperatively,
 then do 16 multiply-adds from shared. Cuts global reads by ~16×.

 2. Thread-level register blocking (4×4): each thread holds a 4×4
 output block in registers. Outer-product structure means each shared-memory load
 gets reused 4× across the register accumulator. This is where matmul stops being
 bandwidth-bound and starts being compute-bound.

**matmul size**

**naive ms**

**tiled ms**

**blocked4 ms**

**vs naive**

 256³ 0.87 0.72 0.45 1.93×

 512³ 1.74 0.86 0.64 2.72×

 1024³ 6.43 2.85 1.80 3.58×

 2048³ 47.24 17.23 9.12 5.18×

 Blocked4 was wired into train.wgsl as a drop-in replacement (same
 bind-group layout as the naive kernel). The end-to-end parity test confirmed it
 produces equivalent training:

```
historical result — adapter identity and raw timings not retained
preset d_model speedup (WebGPU vs WASM SIMD mt) loss drift
Small 96 2.6× 1.1%
Medium 128 6.8× 1.4%
Large 192 9.3× 1.9%
XL 256 12.1× 2.5%
 (requires paired hardware reproduction)
```

 This curve is preserved because it shaped the optimization work, but it is
 not a current verified performance claim. The retained end-to-end runner
 checks loss parity; it does not emit the raw timings or reject software
 adapters. The canonical hardware protocol is in
 [the performance notes](https://posttrainllm.com/docs/performance/performance/).

### What didn't work

 f16-packed storage — store weights as two f16 per u32 via
 pack2x16float , halve global bandwidth. Standalone benchmark: 1.7×
 faster than naive WebGPU at 2048³. Sounded great. But when compared
 against the right baseline (the already-tiled kernel), the combined
 tiled+f16 ran slower than plain tiled at 2048³: 17.78 ms vs
 16.90 ms. Once tiling has amortized global reads, the kernel is compute-bound
 on shared-memory ops — halving global bandwidth has nowhere left to help.
 The 1.7× win was real but not additive — same underlying mechanism as
 tiling, captured worse.

 8×8 register block — hypothesis was that scaling from 4×4 to
 8×8 (with a 128×128 workgroup output tile) would 4× the arithmetic intensity
 per shared-mem load. Lost at every benchmarked size:

```
size 1024: blocked4 1.78 ms vs blocked8 1.96 ms (0.91×)
size 2048: blocked4 10.15 ms vs blocked8 11.52 ms (0.88×)
```

 Most likely cause: 64 floats per thread for the accumulator exceeds the
 per-thread register budget on Apple GPUs, forcing register spill to local
 memory. Lower workgroup occupancy (16 KB shared per workgroup vs 4 KB) compounds
 it. Kept in the codebase as a documented negative result.

 vec4 global loads — broke once, then root-caused.
 Same blocked4 algorithm but issuing 128-bit memory transactions for A and B.
 Standalone bench at 2048³: 1.37× faster than scalar blocked4 . Best
 single-kernel measurement in the project. First integration attempt diverged
 loss to 88.67 vs WASM's 2.94 — 30× off. The standalone bench used square shapes
 with WebGPU's layout: "auto" (which inferred read-only-storage to
 match the WGSL var<storage, read> declaration); production
 uses an explicit pipeline layout declaring buffer: { type: "storage" }
 — read-write. WGSL access mode and bind-group-layout type disagreeing
 is undefined behaviour on Chromium/Apple — silently returns wrong data instead
 of erroring at validation. Fix was one line per binding: declare all
 six as var<storage, read_write> in train_vec4.wgsl
 (the kernel only reads from g0/g1; the decoration just has to match). Parity
 test now passes at 1.6% drift; vec4 is the default forward matmul.

 Lesson learned, three times over. "More aggressive" is not the
 same as "faster," and standalone benchmarks miss bugs that show up in real
 training. The end-to-end parity test ( tests/test_webgpu_train.mjs )
 is now the bar — it runs 50 training steps under WASM and 50 under WebGPU on
 the same seed, asserts loss drift is below 5%. Every kernel integration goes
 through this gate.

## Speed evolution — the cumulative picture mixed evidence

 WASM stages retain direct measurements. WebGPU rows retain the historical
 curve without raw timings or adapter identity. See the same chart, with
 longer evidence notes, on [/roadmap](https://posttrainllm.com/roadmap).

**Build**

**Step time**

**vs scalar WASM**

**Notes**

 scalar WASM baseline 1.0× single-threaded, no SIMD

 + WASM SIMD – 1.6× -msimd128

 + multi-thread (4 wrk) – 3.2× Web Workers + SharedArrayBuffer

 + WebGPU full stack — Small (d=96) — ~8.3× historical 2.6× · unqualified

 + WebGPU full stack — Medium (d=128) — ~22× historical 6.8× · unqualified

 + WebGPU full stack — Large (d=192) — ~30× historical 9.3× · unqualified

 + WebGPU full stack — XL (d=256) — ~39× historical 12.1× · unqualified

 The historical curve suggests that GPU work amortises better as
 d_model and ctx grow. That is a plausible mechanism,
 not proof of the retained 2.6×–12.1× numbers; they need a fresh paired run.

## The lever that actually shipped — Flash Attention 2 measured win

 This was originally the headline entry in the "what's next" list. It
 moved up over the course of two sessions.

 Forward — workgroup-cooperative tiling. One workgroup
 per (batch, head, Q-tile of 16 rows) ; K and V walked in
 blocks of 16; the online-softmax state ( m , l ,
 O ) stays in registers across K blocks. Per-thread private
 memory drops from array<f32, 1024> (the FA1-style
 fallback) to 1 + 1 + hd floats — ≤ 66 at hd=64 ,
 well inside the register file. Default attention path for every preset
 up to Behemoth.

 Backward — recomputes attention on the fly from
 q , k , and a saved log-sum-exp
 L = m + log(l) per Q row. Two new kernels
 ( attn_dscores_fa2 + attn_dv_fa2 ) replace the
 ones that previously read the cached attn matrix.

 The real memory win: with backward no longer reading
 the cached attn matrix, the forward dropped its second-pass writeback
 entirely. At Mega-class shapes (B=4, H=8, T=512) that's about 67 MB of
 global memory traffic per layer per step that now stays on-chip.

```
WASM SIMD 6.8 s loss 2.9385
WebGPU + FA2 fwd
 + FA2 back 0.7 s loss 2.8650 2.5% drift
```

 Lesson learned, definitively this time: the
 algorithm-in-JS-first habit. Both halves of FA2 (forward and
 backward) were pinned in a Node parity test against a naive
 reference before any WGSL got written. That made each
 shader "transcribe the proven algorithm" rather than "debug from a
 wall of NaN." Without the JS reference I'd have been chasing
 register-spill bugs that the math itself would never have caused.
 Worth keeping a separate Node-side parity test for every future
 kernel that does anything non-trivial.

## What's next — what genuinely remains projected

 Most of the speed work is done. Two real threads left:

- Pre-trained model gallery — Cloudflare R2-hosted; manifest-driven UI; lets visitors load and continue-train from real checkpoints instead of just the one shipped demo. Deferred until the speed work was fully shipped so the gallery's implicit "you can train these too" promise stays honest. That bar is met now.

- Native macOS app — MLX-Swift + SwiftUI, mirrors the playground but lifts the model-size ceiling into the 7B–30B range on Apple Silicon. Same .tinygpt file format both ways. The biggest single new project that could grow out of this one.

## The bug that wasn't in any kernel honest miss

 The most expensive bug in this project never made it into a WGSL file. The
 browser default learning rate was 3e-3 — ten times the Python
 reference's 3e-4 . Training plateaued at loss ~2.45 on real
 corpora and looked exactly like a modelling ceiling: smooth curve, slowly
 asymptoting, no obvious tell. I spent two days suspecting the GPU kernels
 before noticing the LR.

 Lesson: kernel parity tests caught every numerical
 deviation in the maths. Nothing was catching deviation in the
 config defaults . The Python reference is the oracle for the
 hyperparameters too — and the defaults need to be parity-checked the
 same way the gradients are. Fixed in
 browser/src/types.ts and
 browser/src/pages/index.astro . Full write-up in
 [docs/archive/lessons.md](https://github.com/PostTrainLLM/posttrainllm/blob/main/docs/archive/lessons.md).

 Same investigation, two more honest items:

 The default corpus was 863 bytes. The playground was
 initializing on an inline meta-explainer paragraph instead of real text.
 No amount of model scaling lowers loss below the dataset's intrinsic
 entropy. Default is now the full TinyShakespeare (~1.1 MB,
 /shakespeare.txt ) fetched on init.

 The Memory64 build's ABI layer was untested. I shipped
 the 64-bit WASM module claiming "trains 473M params in Node." It did —
 when called directly from a one-off script. But tests/bench_wasm.mjs
 loads the 32-bit module, so the 64-bit pthread+Memory64 path had
 never been exercised in Node. The browser was calling into a
 broken JS↔WASM bridge: _malloc returns Number, cwrap
 pointer args expect BigInt, the conversion throws. Reproducer:
 tests/test_wasm64_xl_node.mjs . The OOB the browser was
 hitting at d_model ≥ 256 was a downstream consequence, not a kernel
 bug. Tracked as task #66; the loader falls back to the 32-bit module
 for XL/Massive/Mega/Behemoth in-browser.

## Notes on pair-programming with AI

 Most of this work was done in conversation with Claude. A few things that stood out:

 The AI's first answer is often the most aggressive one.
 Initial proposals were "let's do 8×8 blocking, that's 4× the reuse." Bench said no.
 Same with "f16 stacks on top of tiled." Bench said no. Always bench.

 Negative results were the most useful part. Documenting why
 f16-on-top-of-tiled doesn't compound, why 8×8 lost to 4×4, why vec4 broke
 non-square — those are now in the roadmap as honest entries. The next person (or
 next AI) won't waste time trying them.

 End-to-end parity tests catch what kernel benches miss. Standalone
 WebGPU matmul benchmarks at 256³/512³ passed every test. Wire it into a real
 training step and the loss diverges 30×. The end-to-end test
 ( tests/test_webgpu_train.mjs ) became the bar — every integration runs
 through it before claiming victory.

 "How fast can it be?" is the wrong question. "What does it take to
 produce real text?" is the right one. The model needs val loss below 1.5
 for the prose to start looking like prose. That's a dataset+model-scale problem,
 not a kernel-speed one. Speed makes the runs feasible ; the data + step
 budget determines whether they actually produce something readable .