Devlog — how the lab got built

The site you're reading — a dark research-lab redesign shipped

The site had drifted into "clean editorial SaaS" — cohesive but AI-looking, and split across a bright web and a dark app. This pass unified everything into one instrument: a single dark design system (crafted deep-ink palette with a faint grain, teal as the live signal, Bricolage Grotesque for display), the loss-curve logo as the universal mark, a manifesto, machine-transparent claims, and structured-data SEO.

The factory loop closed — and the first ship-decision was "retry" measured

The north star stopped being "another surface" and became one canonical run: target → data → post-train → eval → package → report, ending in a documented ship/reject/retry decision. On 2026-07-04 it executed end-to-end for the first time on a real candidate — the frozen qwen06-sql-hygiene-dpo target — and produced a schema-valid run folder that factory-run validate accepts.

The DPO saga — a policy collapse, then a base-model prior honest miss

The SQL specialist generated correct queries wrapped in prose ("Answer: SELECT …; Explanation: …"). The goal: strip the wrapper without losing execution accuracy. Three runs, each teaching the next:

2026-07-04  ref-free SimPO          exec 0.860 → 0.080   COLLAPSE
2026-07-11  reference-anchored DPO   exec 0.860 → 0.900   no collapse
2026-07-11  higher-pressure DPO      exec 0.860 → 0.920   loss → 0.007

Reference anchoring cured the collapse (the DPO sanity check — step-1 loss ≈ log 2 — confirmed it), and execution actually improved. But clean-SQL rate stayed 0.000 across every pressure regime. The diagnosis, corrected in the open: the 108 SFT targets were already bare SELECT — the wrapper is a base-model prose prior a rank-4 adapter can't override. So it's a generation-strength problem (constrained decoding), not a preference-tuning one. A clean negative result: composed DPO is ruled out for output hygiene.

Frontier parity at 4B — the distillation that worked measured win

The clearest win in the repo: a Qwen3-4B distilled on multi-turn file-ops trajectories reached 100% on the BFCL hard gate, up from 58%. The honest cost is on the same page: out-of-domain breadth dropped 59.6% → 42.3% — real catastrophic forgetting. So it ships as a routed specialist, never a general planner. A number without its regression would have been marketing.

The eval that was broken honest miss

Early on, a fine-tune looked like it beat its baseline — until the baseline turned out to be non-reproducible and the v1 fixture set was measuring the wrong thing. The whole factory's eval discipline came out of that miss: frozen baselines stamped before training, exact-match vs execution-accuracy treated as different metrics, contamination checks, slice metrics, and gates that exit non-zero on regression. Bad evals create fake progress; that's the most expensive bug in the project.

The Mac runtime — MLX-Swift, OpenAI-compatible, on one machine shipped

The factory is a native Swift/MLX CLI — 100+ subcommands, one binary. It serves an OpenAI- and Ollama-compatible endpoint on one socket (Continue.dev and any OpenAI client plug in unchanged), runs an agent loop with FSM-constrained JSON, and packages to MLX, safetensors, and CoreML. Building it surfaced a real toolchain wall: MLX's Metal shaders need the full Xcode Metal Toolchain, not CommandLineTools — a scoped DEVELOPER_DIR override builds it without a global system change.

Learning from y = mx + b up shipped

Running alongside the build is a ground-up curriculum: ten modules from "a model is a parameterized function" to a self-improving factory, each with a toy exercise, a repo anchor, and a mastery gate. A coverage map ties every subsystem in the codebase to a learning anchor so nothing is a black box. The point isn't to finish a course — it's to know when a post-training result is real. Read it as the learning log.

Earlier chapters — the browser / WebGPU learning track. Where it started: a GPT-2-shaped model from scratch in Python → C++/WASM → WebGPU, and the speed work that followed. The interesting part is which optimizations worked, which didn't, and why.

Memory64 — breaking the 4 GB tab ceiling measured win

Before this work, the browser playground couldn't allocate a model bigger than ~250M parameters in fp32. V8 caps each tab's WebAssembly heap near 4 GB using 32-bit pointers; weights + AdamW optimizer state (≈12 bytes per parameter) hits that wall at exactly that size.

WebAssembly's -sMEMORY64=1 + -sWASM_BIGINT flags switch the module to 64-bit pointers, lifting the cap into the tens of GB on Chromium 133+ and Firefox 134+. The build script (wasm/build_wasm64.sh) produces a separate posttrainllm64.{js,wasm} module — same C++ source, just compiled with the new flags. Runtime feature-detection picks the right module.

handle: 80312
params: 473,244,160
alloc time: 3,703 ms
1 train step:  loss 5.78  in 82.2 s  (initial loss for random init, sane)
freed cleanly

The same allocation hard-OOMs the 32-bit module. The Behemoth preset in the playground deliberately surfaces this — pick it and the "Memory64 ✓" capability pill lights up, and a pre-flight check blocks the run on browsers that don't support it (telling you which browsers do).

Matmul kernel sweep — what worked, what didn't measured win

Most of training time is matmul. So most of the speed work was matmul. The bench button on the playground runs a side-by-side sweep across kernel variants at realistic sizes (256³ → 2048³, inputs uploaded outside the timed loop so we measure dispatch cost, not packing). The data anchors every speed claim here.

What worked, in order

1. Workgroup-shared tiling (Goto/VandeGeijn 16×16): the canonical first optimization. Load a 16×16 tile of A and B into shared memory cooperatively, then do 16 multiply-adds from shared. Cuts global reads by ~16×.

2. Thread-level register blocking (4×4): each thread holds a 4×4 output block in registers. Outer-product structure means each shared-memory load gets reused 4× across the register accumulator. This is where matmul stops being bandwidth-bound and starts being compute-bound.

Blocked4 was wired into train.wgsl as a drop-in replacement (same bind-group layout as the naive kernel). The end-to-end parity test confirmed it produces equivalent training:

preset   d_model   speedup (WebGPU vs WASM SIMD mt)   loss drift
Small      96            2.6×                              1.1%
Medium    128            6.8×                              1.4%
Large     192            9.3×                              1.9%
XL        256           12.1×                              2.5%
                                  (drift = pure float-reorder noise)

What didn't work

size 1024:  blocked4 1.78 ms  vs  blocked8 1.96 ms  (0.91×)
size 2048:  blocked4 10.15 ms vs blocked8 11.52 ms (0.88×)

Speed evolution — the cumulative picture measured

Each bar is anchored to a measurement on this machine, not an extrapolation. (See the same chart, with longer captions, on /roadmap.)

The speedup is a curve, not a single number. Small preset 2.6×, XL 12.1×, trending upward because the blocked-4×4 matmul kernel's win scales with matmul size — GPU work amortises better as d_model and ctx grow. Loss drift stays under 2.5% across the curve, pure float-reorder noise from different accumulation order in the GPU kernels.

The lever that actually shipped — Flash Attention 2 measured win

This was originally the headline entry in the "what's next" list. It moved up over the course of two sessions.

WASM SIMD          6.8 s   loss 2.9385
WebGPU + FA2 fwd
       + FA2 back   0.7 s   loss 2.8650   2.5% drift

What's next — what genuinely remains projected

Most of the speed work is done. Two real threads left:

• Pre-trained model gallery — Cloudflare R2-hosted; manifest-driven UI; lets visitors load and continue-train from real checkpoints instead of just the one shipped demo. Deferred until the speed work was fully shipped so the gallery's implicit "you can train these too" promise stays honest. That bar is met now.
• Native macOS app — MLX-Swift + SwiftUI, mirrors the playground but lifts the model-size ceiling into the 7B–30B range on Apple Silicon. Same .tinygpt file format both ways. The biggest single new project that could grow out of this one.

The bug that wasn't in any kernel honest miss

The most expensive bug in this project never made it into a WGSL file. The browser default learning rate was 3e-3 — ten times the Python reference's 3e-4. Training plateaued at loss ~2.45 on real corpora and looked exactly like a modelling ceiling: smooth curve, slowly asymptoting, no obvious tell. I spent two days suspecting the GPU kernels before noticing the LR.

Same investigation, two more honest items:

Notes on pair-programming with AI

Most of this work was done in conversation with Claude. A few things that stood out:

The AI's first answer is often the most aggressive one. Initial proposals were "let's do 8×8 blocking, that's 4× the reuse." Bench said no. Same with "f16 stacks on top of tiled." Bench said no. Always bench.

Negative results were the most useful part. Documenting why f16-on-top-of-tiled doesn't compound, why 8×8 lost to 4×4, why vec4 broke non-square — those are now in the roadmap as honest entries. The next person (or next AI) won't waste time trying them.

End-to-end parity tests catch what kernel benches miss. Standalone WebGPU matmul benchmarks at 256³/512³ passed every test. Wire it into a real training step and the loss diverges 30×. The end-to-end test (tests/test_webgpu_train.mjs) became the bar — every integration runs through it before claiming victory.

"How fast can it be?" is the wrong question. "What does it take to produce real text?" is the right one. The model needs val loss below 1.5 for the prose to start looking like prose. That's a dataset+model-scale problem, not a kernel-speed one. Speed makes the runs feasible; the data + step budget determines whether they actually produce something readable.