Skip to content
posttrainllm docs
Esc
↑↓navigate↵open⌘Jpreview
On this page

Session — 2026-06-08, the factory-validation night

Session — 2026-06-08, the factory-validation night

Date: 2026-06-07 evening → 2026-06-08 early morning Premise: N02 base finished. Test the factory by trying to ship a real FC specialist. Find every gap.

Tonight’s actual wins

# Shipped Significance
1 GQA + head_dim fix in attention path (PRD shipped) Qwen3 family (1024 hidden, 16 heads, 128 head_dim) loads + trains. The reshape bug was off-by-2× because we assumed head_dim = hidden_size/num_heads.
2 QK-Norm support (Qwen3 RMSNorm on Q/K) shipped Qwen3 base attention was mathematically wrong without this. Step-1 SFT loss dropped 12.8 → 0.68 (19× improvement) after wiring.
3 DoRA serialization fix (both writers) shipped Default posttrainllm sft uses DoRA. Both writers (LoraIO + LoraHF) only checked as? LoraLinear — every DoRA-trained adapter was silently empty (209 bytes). Now DoRA round-trips as LoRA (m vector deferred to v2).
4 Theme-completer trained from scratch in 38 seconds 842K-param Tiny preset, 5050 hex palette pairs, loss 5+ → 0.114. Path-1 from-scratch validated on a non-text task. Echoes input colors (memorization on tiny corpus) but produces valid hex completions.
5 Pace specialist arc end-to-end prep System prompt extracted, tool-tag GBNF grammar drafted, recipe at docs/recipes/pace-planner.md. Tomorrow’s flagship arc is ready to fire.

Honest failures + what they taught

Attempt Outcome Real lesson
N02 FC SFT (22M base, ctx=256, hermes-fc 1150-token avg) Empty LoRA (loss=0 from truncation) From-scratch bases need ctx ≥ 2048 for instruction-following SFT. N02 is dead for downstream SFT. Future runs must train at bigger context.
Qwen3-0.6B FC SFT (first attempt) Reshape crash Surfaced the head_dim override gap. Fixed.
Qwen3-0.6B FC SFT (second, post-GQA-fix) Oscillating loss 9-16, no convergence QK-Norm missing → mathematically wrong attention. Fixed.
Phi-3 FC SFT (multiple attempts, parallel) Thermal pressure (fan 100%) One training run at a time. Hard rule.

The four factory gaps surfaced and closed

  1. head_dim override in Qwen3 family ✅ shipped (factory-gqa-support.md)
  2. QK-Norm RMSNorm on Q/K (Qwen3) ✅ shipped (covered in same PRD)
  3. DoRA serialization in writers ✅ v1 shipped (factory-dora-serialization.md)
  4. LoRA application for TinyGPTModelHF ⬜ tomorrow (#250)

Without (4), we cannot inference-test the adapters we trained. Tomorrow’s first priority.

State of artifacts

Path Size What
~/.cache/posttrainllm/runs/n02-20260606-1128/huge-base-v1.tinygpt 253 MB N02 22M from-scratch, val 4.32. Useful for interp/B13. Not for SFT.
~/.cache/posttrainllm/runs/theme-v1/theme-v1.tinygpt ~3 MB Tiny theme-completer. Validates path-1.
~/.cache/posttrainllm/runs/qwen3-fc-v1/qwen3-fc-v1.lora 8.8 MB First real Qwen3 LoRA. Trained pre-QK-Norm fix → numerics suspect. Replace with v2.
~/.cache/posttrainllm/runs/qwen3-fc-v2/qwen3-fc-v2.lora in flight Real Qwen3 FC SFT with QK-Norm + --no-dora. Expected size ~8-15 MB.

Tomorrow’s plan

Priority Task Effort
1 LoRA support for TinyGPTModelHF (#250) — hf-load --lora + serve --lora 2-3 hrs
2 Verify qwen3-fc-v2 generates FC-shaped output for hermes-fc test prompts 30 min
3 Pace specialist arc fire: pace prompts → posttrainllm synthesize (LM Studio teacher) → posttrainllm sft Qwen3 → eval against Pace fixtures half day
4 DoRA format v2 (round-trip with m vector) 2-4 hrs
5 Theme-completer v2: scrape ColorHunt for 10K palettes, retrain at Small preset half day

Hard rules learned tonight

  1. One training run at a time. Thermal pressure is real and bypasses pmset warnings — fan-by-ear is the early signal.
  2. Use --no-dora until DoRA format v2 ships. Default SFT writes header-only files otherwise.
  3. From-scratch bases need ctx ≥ 2048 if downstream SFT is intended. N02 at ctx=256 was a recipe-validation experiment, not a deployable base.
  4. GPU contention from parallel training is harder to spot than expected. A “killed” SFT that completed silently kept running; the duplicate launch went to 2× memory pressure.

What changes about the strategy

Nothing fundamental — but factory completeness now means more than just shipping primitives. It means shipping primitives THAT ACTUALLY WORK end-to-end on a real modern HF base. Tonight closed three architectural gaps that would have blocked every specialist arc on Qwen3/Llama-3/Mistral.

The “factory is mostly done” claim from last session was true at the primitive level but had silent gaps at the integration level. Tonight surfaced and closed them.

Was this page helpful?