Session retrospective — 2026-05-31

Multi-hour session that took posttrainllm from "Wave 2.5 stalled" to "every CLI verified + first specialist trained end-to-end + ROI-ordered backlog." What shipped, what was learned, what to pick up next.

Session retrospective — 2026-05-31

Branch: main · Final commit: 813b4da

This doc captures the arcs from a single multi-hour session that moved a lot of pieces. Future-Claude / future-me reads this first on a fresh checkout to know where things are.

The arc

The session opened with Wave 2.5 (Metal kernels) stalled, no Wave 3 specialist trained, and a vague backlog. It closed with:

• A decision-grade audit of the kernel work → 4 of 5 items dropped/deferred-with-reason
• A landscape audit of the competition → repositioning
• 5 product-shaped Wave 2.6 items shipped end-to-end
• Every CLI in the binary smoke-tested green
• A modality × action capability matrix
• A first specialist trained on real data (and a major bug found
  • fixed along the way)
• A living ROI-ordered backlog as single source of truth

What shipped (28 commits this session)

Wave 2.5 — research closed it, mostly with DROP/DEFER

Two research agents audited the 5 kernel items. Outcome:

Decision docs: docs/research/wave_2_5_kernel_audit.md, docs/research/mac_decode_baseline_m5pro.md.

Wave 2.6 — product-shaped features shipped

Wave 4 — landscape research

Single research agent audited TML, Apple Foundation Models, code-agent competitors (Cursor / Continue / Cline / Aider), and the Indic landscape. Key outputs:

• TML’s “Interaction Models” framing (foreground+background) adopted as vocabulary in the roadmap
• Apple FM is the same architecture but walled — posttrainllm positions as open / hackable / multi-specialist / route-anywhere
• Continue.dev/Ollama compat adapter identified as highest-leverage product move → shipped same session
• desi-max correction: it’s a text-to-image LoRA, not a Hindi LLM. Replaced with Sarvam-Edge / AI4Bharat Airavata across the docs.

Doc: docs/research/wave_4_landscape.md.

Feature audit + capability matrix

Every CLI subcommand (30+) smoke-tested end-to-end on M5 Pro. Quantization (HQQ + GPTQ + prune + LASER), fine-tuning (SFT + DPO + distill + ES + tuned-lens + train-heads + train-extractor), PEFT variants (LoRA + DoRA + VeRA + RsLoRA + LoRA-FA + PISSA + LoftQ + AdaLoRA + LayerDrop), and all the inference / eval / data / cloud / screen surfaces — all green.

Plus the modality × action matrix that pins down what posttrainllm does NOT do (vision, audio, multi-modal text+image / text+audio — all research-grade and deferred).

Docs: docs/feature_audit_2026_05_31.md, docs/capability_matrix.md.

First specialist run + a real bug

Trained toolcall-v1 on SmolLM2-135M + hermes-function-calling-v1 (11k records, 2000 steps, RS-LoRA rank 16). End-to-end pipeline works. 135M @ 2000 steps picked up surface JSON format but not functional tool-calling — too small or too few steps for genuine generalization.

Along the way: found + fixed the A0 LoRA-save bug (f566023): SFT’s curated DoRA-on-by-default wasn’t being disabled by PEFT-variant flags (--rs-lora, --vera, etc.). So makeAdapterLinear() returned DoraLinear instances, but the writers only handled LoraLinear — every saved adapter was an empty entries:[] header-only file. The trainable param walker handled both classes, so the count was always right, hiding the bug. All previous SFT / DPO outputs across the repo were silently empty. Fix is one line per PEFT case.

Doc: docs/specialist_v1_findings.md.

Decision docs added this session

• docs/research/wave_2_5_kernel_audit.md — kernel BUILD/DEFER/DROP
• docs/research/wave_4_landscape.md — TML/Apple/agents/Indic
• docs/research/mac_decode_baseline_m5pro.md — bench numbers + cider verdict
• docs/research/indic_evals.md — MILU + IndicGenBench design
• docs/research/data/jitter_*.json — raw bench JSONs (4 files)
• docs/async_tool_dispatch.md — investigated + skipped with reason
• docs/continue_provider.md — Continue.dev quickstart
• docs/tool_call_extractor.md — mini-router scaffold design
• docs/specialist_v1_findings.md — toolcall-v1 result + next steps
• docs/feature_audit_2026_05_31.md — full CLI smoke
• docs/capability_matrix.md — modalities × actions
• docs/progress.md — Mac+Web dashboard (live)
• docs/backlog.md — ROI-ordered TODO (living)
• docs/data_inventory.md — dataset reference
• docs/session_2026_05_31.md — this doc

What was learned

Architectural

1. The Mac is already 10× under realtime targets at every model size we have (9.6M → 960M). cider’s W8A8 has marginal payoff until a 3B+ specialist exists. Don’t pre-optimize.

2. Apple owns the architecture we’re chasing — the same on-device-3B + Private-Cloud-Compute model with adapter-only customization. posttrainllm’s edge is open / multi-specialist / route-anywhere, NOT being a better Apple FM.

3. TML is cloud-only for interaction models. Our on-device version is genuine differentiation.

4. The router is a GPT trunk with a classifier head. Backlog item B2b is the bake-off vs pure-GPT-with-FSM-constraint to settle whether the architectural deviation is worth keeping.

Process

1. Parallel agents work when the briefs are tight and the worktree is sliced clean. 4 agents shipped this session (WebGPU SG / ScreenCapture / Indic evals / tool extractor) — all committed their own work. Sandboxed worktrees + clear file-ownership directives kept merge conflict to zero.

2. Research before building saves weeks. Wave 2.5 collapsed from “5 hard items” to “1 build” + “1 adopt-on-M5” + “3 drops/defers” with two days of agent research.

3. Validate save-and-reload paths in tests. The LoRA-save bug was latent for the entire history of the project because no test exercised the SFT → save → reload → infer roundtrip. Backlog C7 plugs this gap.

4. /tmp is reaped on macOS. Lost binary + 50 MB of pulled data mid-session. Builds and downloads should default to ~/.cache/posttrainllm/ and ~/.local/bin/. Backlog C8.

5. CLI-flag interactions matter. The DoRA-on-by-default vs. PEFT-variant flag silent override was a real footgun. Curated defaults should be explicit losers when any explicit choice is made. Fixed for the PEFT family; worth auditing other CLI defaults for similar shadowing.

About the project

The project is now best described as: a text + code + structured- output transformer toolchain with quantization, PEFT, spec-decoding, agent runtime, screen-text, and cloud-escalation — train-able from scratch on Apple Silicon and browser WebGPU, with the full Wave 2.6 product surface (Continue.dev compat, AX tree, SSE streaming, mini-router, cloud escalation) shipped.

Not multi-modal in the vision/audio sense yet; that’s research-grade work deferred behind specialist training.

What to do when the laptop is back

Pick up from docs/backlog.md. The top Tier A items today:

1. A1 Train the first specialist end-to-end (toolcall-v2 with more steps, OR pivot to bigger base SmolLM2-360M, OR switch to debugger specialist on GitHub data — see specialist_v1_findings.md for the ranked options C → D → A → B)
2. A2 Re-pull tier-1 datasets (/tmp was reaped, but the cache layer at ~/.cache/posttrainllm/datasets/ is persistent — most should be one re-pull away)
3. A3 Fetch GitHub issue→PR for a real OSS repo (debugger data)
4. A4 BFCL + τ-bench via extractor-data
5. A5 Indic eval datasets (MILU + IndicGenBench-XQuAD)
6. A6 Dataset inventory doc — DONE this session
7. A7 Real-data MILU baseline

Tier C polish items if blocked:

• C6 ChatML template: detect inline system: prefix and split
• C7 Save+reload XCTest for LoRA (plugs A0 regression-coverage)
• C8 Install-path discipline (no more /tmp)

Deferred-with-trigger items (Tier D) — don’t touch unless trigger fires. See docs/backlog.md for the trigger conditions.

Total ledger

• Commits this session: ~28
• Files added: ~20 docs, multiple Swift modules (Screen, EvalIndic, tool-router pipeline, Ollama handlers, Continue.dev doc, etc.)
• Bug fixed: A0 LoRA save (was silently empty since project start)
• CLIs verified: 30+ (full audit green)
• Specialists trained: 1 (toolcall-v1, pipeline-validation quality, not deployment-ready)
• Research dives: 2 (Wave 2.5 kernels, Wave 4 landscape)
• Modalities documented: 12 (text/code/structured/multilingual/ screen-AX/screen-image/vision/audio/multi-modal × shipped/partial/ deferred)

How this doc fits with the others

docs/session_2026_05_31.md ← you are here (retrospective)
    │
    ├── points at ──→ docs/progress.md       (live Mac+Web dashboard)
    ├── points at ──→ docs/backlog.md        (live ROI-ordered TODO)
    ├── points at ──→ docs/capability_matrix.md  (modalities × actions)
    ├── points at ──→ docs/data_inventory.md (dataset reference)
    ├── points at ──→ docs/specialist_v1_findings.md  (first run + next steps)
    ├── points at ──→ docs/feature_audit_2026_05_31.md  (CLI smoke)
    │
    └── decision docs:
        ├── docs/research/wave_2_5_kernel_audit.md
        ├── docs/research/wave_4_landscape.md
        ├── docs/research/mac_decode_baseline_m5pro.md
        ├── docs/research/indic_evals.md
        ├── docs/async_tool_dispatch.md
        └── docs/tool_call_extractor.md

progress.md and backlog.md are the two living docs — those get updated as work ships. The rest are point-in-time decision logs that should not be edited; supersede them with new dated docs if the analysis is revisited.