---
title: Session retrospective — 2026-05-31
description: Multi-hour session that took posttrainllm from "Wave 2.5 stalled" to "every CLI verified + first specialist trained end-to-end + ROI-ordered backlog." What shipped, what was learned, what to pick up next.
---

# Session retrospective — 2026-05-31

**Branch**: `main` · **Final commit**: `813b4da`

This doc captures the arcs from a single multi-hour session that
moved a lot of pieces. Future-Claude / future-me reads this first
on a fresh checkout to know where things are.

## The arc

The session opened with Wave 2.5 (Metal kernels) stalled, no Wave 3
specialist trained, and a vague backlog. It closed with:

- A decision-grade audit of the kernel work → 4 of 5 items
  dropped/deferred-with-reason
- A landscape audit of the competition → repositioning
- 5 product-shaped Wave 2.6 items shipped end-to-end
- Every CLI in the binary smoke-tested green
- A modality × action capability matrix
- A first specialist trained on real data (and a major bug found
  + fixed along the way)
- A living ROI-ordered backlog as single source of truth

## What shipped (28 commits this session)

### Wave 2.5 — research closed it, mostly with DROP/DEFER

Two research agents audited the 5 kernel items. Outcome:

| Item | Verdict | Why |
|---|---|---|
| Flash Attention Metal kernel | **DROP** | MLXFast SDPA already fused; our 9 call sites hit the fast path |
| Int4 packed matmul Metal kernel | **DROP** | MLX `quantized_matmul` is already hand-tuned |
| Int8 W8A8 / cider | **DEFER** | Mac is already 10× under realtime targets at current model sizes; revisit at 3B+ |
| ANE + GPU heterogeneous routing | **DEFER** | Apple's Stateful Models API is the gating event |
| WebGPU subgroup matmul | **BUILD** | Shipped; gate failed (1415% mean_rel); fallback engaged unchanged |

Decision docs: `docs/research/wave_2_5_kernel_audit.md`,
`docs/research/mac_decode_baseline_m5pro.md`.

### Wave 2.6 — product-shaped features shipped

| Item | Commit | Notes |
|---|---|---|
| Cloud API client (Anthropic + OpenAI) | `ef0e5e3` | curl shell-out, env-var auth |
| SSE streaming on serve | `e754d6c` | OpenAI-compatible chunks |
| SSE cancellation on disconnect | `c11265b` | SIGPIPE-safe |
| CloudEscalate → AgentLoop | `4119216` | synthetic `escalate` tool wired into agent runtime |
| Ollama-compat provider adapter | `79d6f40`, `9b76089` | Continue.dev / Cline / Aider drop-in |
| ScreenCaptureKit + AX tree | `08a7689` | AX works end-to-end; SCKit capture has CGS quirk from CLI |
| Tool-call extractor (mini-router) | `b5bbdd9` | full scaffold: data → train → infer → agent flag |
| MILU + IndicGenBench eval CLI | `3385a76` | wired with smoke fixtures; real-data baseline pending |
| Continue.dev walkthrough doc | `79d6f40` | `docs/integrations/continue_provider.md` |

### Wave 4 — landscape research

Single research agent audited TML, Apple Foundation Models,
code-agent competitors (Cursor / Continue / Cline / Aider), and the
Indic landscape. Key outputs:

- **TML's "Interaction Models" framing** (foreground+background)
  adopted as vocabulary in the roadmap
- **Apple FM is the same architecture but walled** — posttrainllm
  positions as open / hackable / multi-specialist / route-anywhere
- **Continue.dev/Ollama compat adapter** identified as
  highest-leverage product move → shipped same session
- **desi-max correction**: it's a text-to-image LoRA, not a Hindi
  LLM. Replaced with Sarvam-Edge / AI4Bharat Airavata across the
  docs.

Doc: `docs/research/wave_4_landscape.md`.

### Feature audit + capability matrix

Every CLI subcommand (30+) smoke-tested end-to-end on M5 Pro.
Quantization (HQQ + GPTQ + prune + LASER), fine-tuning (SFT + DPO +
distill + ES + tuned-lens + train-heads + train-extractor), PEFT
variants (LoRA + DoRA + VeRA + RsLoRA + LoRA-FA + PISSA + LoftQ +
AdaLoRA + LayerDrop), and all the inference / eval / data / cloud /
screen surfaces — all green.

Plus the modality × action matrix that pins down what posttrainllm does
NOT do (vision, audio, multi-modal text+image / text+audio — all
research-grade and deferred).

Docs: `docs/audits/feature_audit_2026_05_31.md`, `docs/capability_matrix.md`.

### First specialist run + a real bug

Trained `toolcall-v1` on SmolLM2-135M + hermes-function-calling-v1
(11k records, 2000 steps, RS-LoRA rank 16). End-to-end pipeline
works. 135M @ 2000 steps picked up surface JSON format but not
functional tool-calling — too small or too few steps for genuine
generalization.

**Along the way: found + fixed the A0 LoRA-save bug** (`f566023`):
SFT's curated DoRA-on-by-default wasn't being disabled by
PEFT-variant flags (`--rs-lora`, `--vera`, etc.). So
`makeAdapterLinear()` returned `DoraLinear` instances, but the
writers only handled `LoraLinear` — every saved adapter was an
empty `entries:[]` header-only file. The trainable param walker
handled both classes, so the count was always right, hiding the
bug. **All previous SFT / DPO outputs across the repo were silently
empty.** Fix is one line per PEFT case.

Doc: `docs/sessions/specialist_v1_findings.md`.

### Decision docs added this session

- `docs/research/wave_2_5_kernel_audit.md` — kernel BUILD/DEFER/DROP
- `docs/research/wave_4_landscape.md` — TML/Apple/agents/Indic
- `docs/research/mac_decode_baseline_m5pro.md` — bench numbers + cider verdict
- `docs/research/indic_evals.md` — MILU + IndicGenBench design
- `docs/research/data/jitter_*.json` — raw bench JSONs (4 files)
- `docs/async_tool_dispatch.md` — investigated + skipped with reason
- `docs/integrations/continue_provider.md` — Continue.dev quickstart
- `docs/tool_call_extractor.md` — mini-router scaffold design
- `docs/sessions/specialist_v1_findings.md` — toolcall-v1 result + next steps
- `docs/audits/feature_audit_2026_05_31.md` — full CLI smoke
- `docs/capability_matrix.md` — modalities × actions
- `docs/progress.md` — Mac+Web dashboard (live)
- `docs/backlog.md` — ROI-ordered TODO (living)
- `docs/data_inventory.md` — dataset reference
- `docs/sessions/session_2026_05_31.md` — this doc

## What was learned

### Architectural

1. **The Mac is already 10× under realtime targets** at every model
   size we have (9.6M → 960M). cider's W8A8 has marginal payoff
   until a 3B+ specialist exists. Don't pre-optimize.

2. **Apple owns the architecture we're chasing** — the same
   on-device-3B + Private-Cloud-Compute model with adapter-only
   customization. posttrainllm's edge is open / multi-specialist /
   route-anywhere, NOT being a better Apple FM.

3. **TML is cloud-only** for interaction models. Our on-device
   version is genuine differentiation.

4. **The router is a GPT trunk with a classifier head.** Backlog
   item B2b is the bake-off vs pure-GPT-with-FSM-constraint to
   settle whether the architectural deviation is worth keeping.

### Process

1. **Parallel agents work when the briefs are tight and the worktree
   is sliced clean.** 4 agents shipped this session (WebGPU SG /
   ScreenCapture / Indic evals / tool extractor) — all committed
   their own work. Sandboxed worktrees + clear file-ownership
   directives kept merge conflict to zero.

2. **Research before building saves weeks.** Wave 2.5 collapsed
   from "5 hard items" to "1 build" + "1 adopt-on-M5" + "3
   drops/defers" with two days of agent research.

3. **Validate save-and-reload paths in tests.** The LoRA-save bug
   was latent for the entire history of the project because no test
   exercised the SFT → save → reload → infer roundtrip. Backlog
   C7 plugs this gap.

4. **`/tmp` is reaped on macOS.** Lost binary + 50 MB of pulled
   data mid-session. Builds and downloads should default to
   `~/.cache/posttrainllm/` and `~/.local/bin/`. Backlog C8.

5. **CLI-flag interactions matter.** The DoRA-on-by-default vs.
   PEFT-variant flag silent override was a real footgun. Curated
   defaults should be explicit losers when any explicit choice is
   made. Fixed for the PEFT family; worth auditing other CLI
   defaults for similar shadowing.

### About the project

The project is now best described as: **a text + code + structured-
output transformer toolchain with quantization, PEFT, spec-decoding,
agent runtime, screen-text, and cloud-escalation — train-able from
scratch on Apple Silicon and browser WebGPU, with the full Wave 2.6
product surface (Continue.dev compat, AX tree, SSE streaming,
mini-router, cloud escalation) shipped**.

Not multi-modal in the vision/audio sense yet; that's research-grade
work deferred behind specialist training.

## What to do when the laptop is back

Pick up from `docs/backlog.md`. The top Tier A items today:

1. **A1** Train the first specialist end-to-end (toolcall-v2 with
   more steps, OR pivot to bigger base SmolLM2-360M, OR switch to
   debugger specialist on GitHub data — see `specialist_v1_findings.md`
   for the ranked options C → D → A → B)
2. **A2** Re-pull tier-1 datasets (`/tmp` was reaped, but the cache
   layer at `~/.cache/posttrainllm/datasets/` is persistent — most should
   be one re-pull away)
3. **A3** Fetch GitHub issue→PR for a real OSS repo (debugger data)
4. **A4** BFCL + τ-bench via `extractor-data`
5. **A5** Indic eval datasets (MILU + IndicGenBench-XQuAD)
6. **A6** Dataset inventory doc — DONE this session
7. **A7** Real-data MILU baseline

Tier C polish items if blocked:
- C6 ChatML template: detect inline `system:` prefix and split
- C7 Save+reload XCTest for LoRA (plugs A0 regression-coverage)
- C8 Install-path discipline (no more /tmp)

Deferred-with-trigger items (Tier D) — don't touch unless trigger
fires. See `docs/backlog.md` for the trigger conditions.

## Total ledger

- **Commits this session**: ~28
- **Files added**: ~20 docs, multiple Swift modules (Screen, EvalIndic,
  tool-router pipeline, Ollama handlers, Continue.dev doc, etc.)
- **Bug fixed**: A0 LoRA save (was silently empty since project
  start)
- **CLIs verified**: 30+ (full audit green)
- **Specialists trained**: 1 (toolcall-v1, pipeline-validation
  quality, not deployment-ready)
- **Research dives**: 2 (Wave 2.5 kernels, Wave 4 landscape)
- **Modalities documented**: 12 (text/code/structured/multilingual/
  screen-AX/screen-image/vision/audio/multi-modal × shipped/partial/
  deferred)

## How this doc fits with the others

```
docs/sessions/session_2026_05_31.md ← you are here (retrospective)
    │
    ├── points at ──→ docs/progress.md       (live Mac+Web dashboard)
    ├── points at ──→ docs/backlog.md        (live ROI-ordered TODO)
    ├── points at ──→ docs/capability_matrix.md  (modalities × actions)
    ├── points at ──→ docs/data_inventory.md (dataset reference)
    ├── points at ──→ docs/sessions/specialist_v1_findings.md  (first run + next steps)
    ├── points at ──→ docs/audits/feature_audit_2026_05_31.md  (CLI smoke)
    │
    └── decision docs:
        ├── docs/research/wave_2_5_kernel_audit.md
        ├── docs/research/wave_4_landscape.md
        ├── docs/research/mac_decode_baseline_m5pro.md
        ├── docs/research/indic_evals.md
        ├── docs/async_tool_dispatch.md
        └── docs/tool_call_extractor.md
```

`progress.md` and `backlog.md` are the two **living** docs — those
get updated as work ships. The rest are point-in-time decision logs
that should not be edited; supersede them with new dated docs if
the analysis is revisited.