Audit 2026 — what we tried, what worked, what we flagged

Audit 2026 — what we tried, what worked, what we flagged

After ~70 techniques shipped across the project, this doc is the honest reckoning. Each entry: what it claimed, what we measured, and verdict (🟢 KEEP / 🟡 FLAG / 🔴 DELETE).

2026-07-05 exactness note: this remains the prose audit. The structured row-level inventory now lives in docs/techniques/audit-inventory.md: 83 tracked audit rows, including intentional duplicate mentions such as YOCO and BPE-dropout. Use that inventory for classification; use docs/attempt-ledger.md only for concrete run attempts.

Conviction bar for DELETE: only items I’m genuinely convinced are useless to the project under ANY scenario. Items where the test was narrow (e.g., tested at 22M but might work at 1.5B base) or the implementation was incomplete (e.g., GaLore’s optimizer state) or that need future infrastructure (e.g., int8 matmul kernels) get FLAGGED, not deleted. The flag records: “tested at config X, didn’t see win, not in default recipe, here’s when it’d help.”

The convict-with-deletion list is short. Most items become FLAG.

The audit is informed by the project’s north star: on-device agent models on Apple Silicon + Chrome. Techniques are evaluated against “does this help build/train/run/serve an agent-shaped model on a Mac or in a browser?” — not “does this exist in the literature?”

After this audit lands, the codebase shrinks from ~28K lines to ~16-18K lines of Swift, the CLI from ~40 flags to ~12, and the “choose-your-own-adventure” surface area collapses into ONE curated recipe per capability.

TL;DR — revised under the conviction bar

Most items get FLAGGED with: (a) what we tested, (b) what we saw, (c) when this would actually help. The code stays in the repo with a clear note in the source. Users opting into specialist training can revisit any item.

🟢 KEEP — these are the curated defaults

Training

Tokenization

Alignment

PEFT (fine-tuning)

Inference / sampling

Eval / bench

Quality

Infrastructure

Web playground

🟡 EXPERIMENTAL — move to experimental/, keep accessible

Interesting, educational, or might-be-useful-later. Stays in the codebase under --experimental-* flags or experimental/ subdirs.

🟡 FLAG — kept in code, not in default recipe

Each FLAGGED item stays in the repo with an // AUDIT FLAG: tested at X, didn't see win, useful when Y comment at the entry point. The CLI flag remains available under --experimental-* or stays untouched if removing it would break compatibility. The point: no work thrown away, honest record of what was tested and what wasn’t.

Optimizer alternatives — FLAG all 4

Architecture variants — FLAG all 5

Stability tricks — FLAG all 3

Training-time exotic — FLAG both

PEFT variants — FLAG all 6 (was DELETE)

Quantization — FLAG all 4

Pruning — FLAG unstructured + structured-head, KEEP layer pruning

Speculative decoding heads — FLAG both

🔴 DELETE — convinced useless (with this bar: maybe nothing)

Under the strict conviction bar, I’m not confident anything in the audit is convict-with-deletion useless. Every technique has a scenario.

The honest move: DELETE list = empty for now. The codebase stays roughly its current size. The audit’s contribution is the FLAGS — clear notes per technique about what we tested, what we saw, when it would help.

If you want a smaller binary or cleaner CLI, the right move is CLI curation (hide flags under --experimental-*), not source-code deletion. Source-code is cheap to keep; the maintenance cost is in keeping it tested + documented, which the FLAG annotations address.

What the CLI looks like AFTER the cuts

Current state:

posttrainllm train --preset huge --tokenizer ... --dtype bfloat16 \
    --optimizer adamw --grad-checkpoint \
    --z-loss-weight 1e-4 --embedding-rmsnorm \
    --galore-rank 0 --bpe-dropout 0 --qat 0 \
    --moe-experts 1 --mtp-horizons 1 \
    --diff-attn --mod --yoco --alibi --sliding-window 0 \
    [+ 20 more flags]

After cleanup:

posttrainllm train <corpus>          # AdamW + bf16 + cosine + clip — recipe defaults
  --preset huge|mega|behemoth
  --tokenizer <hf-dir>
  --grad-checkpoint              # for mega+ models
  --resume <path>
  --save-every N

posttrainllm finetune <base> <data>  # DoRA + SFT — recipe defaults
  --rank R
  --lora-fa                      # halve params if you want

posttrainllm align <base> <prefs>    # SimPO — recipe default
  --loss-type dpo|simpo|orpo|kto # if you really want to pick

posttrainllm sample <model>          # KV cache + KIVI int8 + speculative — defaults
  --prompt "..."
  --tokens N

posttrainllm quantize <model>        # AWQ → int4 — recipe default
  --bits 4|8

# All other techniques: --experimental-X for the alternatives

Revised execution plan (conviction-bar version)

1. Draft this doc — done
2. Per-FLAGGED-technique inline annotation — add // AUDIT FLAG: block at each entry point in source, with what-we-tested / what-we-saw / when-it’d-help. ~1-2 days mechanical.
3. Default-CLI curation — posttrainllm train etc. = curated recipe with no flags needed; FLAGGED features move to --experimental-*. ~2 days.
4. Help text + landing page rewrite — preach the ONE curated recipe; “Advanced/experimental” gates the rest. ~2 days.
5. No source-code deletion this round — keep everything; the FLAG annotations are the documentation.

Estimated effort: 3-5 days focused. Bias toward honest documentation over aggressive deletion. Source-code is cheap; the cost of accidentally deleting something useful is high.

After this curation, the codebase is ready for the on-device agent model factory focus. The curated tools above are the recipe for the debugger / code-reviewer / SQL-writer / etc. specialists. FLAGGED tools remain available when a specialist’s training shows they help (e.g., VeRA for many-specialist factory, MoD when scatter_add lands).