The operating surface for the Mac-local specialist factory: current state, attempts, exactness audits, technique inventory, public artifacts, and the learning path.
Read these first when you need to know what is true now.
Proof surfaces for attempts, counts, schemas, and publishing gates.
285 visible docs from the rendered repo corpus.
posttrainllm agent <model> --tools tools.json is the product surface for the on-device agent SLM factory. The CLI loads a model (browser-trained .tinygpt, HuggingFace dir, or finetuned sp...
Why async tool dispatch in AgentLoop doesn't justify the work at current bottleneck shape. Logged so the decision doesn't get re-litigated.
This ledger records meaningful posttrainllm attempts as worked, failed, regressed, inconclusive, or not yet tried. It is the human-readable companion to run folders and factory reports.
After ~70 techniques shipped across the project, this doc is the honest reckoning. Each entry: what it claimed, what we measured, and
the existing in-process MLX inference path. This is the "smoke test" that proves the scaffold produces real numbers; it is not a publishable benchmark (n=5, no energy metrics, tiny model).*
"most powerful inference engine for modern transformers" claim; nothing about that claim is credible without a reproducible measurement frame.*
How a Mac-trained BPE checkpoint gets a row on the browser leaderboard without the leaderboard ever needing to know what "BPE" means.
Phase 4–5. Move training/inference into the browser after the Python reference is correct. Order: PyTorch reference → WASM CPU → WebGPU.
Exhaustive map of input modalities posttrainllm supports and what operations (train, distill, tune, quantize, etc.) are available for each. Honest status per cell — what's shipped, what's partial, what's only roadmapped.
Every architectural claim in posttrainllm's code + docs traces back to a primary source here. If a claim doesn't have a citation in this file, treat it as informed-opinion-not-evidence an...
Date: 2026-05-30 Branch: worktree agent-acdbc93fc70df3aed Target machine: M3 Max, 36 GB RAM, macOS 25.5 Build: xcodebuild -scheme posttrainllm -configuration Release (LLVM optimisation on)
Reliable JSON output is the most common production failure mode for small (1-3B) LLMs in agent loops. Even when the model "knows" how to produce JSON, its sampler picks a wrong byte ~1 in...
Status: implemented, measured, four items shipped. Numbers in this doc were recorded against the worktree binary built at the same SHA as the diff that introduced the items. Comparison ba...
Status: research + recommendations. No code shipped. Numbers are estimates unless explicitly called out as measured.
Two complementary data-pipeline regularisers for the Mac trainer:
Live snapshot of ~/.cache/posttrainllm/datasets/ (not checked into the repo). Last refreshed 2026-06-17. Sizes are directory totals; row counts are from decoded JSONLs where available, ot...
Reference doc for every dataset wired into posttrainllm — registry entries, what they're for, how to pull them, what the records look like after conversion, and known gotchas (gated datasets, parquet decode).
This is the meta-log: not the lessons learned, not the open questions, but the moments the direction actually changed. Across one long working session on posttrainllm, the project drifted...
The browser app in browser/ is a Vite static build — no server, no API. It loads a pre-compiled WASM module from browser/public/, so the deploy environment does not need Emscripten; it on...
posttrainllm train --seed <UInt64> now seeds both of posttrainllm's two randomness surfaces:
Distillation is the post-training technique where a SMALL model ("student") is trained to match a LARGER model's ("teacher's") output distribution on a corpus. The result: students that s...
This registry labels the major documentation surfaces so old roadmap material does not compete with the active factory path.
This page tracks whether posttrainllm documentation is actually world-class, not just large.
Started 2026-06-11 after the "miner stopping 30 min before the diamond" check. Closed 2026-06-12 with the full table below.
ES is a finite-difference approach to optimisation: at every step, we sample K random perturbations of the current parameters, evaluate each perturbed model on a shared batch, and update ...
This audit answers one question: are the posttrainllm docs now exact enough to be trusted as the operating system for the project?
This page records products, startups, papers, and blogs that changed the posttrainllm roadmap. It is a teardown ledger: what we learned, what we stole, what we rejected, and what still ne...
Implementation log for the second half of task #47. Forward landed last session (webgpu/attention_fa2.wgsl + tests/test_fa2_parity.mjs + docs/fa2_forward_notes.md); this is the runway for...
Implementation log for task #47 (FA2 in WGSL), forward-only. Backward will be a separate session — see "What's still needed" at the bottom.
Verification results absorbed into docs/PLAN.md §1. Kept here only as the audit's historical anchor.
A backlog of concrete features for the browser posttrainllm, drawn from the famous educational GPT projects worth learning from. Source repos are filtered to
End-to-end first specialist training on M5 Pro. What worked, what didn't, what was unblocked, and what to try next.
The browser playground ships v1 of the gallery: four bundled .bin checkpoints (Shakespeare, TinyStories, Python, Q&A chat) served from browser/public/gallery/. This doc lays out the next ...
This page documents the five posttrainllm train flags introduced in the Tier-2 stability batch:
posttrainllm fetch-github pulls structured training data from GitHub for the code-specialist agent track (debugger, reviewer, commit-message generator). It is the GitHub-side counterpart ...
This doc captures the measurements from wiring custom gradient checkpointing into the Mac training path (Tier 1.7 of the single-machine roadmap). The numbers below were collected on an M5...
Status: PRD (pre-experiment). Owner: Sarthak. Date: 2026-06-12.
posttrainllm download-dataset and posttrainllm list-datasets give the on-device agent-factory direct access to HuggingFace's 100k+ dataset hub. This document covers how to use them, how t...
This audit tracks how much of posttrainllm's historical work is normalized into the structured attempt ledger, classified as technique inventory, or intentionally left as narrative/refere...
This is the external learning track for posttrainllm. Use it after the repo-local archive/learning_roadmap.md: CS336 is the spine, and company docs/blogs are the applied case studies.
Two interpretability surfaces ship in the browser playground:
Date: 2026-05-30 Status: implemented (worktree), unmerged Target HEAD: 645c2f4 (main)
The posttrainllm leaderboard at /leaderboard.html ranks small, browser-runnable language models on a curated set of benchmarks. Three properties are non-negotiable:
You can write code. You have never built or trained a neural network. This guide takes you, in order, through everything in this repository until none of it is a black box.
The owner's learning track is now explicitly ground-up. It still reinforces the posttrainllm factory, but it does not start with post-training jargon.
This tracker makes the owner learning pipeline measurable. The goal is to learn from the ground up while using posttrainllm as the lab.
This doc describes how posttrainllm plugs into EleutherAI's lm-evaluation-harness, the canonical eval framework behind the HuggingFace Open LLM Leaderboard. With the wiring in this commit...
Phase 3. LoRA adapts an already-trained, frozen base model by training small low-rank matrices inside selected linear layers.
The browser playground is the on-ramp. The Mac app is the depth: same architecture, same .tinygpt file format, ~20-30× the training throughput. This doc translates roadmap lever 20 into a...
This restructure split, merged, and archived a few docs. Use this table to find where the content of any old path now lives.
What fits on a 48 GB Mac for training is dominated by four memory costs: weights, optimizer state, gradients, and per-step activations. This guide explains each in concrete numbers and th...
Phase 1–2. Build a tiny GPT-style causal language model. First goal is
MoE replaces a transformer block's single dense MLP with N "expert" MLPs plus a learned router that picks the top-K experts for each token. The result: a model with ~N× the parameter capa...
Standard language-model training predicts ONE token ahead per position. Multi-Token Prediction (MTP) predicts H tokens ahead per position simultaneously, using H output heads that share t...
This is the active queue. It intentionally ignores most historical PRDs.
How a documented scaffold became a verified, end-to-end implementation of the modern LLM stack: a GPT trained from scratch, adapted with LoRA, ported to hand-written WebAssembly kernels, ...
Audience: an ML-curious engineer who can read C++/WGSL but hasn't derived the trick before. This doc walks through what "online softmax" means, why it matters for attention specifically, ...
posttrainllm ships five optimizers, selectable on posttrainllm train, sft, and dpo via --optimizer {adamw|lion|sophia|muon|adafactor}. Default is adamw (preserves backward compatibility —...
posttrainllm (the factory) refocuses on local-LLM research. This doc freezes the state of everything posttrainllm owes or has delivered to Pace (the consumer), so either repo can pick up ...
action). Sub-head: "CodeVetter is a desktop review cockpit for the diffs your agent ships. Catch what Cursor, Claude Code, and Devin missed — vulnerabilities, regressions, and silent drif...
Canonical inventory of all models Pace needs (goal set 2026-06-10: all roles decided with runnable artifacts by end of day). One row per role; this doc is the single source of truth — upd...
A bundle of parameter-efficient fine-tuning (PEFT) flavours, all wired into the existing finetune / sft / dpo commands as opt-in flags. The default behaviour (vanilla LoRA r=4 α=8 on q/v ...
Two-item perf + correctness audit, scope-limited to:
A working document for the in-flight push to give users on the latest Chrome every GPU acceleration the platform exposes — without ever degrading model quality. Started during the HN-laun...
A working document for the "best in market" target. Tonight's session landed KV-cached sampling (~2× sustained), 4-bit palettization (6× smaller files), and the ANE conversion path. This ...
How fast posttrainllm trains, what has been done to speed it up, and what is left. All numbers are from an Apple M5 Pro laptop.
This is the stop point for planner-model churn. Starting tomorrow, planner work should be integration, routing, latency, and daily-use fixes, not another base-model search.
Single source of truth for what's shipped, skipped, and still to build. Consolidated from docs/roadmap/*, docs/progress.md, docs/backlog.md, docs/feature_audit_2026_05_31.md, and docs/roadmap/recent_research.md (paper catalogue). Replaces them as the canonical reference; the older docs are now pointer stubs or archived under docs/archive/.
How posttrainllm ensures every accelerated code path preserves model quality before it activates for the user. The non-negotiable rule from the opportunistic-acceleration scope (see docs/...
Consolidated into docs/PLAN.md §1 (shipped). Headline metrics live there.
Generated: 2026-07-02T00:00:00.000Z
Pruning is the post-training technique where you remove parts of a trained model and accept a small quality hit in exchange for a smaller model. There are two flavours here, with very dif...
A chronological reconstruction of the working session — the questions asked, what shipped in response. Format is question-then-answer, with an Outcome line per entry pointing at what actu...
Three loose pieces of the build/test infrastructure shipped together so the next agent inherits a CI floor that catches the obvious regressions before they reach main.
This drop ships four new quantization features on top of the existing AWQReader (HF AWQ safetensors loader) and HQQ (storage-only Half-Quadratic Quantization) that already live in the tree:
quickstart turns a data file into a trained, runnable specialist on your Mac with one command and zero ML knowledge: it inspects the data, auto-picks a base from the gallery, infers a LoR...
This is the canonical entrypoint for posttrainllm documentation.
Written 2026-06-11 after the clarify-v1 experiment failed and the small-specialist training bet closed. This doc replaces no other; it records what we learned at high cost so future work ...
Consolidated into docs/PLAN.md §3 (TODO). Tier A/B/C/D ordering preserved.
This document captures a single long working session on posttrainllm — what was believed coming in, what the contradictions turned out to be, what got shipped, and what the transferable l...
Multi-hour session that took posttrainllm from "Wave 2.5 stalled" to "every CLI verified + first specialist trained end-to-end + ROI-ordered backlog." What shipped, what was learned, what to pick up next.
Two joint-trained draft-head architectures that bolt onto a frozen base LM and propose multiple tokens per base-forward, verified in a single extra base forward pass. Implemented in the n...
Two changes that let posttrainllm sample survive arbitrarily long generations on a memory-bounded Mac:
The technical material this session touched, organized so you can pick what to learn at your own pace. The chapters are thematic, not chronological. Each one answers the same two question...
posttrainllm targets EXEMPLARY test coverage across all three runtimes: Swift (native Mac), Python (reference impl), and TypeScript (browser).
Design + training recipe for posttrainllm's tiny encoder model that picks which tool a user query needs, before the full LM forward pass.
posttrainllm train --depth N derives every pretrain hyperparameter from one number — architecture and the learning-rate / batch / step schedule — following the nanochat surface (karpathy/...
posttrainllm exposes an Ollama-compatible HTTP surface so it drops straight into Continue.dev, Cline, and Aider configs as a local provider. Setup + caveats.
This document records the v9 baseline against each of the six v11 ship-gate dimensions (pace-planner-v11-ship-gate.md). Frozen at commit time. Not for revision — v11 progress measured aga...
This doc captures the validation runs and workflows that exercise the features shipped in Phases 1-10. The goal isn't peer-review rigour — it's enough evidence that each surface RUNS end-...
Researched 2026-06-10 via two web sweeps (platform + product). Sources at bottom; unverified items marked. macOS 27 is "Golden Gate", fall 2026.
This doc captures the implementation and smoke-test measurements for YOCO (Lin et al., 2024) in the Mac training path (Tier 3.8 of the single-machine roadmap). All numbers below were coll...
Batch-first post-training is the factory default for Mac-local work:
Use this for every public artifact page or report that claims learning.
World-class docs are not enough by themselves. posttrainllm needs validators that refuse weak artifacts.
The eval protocol exists to prevent training against noise.
Hugging Face Hub is the default public artifact store for posttrainllm factory outputs.
LoRA geometry is a learning artifact for the factory. It answers:
posttrainllm's active product is the specialist factory.
A shipped specialist needs a small, inspectable package. The package metadata is committed; the large model or adapter artifact usually stays under ~/.cache/posttrainllm/models/ or anothe...
posttrainllm's active center is a Mac-local specialist factory, not a generic fine-tuning notebook.
Public artifacts are first-class factory outputs. They are different from local run files: a public artifact should be understandable outside this repo, have a small committed metadata su...
Start here for active posttrainllm work.
Every factory run should end with a report, even if the decision is reject.
Each real factory run should write a local run directory:
This is the structured companion to ../audit_2026.md.
posttrainllm should not confuse having a method on the roadmap with having a recipe that is likely to work.
This directory is the canonical place for post-training tactics that may improve posttrainllm factory runs.
This is the SQL-specific ledger of methods, recipes, status, and next smallest tests. It exists to prevent the roadmap from looking complete just because broad method names are present.
This note records the useful techniques extracted from TrainLoop case studies and maps them into posttrainllm recipes. The point is not to copy marketing language. The point is to convert...
Senior/staff topics spanning modern transformer architecture, the linear-algebra↔transformers whiteboard, RAG, agents, evaluation (LLM-as-judge, perplexity, contamination), ML system-design rounds, and classic-ML depth — each with the best external source and repo anchor.
Senior/staff inference-optimization interview topics — KV cache & paging, batching, speculative decoding, quantization, attention kernels/variants, long context, serving architecture — mapped to the best external source and to where this repo touches it.
Senior/staff interview topics for LLM training at scale (parallelism, precision, MoE, data) and post-training (RLHF/DPO/GRPO/reward modeling/distillation), each mapped to the best external source and to where this repo touches it.
The L1/L2/L3 framing for agent context engineering — what we stole from the Shortcut vertical-agents essay and where each steal lives in this repo.
Status: research landed 2026-06-07 Supports: docs/prds/factory-planner-v7-tools-in-prompt.md
What Apple's FoundationModels framework + on-device model give you, the bridge we built to score it on our gates, the measured verdict (can't ground actions; 4096-token context can't hold a tool catalog; not faster), and the strategic decision — use it as a free floor for routing, never as a dependency.
Patterns lifted from castform.com (RL fine-tune SaaS) into posttrainllm's specialist-training surface. Sibling page to docs/learn/agent-context-hierarchy.md.
A 2026 map of the products around "make a model good at your task" and "is my agent any good," with the Mac-first whitespace called out. Evidence behind docs/sessions/2026-06-13-market-landscape-mac-first.md.
This map is the guarantee that the learning corpus covers everything in the project. The ground-up curriculum.md is the spine (10 modules, first principles to self-improving factory). Thi...
This is the owner learning roadmap from first principles to a self-improving factory for Mac-local specialist models: a self-improving factory in practice, not a pile of disconnected cour...
Three transferable training methods from the VibeThinker papers — Diversity-Exploring Distillation, MaxEnt-Guided Policy Optimization (MGPO), and specialist weight-merging — mapped to where each upgrades our own self-improvement loop. The "how a 3B matches flagship reasoning" recipe, and why one of its steps is the direct antidote to the negative transfer we measured.
Status: v1 research synthesis landed 2026-06-07 Supports: docs/prds/factory-ane-inference-pace.md
v8 was trained on v5's 248-row corpus + 59 hand-crafted examples targeting v2's failure modes (semantic disambiguation, multi-element reasoning, abstract reference). Same hyperparameters ...
endpoint that runs zero model inference. Every Pace LoRA we trained scores ≤ this baseline. The fixtures test framework, not model.
Curated articles, papers, and projects relevant to posttrainllm. Each entry is one-sentence-what + one-sentence-why-for-us + link, per the docs preference for leaning on authoritative ext...
A running log of questions, hunches, "wait, why does anyone do it this way?" moments, and tangents that come up during the curriculum sessions.
posttrainllm's learning corpus — a ground-up curriculum from "what's a neural net" to modern training mechanics, plus reference + research notes.
For a 12-layer Huge model: ~108 matmuls in the body + ~1 input lookup + 1 output projection = ~110 matmuls per next-token prediction.
Empirical diagnostic of the ANECCompile / runtime failure on the Qwen3-0.6B stateful CoreML path.
The learning + build agenda for mastering Mac-local AI. Every capability buildable on a single Mac, annotated with fleet coverage (built / partial / to-learn), plus an explicit single-machine ↔ distributed boundary. The spine for "learn everything + build everything buildable on this Mac."
The single most useful lens on this project — which parts compute the model's function (irreducible math, the correctness oracle) vs which parts only make that function cheaper/faster (the optimization layer). Plus the one number that polices the boundary: loss drift.
The architectural distinction between a fine-tuned model and the agent that runs it, and what "making a good model into an agent" actually requires. Mapped onto this repo.
Documented 2026-06-08 from huggingface/transformers/src/transformers/models/qwen3_vl/modeling_qwen3_vl.py. Closes research tasks #277 and #278.
Parked blueprint. The fleet's AI game (autonomous characters in a world) is already a multi-agent RL environment — the scarcest RL ingredient. This is the plan to turn it into a self-improving-agents testbed on the Mac, and the staged roadmap toward multi-agent dynamics. Revisit after distillation + the first GRPO loop.
input → same output, every time.
A neural net is, at the bottom, a function:
Before we can search for "good" values, we need to define what good means. The whole search machinery is built on this one definition.
What can this setup not do, and how do we fix it?
When people say "supervised learning," "neural net," "gradient descent," "RL," "tree regression" — they're naming things from three different axes that are mostly independent and freely c...
You just watched a 22M-param model learn English structure but flunk basic factual recall. The natural question: what changes at 100M, at 1B, at 100B? And: is bigger the only knob?
Sessions 1–5 covered what a neural net IS and what it learns. But we've been hand-waving over a key question: the model is a function on numbers. Where do the numbers come from when the i...
A language model's job is: given the text so far, predict the next token. An agent's job is: given the situation, pick an action. Both are "functions on data" — the math we built in Sessi...
Session 2 covered gradient descent in the abstract: pick parameters, compute loss, take a step. Sessions 3–7 covered what the model IS, what data goes in, what paradigms exist. But the ac...
Session 2's model was y = mx + b: one input scalar x, two scalar parameters m and b. A transformer does the same kind of arithmetic — multiply, add, repeat — but over vectors and matrices...
By now a sequence is a matrix: (seq_len, d_model), one vector per token (Session 6), and you can trace a matmul's shapes (Session 9). But those token vectors are still isolated — nothing ...
You can now train and post-train a model (Module 9). The dangerous moment is right after: a number went up and it feels like progress. Most of the skill in this factory is refusing to bel...
SFT tricks, RL, eval, on-device serving — distilled to what's stealable on a Mac.
Voice-pipeline latency, WER, speech-to-speech, fine-tuning debugging, feature selection, queues vs websockets, FSDP2 — each mapped to the best external source and to where this codebase (or Pace) actually does it.
The mental model you need to read posttrainllm's WGSL compute kernels and explain them to another engineer — device/queue, pipeline, dispatch, workgroups, invocations, bind groups, the memory hierarchy — then how our matmul + attention shaders map onto it.
metric, an honest size curve, and a distillation result that closes most of the gap.
Train a 22M-class model with reasoning-style supervision (GRPO / DAPO on verifiable rewards; SFT-on-traces as the warm-up), publish the
Run the Snell et al. 2024 methodology end-to-end at 22M: vary the test-time compute budget (N samples for Best-of-N, M tokens in CoT, K depth in tree search) and measure the quality-vs-FL...
Train a LLaVA-style VL model from scratch on M5 Pro: pre-trained ViT encoder (frozen first cut) + small projector + small-from-scratch posttrainllm decoder + cross-attention or projector....
Train a small (22M-class) diffusion language model from scratch on the masked-denoising objective (Austin et al. 2021; DiffusionBERT, Sahoo et al. 2024). Different generation paradigm tha...
Build the missing-upstream Metal kernels (sparse top-K routing, scatter_add equivalent, expert-parallel matmul gather) so MoE in posttrainllm can run hard routing — actually skipping inac...
Train a from-scratch autoregressive decoder over discrete audio tokens (EnCodec codebook IDs) conditioned on text, à la VALL-E / MusicGen. Single-speaker (LJSpeech), recognizable-but-not-...
Build the smallest useful "explainer video" system: prompt or document → script → storyboard DSL → deterministic SVG/Canvas renderer → MP4 with captions + voiceover. NOT a competitor to S...
Ship one trained specialist that beats Pace's current 0-shot floor (qwen3-4b-instruct-2507) on BFCL average by ≥ 3pp on a Mac-runnable artifact (≤ 8 GB on disk). Validates the platform's ...
Run the factory recipe shape against SQL to validate that the platform is not accidentally tied to a tool-calling target. Same factory loop, different data and eval.
Ship posttrainllm quality-classify <corpus.txt> that scores every document on an educational-quality axis, writes the per-doc scores to a sidecar, and filters to a top-X% subset for downs...
Add --lr-schedule wsd to posttrainllm train (posttrainllm finetune / sft / distill get it for free since they share the optimizer wrapper) with three flags: --warmup N (steps to peak LR),...
Add a grad-norm tracker that triggers an automatic `--resume <step-K>
Replay the shipped interp tools (posttrainllm sae, posttrainllm memit, posttrainllm rome, posttrainllm patch, posttrainllm causal-trace) across the multi-checkpoint history a single postt...
Wire --draft-model <path> into posttrainllm serve and posttrainllm sample so a small "draft" model proposes K tokens per step and the target model verifies them in one parallel forward. E...
Add --llrd γ to posttrainllm sft (and the DPO family for symmetry). For γ < 1, each transformer block's LR is multiplied by γ^(depth_from_top), so the embedding + lower blocks see exponen...
Verify, on posttrainllm's actual MLX path, the Apple ML Research, 2026 claim that the M5 Neural Accelerator delivers 3.5–4× faster prefill vs M4 on the same workload. Bump mlx-swift to th...
Add posttrainllm sae export --format saelens <input.sae> --out <dir> that converts our shipped .sae sidecar format to the SAELens on-disk format (Neuronpedia ingest target). One-way expor...
posttrainllm train --depth 12 ... should auto-derive d_model, n_heads, d_mlp, peak_lr, batch_size, and total_steps from compute- optimal scaling laws. UX win — most users don't want to tu...
Add posttrainllm sae --group-layers SPEC so a single SAE is trained on the concatenated activations of a contiguous block of layers — e.g. --group-layers "0-3,4-7,8-11,12-15" trains 4 SAE...
Bundle the four router items from PLAN.md Tier B into one coordinated PRD because they are sequentially dependent on each other:
Replace the current "hand-wave a 50/30/20 code/web/math split" pattern with a small AutoMixer-style ratio search. Train ~6–12 short proxy runs across candidate ratios, score them on a fix...
Replace single-shot agent eval runs with the Poolside-class protocol: each task is run K times under fixed budget constraints, results are averaged with confidence intervals, and the exac...
Train a Mac-runnable specialist that takes (query, long_context) and returns the subset of sentences relevant to the query — i.e. extractive compression. Submit to the public ScaleDown Ch...
Today, posttrainllm serve --tools <catalog.json> injects every tool's full JSON schema into the system prompt of every request via ServeToolsSpec.systemPrompt() (DynamicGrammar.swift). Fo...
Ship a typed CompositeReward abstraction: a reward is a bag of named scalar dimensions + weights → total. Every dimension is independently logged, plotted, and inspectable. DPO / ES / GRP...
browser/public/gallery/manifest.json so it lists not just the browser-loadable from-scratch .bin models but also the
Ship posttrainllm eval-gate — a single command that runs a project's declared eval suites against a model + adapter, compares to a stored baseline, and exits non-zero when any suite regre...
posttrainllm quickstart <data.jsonl> takes a user's task data and walks them to a trained, evaluated, runnable specialist on their Mac with one command and zero prior knowledge — auto-pic...
Make tinyGPT's eval harness fast and scalable by stealing the two oMLX techniques that fit eval's workload — continuous batching and a
Prove the local-only specialist agent thesis on one concrete vertical: a code reviewer that runs entirely on the user's Mac on a posttrainllm fine-tuned 12B model, with zero cloud depende...
Train a specialist to emit {"defer_to_cloud": true, "reason": "..."} when it shouldn't answer itself — too uncertain, off-domain, or explicitly destructive — instead of fabricating. Today...
The product-shaped artifact that the platform's pitch hinges on: a Mac app the user can open, point at their data, pick a base model, train a LoRA, and run inference on the specialist the...
Train a multilingual SFT/LoRA specialist on a Sarvam-Edge or Airavata base (both are Indic-LLM bases optimized for English + 10+ Indian languages) for a Mac-runnable Indic-capable agent. ...
Measure energy-per-generated-token across the model zoo and across training runs. Surface a j_per_token column on the SLM leaderboard and a per-epoch number on training runs. Answers "wha...
Ship a browser page (/train-viewer.astro) that drag-drops a posttrainllm train run-history directory (or watches it live via OPFS on the same machine) and renders the canonical live-train...
DoRA already trains in-session under posttrainllm sft (default PEFT variant since 2026-05; PeftVariants.swift). It does NOT yet persist to disk in a reusable format — the adapter lives on...
posttrainllm train-extractor (the mini-router-trainer that ships the "intent + tool" classifier on top of the residual stream) currently assumes a byte-level tokenizer. Bases the user act...
Add scripts/bench_decode_thermal.py that runs the existing decode bench in a sustained loop for N minutes (default 30), captures per-window p99 tok/s + die temperature + power draw via po...
Add posttrainllm train --replay-step N --replay-from <ckpt> that runs exactly step N of an earlier training run, deterministically. The output is a logged forward + backward trace identic...
Measure and preserve a small model's GENERAL intelligence when we specialize it (tool-calling, style, persona). This session proved specialization erodes breadth — a retention eval battery + retention techniques, run on every fine-tune. Future work; captured for later.
promote it. The model gets better from real usage — and privacy isn't a compliance checkbox, it's the default.
Ship posttrainllm eval-scaledown <model> that runs the official ScaleBench harness against a posttrainllm-loaded model and emits results in the shared E0 EvalCompare.Row schema. B25 speci...
Tracks the "make the factory complete" thrust spawned 2026-06-07.
Owner approved the previously gated v7 work. The first implementation slice landed:
The VLM elf shipped M1+M2+M3 cleanly. At M3 (and after the leverage-first gate added to the PRD mid-flight), the elf identified that UI-Venus-1.5-2B is the best open-weights student base ...
Context already shipped (M1-M3):
M1-M3 are shipped. The M4 architecture decision is now made in factory-vision-m4-architecture-decision.md: Option A, full Qwen3-VL port with UI-Venus-1.5-2B as base.
Turn the fleet's AI game (autonomous characters in a world) into a Mac-local RL environment and train a self-improving NPC with GRPO. Environments are the scarcest RL ingredient — we own a live one. The parked trigger (distillation closed + a working GRPO loop) is now MET.
Apply GEPA (Genetic-Pareto Prompt Evolution, ICLR 2026 Oral) to automate the system-prompt + action-description iteration loop. Stop hand-tuning prompts; let evolutionary search find the ...
The PRDs below can't be finished-and-verified in a CPU sandbox: they need a GPU (training is the deliverable), special hardware, network installs, or are multi-week from-scratch model bui...
Pit Mac-local models against each other and against frontier models in turn-based strategy games, then self-play-RL a local model until it beats a frontier model playing zero-shot. The match win/lose IS a verifiable reward — RLVR with no hand-authored golds or checker.
Status: IN PROGRESS (2026-06-10) — Phase A SHIPPED, Phase B in flight. Spawned out of the 2026-06-09 research sweep + anemll-vs-M8 decision. Target: 17 tok/s → ~30 tok/s on the existing Q...
A Mac-local, frontier-gated, STATEFUL multi-turn eval — the single biggest blind spot in our tool-calling work. Single-turn numbers (88.7 for our best 4B) overstate agentic ability; small models are known to cliff hard on multi-turn. This is the prerequisite for judging any model as an agent/planner.
v1 through v10 of the Pace planner cycled because we trained → measured → argued whether the version was better. No fixed bar, no fixed eval, no immutable record of pass criteria. The v8 ...
Produce ~450 training rows that teach a Qwen3-0.6B planner to emit three new intent classes:
Status: PRD (2026-06-10). Triggered by a real user moment: bulk-downloading files via the Claude browser extension was painfully slow. The standing rule (memory pace-does-it-not-claude): ...
This file is the working priority map for every PRD currently on disk.
Status: PRD (2026-06-10). Phase-3 centerpiece of the roadmap (specialist close-out → VLM/Pace → this). Implements the 2026-06-10 strategic shift: focus moves from training small models to...
Status: PRD (2026-06-09). Phase 1 SHIPPED 2026-06-10 (serve --quantize); native packed load (the design below) still open.
Active work starts in ../NEXT.md, not here.
Close the act -> score -> learn -> curriculum loop so a Mac-local agent improves itself with NO teacher, using a verifiable reward. The smallest proof: a teacher-free ReST loop on the file-ops env that raises pass-rate round over round. The compounding engine: an automatic curriculum that keeps proposing tasks at the edge of the agent's ability.
Owner approval for heavy work was granted. LM Studio is reachable at http://127.0.0.1:1234/v1/models and includes qwen/qwen3-30b-a3b.
Historical, code-cross-checked status of every active PRD in docs/prds/. Each verdict was verified against native-mac/Sources/, scripts/, evals/, and browser/src/ — not the PRD's own stat...
Status: positioning doc (2026-06-10). Written as the specialist track closes (v11 = final 0.6B planner run, verdict pending) and focus shifts to fine-tuning/distilling larger open models....
Status: PRD (2026-06-09). The 2026-06-09 research sweep flagged Qwen3-VL-2B as a competitor to our UI-Venus-1.5-2B port target. This A/B decides which one we actually port to MLX-Swift / ...
Stop hand-waving "50/30/20 code/web/math". posttrainllm automix searches the ratio: sample candidate mixes, score each with a short proxy train run, fit a quadratic surrogate, and propose...
and outputs an extractive compressed document — a subset of the original sentences that preserves the answer-relevant span — and submit to the ScaleDown.ai challenge.
Use B30 ahead of B29 to control single-hop / multi-hop / comparison ratios in the SFT corpus.
What this is not: a magic 70B replacement. The goal is a small specialist that stays in character on a narrow world better and faster than the base model.
What this is not: an editor plugin. Continue.dev and Aider already know how to call OpenAI-compatible local servers; posttrainllm only has to serve the model.
What this is not: automatic correctness. Pydantic validates shape; your eval still has to measure whether the values are right.
What this is not: a claim that a tiny model beats a frontier model on broad agent tasks. This is for narrow specialists: function calls, schemas, repo idioms, and other repeated local pat...
Compress a big model's capability on your task into a small, cheap, local model on the Mac. This is the validated cost-compression lane — the lever that
(3.8B) on function-call accuracy while running 15-30× faster and using 150× less memory on the same Mac.
Gate a posttrainllm specialist in CI — `posttrainllm eval-gate` exits non-zero when any declared eval suite regresses past threshold, on a self-hosted Mac runner so the model never leaves the device.
Use this when a Pace planner candidate needs an unhappy-path score that is publishable or comparable across runs. Model selection itself is locked in docs/planner-lock-2026-06-19.md; this...
B22 + B29 + B30 + posttrainllm sft chained — the closed substrate→training loop, V1.
Use posttrainllm export-mlx when a posttrainllm-trained artifact needs to leave the posttrainllm binary and be loaded from Python MLX, MLX-Swift, or another Mac-local tool.
The factory's first product-validating arc. Distill Pace's qwen3-30b-a3b planner into a small student. See docs/prds/specialist-pace-planner.md for full spec.
Copy-paste workflows for using posttrainllm specialists outside the core CLI.
Read-and-evaluate write-up on the modded-nanogpt speedrun's "cross-stream attention" trick; decide whether to adopt for posttrainllm.
How posttrainllm scores Indic-language ability — MILU multi-choice across 11 langs, IndicGenBench XQuAD extractive QA. The Wave 4 gate before claiming Hindi support.
between my Jan 2026 knowledge cutoff and current state. Includes URLs for verification.*
The Mac is 10-20× faster than the realtime/interaction-model targets across every model size we have. cider's prefill win is real but immaterial at current scales; defer until there's a 3...
One artifact that cross-cuts decode speed, BFCL, τ-bench, and Pace unhappy-paths — the publication-shape view we're missing.
between my Jan 2026 knowledge cutoff and current state. Includes URLs.*
the engineering vs MLX-Swift / CoreML / Apple defaults?
should know about before training specialists?
Verified URLs, sizes, and licenses for everything in the pretrain → SFT → DPO pipeline. Picked for "this fits on a 48 GB Mac" or "we can stream a slice."
This doc supersedes the rough Tier 5 sketch with a sharper product vision. The project is now:
Consolidated into docs/PLAN.md §2.
Consolidated into docs/PLAN.md.
Consolidated into docs/PLAN.md. Several markers in the old version were stale; the merged doc is verified against code.
Consolidated into docs/PLAN.md. Most phases shipped; remainder in §3.
Consolidated into docs/PLAN.md §3.
Consolidated into docs/PLAN.md.
Consolidated into docs/PLAN.md.
Consolidated into docs/PLAN.md.
Consolidated into docs/PLAN.md §2.
Consolidated into docs/PLAN.md §3 Tier 5.
Consolidated into docs/PLAN.md.
The third of three training phases. See pretrain.md and sft.md for what comes before.
The first of three training phases. See docs/training/index.md for the overview, docs/training/sft.md and docs/training/dpo.md for what comes next.
The second of three training phases. See pretrain.md for what comes before and dpo.md for what comes after.
A modern useful language model is the product of three distinct training phases, each with its own dataset shape, loss function, and goal. This section walks the whole pipeline as it exis...
ANE/CoreML work produced useful measurements and negative results. It is parked until a shipped specialist needs battery/perf optimization.
Broad Mac app polish is parked. The app should not become the center before the factory loop is proven.
The browser playground and WebGPU path are successful completed assets. They are parked for active factory work.
These lanes are not abandoned. They are paused so the factory proof can finish.
These are learning-rich but not the current factory proof.
Vision-language work is parked for the factory proof.
The conversation, condensed. Long replies have been squeezed to their actionable conclusion; emoji-free; some early phases reconstructed from compressed records. What follows is a chat-st...
Phase 9 — and ongoing. Know what the model is actually doing.
The path to learning the whole stack without drowning in complexity. This is the curriculum; model_guide.md, lora_guide.md, and browser_notes.md are the implementation detail.
What didn't work, what surprised me, and what I'd carry forward. The kernel optimisations are documented in docs/performance.md and the milestones in docs/archive/status.md. This is the m...
User decision (this session): hold these four options and focus on the HF-compat capabilities (SwiGLU + RoPE + GQA + BPE tokenizer) first. Once those land, the Mac app can load any modern...
This doc closes out the remaining Phase 9 (quantization) and Phase 10 (architecture menu) items. For each: what's shipped today, and for the items not yet shipped, what's needed to land t...
The 2024-2026-era body of work that informed Tier 1-3. Each entry is just enough context to know whether to dig in.
posttrainllm will ship as two binaries: a browser playground (Chromium + M-series targeted) and a native macOS app (M-series only). They are separate implementations of the same model, op...
A review-oriented snapshot of where posttrainllm stands. The detailed docs are linked at the bottom; this page is the map.
The posttrainllm playground has a small interpretability lever bolted onto the Sample card: click any byte the model just generated and see the two things the model "actually knew" at tha...
be training the model and working on site projects. Before firing the long training run (N02), close every gap that would block scoring the resulting model — schema, harness, baseline com...
benchmarks." The real thesis is: a Mac app where individuals build a specialist for their specific task — bring data, pick a teacher, ship a fast/cheap/comparable-quality model. Distillat...
PRD: docs/prds/specialist-embedder.md
Recommendation: partial-proceed. Do not wrap vllm-mlx into posttrainllm serve yet. Keep native serve as the default and as the eval/logprob backend. Treat vllm-mlx as a production-chat ca...
real FC specialist. Find every gap.
(fine-tune SaaS) and "is my agent any good" (eval/observability). Where does a Mac-first SLM toolkit fit, and what is genuinely unowned? Triggered by the castform.com scan; researched the...
Date: 2026-06-16. The arc, the decisions, and the transferable lessons. Numbers and mechanics live in tool-calling-frontier-parity.md §8.1–8.5; this is the meta-narrative.
Date: 2026-06-17. A deliberate pause to inventory the assets, name the wall, and rank what's next by ROI — before committing to another build. Narrative of the arc lives in the 2026-06-16...
Date: 2026-06-17 (afternoon). Companion to the morning's step-back. Captures the strategic decision triggered by Vercel's Eve launch (blog, public preview June 2026).
LR decay) · Ship gate: +3pp on the Berkeley Function-Calling Leaderboard (BFCL) over the stock base.
Cookie-cuts the A1 recipe onto a second domain to prove platform generality.
Status: expanded 0.6B factory POC complete; next step is preference tuning or a public benchmark slice.