Industry learning roadmap

This is the external learning track for posttrainllm. Use it alongside the canonical ground-up curriculum and practical paths. CS336 supplies external course depth; company docs and papers supply applied case studies. External module numbers below are independent of the ten ground-up curriculum modules.

The goal is not to copy frontier-scale infrastructure. The goal is to extract small, testable ideas that fit posttrainllm: better data, cleaner evals, stronger specialist training, and Mac-first runtime discipline.

How to read

1. Read the source.
2. Write the one sentence lesson.
3. Map it to a posttrainllm artifact: code, doc, eval, or explicit skip.
4. Record a learning checkpoint in learning-progress.md.
5. Start implementation only after the owner opens a fresh, scoped question under NEXT.md. Historical actions below are retained study context, not an active queue or authorization for downloads, training, or benchmarks.

Module 0 - Course spine: Stanford CS336

Source: Stanford CS336 - Language Modeling from Scratch

Why it matters: CS336 is almost exactly posttrainllm’s educational contract. It walks through tokenizer/model/optimizer basics, systems profiling, FlashAttention, distributed memory efficiency, scaling laws, data filtering and deduplication, and SFT/RL-style post-training.

posttrainllm mapping:

Action: add CS336 as the default external course for anyone learning the repo. Do not import assignments wholesale; use it as a reading and audit checklist.

Module 1 - Small model data recipes

Sources:

• Hugging Face SmolLM blog
• FineWeb / FineWeb-Edu paper

Lesson: small models do not win by architecture alone. They need unusually good data: educational-quality text, code subsets, deduplication, and scale-aware evaluation.

posttrainllm actions:

• Keep B10: quality classifier on pretrain data.
• Keep dedupe and MinHash dedupe in the default data path.
• Add every future training run to a manifest with corpus hash, filter settings, and eval set.

Module 2 - Open post-training recipes

Sources:

• Ai2 Tulu 3
• Tulu 3 technical blog
• Tulu 3 report

Lesson: post-training is a recipe, not a single dataset. The useful shape is SFT -> preference tuning -> verifiable-reward RL, with explicit data mixtures and evaluation.

posttrainllm actions:

• Keep docs/training/ as the canonical pretrain/SFT/DPO pipeline.
• Add held-out task evals before claiming any specialist win.
• Treat RLVR as a Tier 5 learning experiment until SFT/DPO specialists are real.

Module 3 - Reasoning and RLVR

Source: DeepSeek-R1 official repo/report

Lesson: reasoning gains come from verifiable rewards and long rollouts, but this is only meaningful after the base model and SFT path are stable.

posttrainllm actions:

• Use math/code tasks with exact checkers before any LLM judge.
• Start with GRPO/DAPO as mental models, not immediate production features.
• Log full trajectories, rewards, and token ids if any RL-style run happens.

Module 4 - Agent design

Sources:

• Anthropic - Building effective agents
• Mistral Agents docs
• Mistral handoffs docs

Lesson: most useful agent systems are simple workflows with good tools. Multi- agent handoffs help only when the boundary is crisp.

posttrainllm actions:

• Prefer tool quality and eval harnesses over more agent layers.
• Keep the router/specialist boundary explicit: one specialist per task family.
• Add handoff only after one specialist can prove it should delegate.

Module 5 - Agentic coding and eval discipline

Source: Poolside Laguna deep dive

Lesson: the stealable pieces are not 30T tokens or 6,144 GPUs. They are data mixing discipline, repeated agent evals, token-preserving trajectories, and careful sandbox budgets.

posttrainllm actions:

• Keep B21: micro-AutoMixer for specialist data mixes.
• Keep B22: token-preserving agent trajectory recorder.
• Keep B23: repeated pass@1 agent eval protocol.
• Keep B24: Muon only after a large/proxy-scale re-benchmark.

Module 6 - Evals as product infrastructure

Sources:

• OpenAI Evals cookbook
• OpenAI structured-output eval example
• OpenAI Model Spec evals note

Lesson: evals should be scenario-shaped and rubric-shaped, not just aggregate leaderboard numbers. Structured-output tasks need schema validity plus semantic grading.

posttrainllm actions:

• Add custom evals for each specialist before training the specialist.
• For JSON/tool/storyboard outputs, score both schema validity and task success.
• Keep public benchmark scores separate from repo-local product evals.

Module 7 - General foundation model reports

Sources:

• Meta Llama 3 Herd of Models
• Qwen docs: key concepts

Lesson: foundation-model reports are useful for phase structure, eval breadth, tokenizer and multilingual choices, and safety/post-training taxonomy. They are not directly actionable at posttrainllm scale.

posttrainllm actions:

• Read these for vocabulary and comparison tables.
• Do not chase 100B-scale architecture changes unless a tiny proxy can falsify the idea first.

Module 8 - Mac/local runtime

Sources:

• Apple MLX open-source project
• Apple MLX M5 LLM research note

Lesson: posttrainllm’s differentiator is not beating CUDA. It is making local, inspectable model training and inference work well on Apple Silicon.

posttrainllm actions:

• Keep Mac-native MLX paths first-class.
• Measure TTFT, throughput, memory, and energy before adding runtime tricks.
• Treat CoreML/ANE/GPU changes as measured runtime work, not roadmap glamour.

Module 9 - Specialized visual/video systems

Sources:

• Lamina Labs
• Qwen Image technical direction for a larger-scale multimodal comparison point.

Lesson: for posttrainllm, the feasible first step is a structured explainer compiler, not pixel-native video generation.

posttrainllm actions:

• Keep Tier 5.7 scoped to prompt/doc -> script -> storyboard DSL -> deterministic render.
• Train a visual-planner specialist only after storyboard data and evals exist.

Case study - Savante / Aryabhata: specialist data and evaluation

Added 2026-09-19. Study only; local reproduction not attempted.

Sources: Savante, Aryabhata paper, model card.

Read after ground-up Modules 9–10, in the post-training and evaluation-and-factory paths. The published recipe combines model merging, curated reasoning traces, SFT, and verifiable-reward RL for JEE mathematics. Use it to study how domain data and a measurable task shape specialization; do not attribute the whole improvement to RL or treat selected historical comparisons as current frontier parity. The reported H100 training is not proof of Mac-local reproducibility.

Exercise: write a one-page recipe teardown separating initialization, data, trace filtering, SFT, reward, and evaluation. Identify missing ablations, possible leakage boundaries, and what data access a comparable specialist would require. Map each component to the existing factory loop.

Mastery gate: explain what the evidence supports, what cannot be attributed to one training stage, and how to freeze an independent test before adapting the recipe to a Mac-sized target. Record the checkpoint; running it would require a new scoped experiment with a resource budget.

Case study - Bonsai 2 27B: capability retained per deployment cost

Added 2026-09-19. Study only; candidate for a future bounded comparison.

Sources: release announcement, MLX model card, runtime and technical report.

Read in quantization-and-packaging, then connect to runtime-and-agents. Study ternary representation, packing overhead, activation transforms, and kernel support. The current card distinguishes a 5.95 GB GGUF language model from 7.67 GB MLX language weights and an 8.60 GB MLX pack including vision. These are artifact sizes, not peak runtime memory. Its custom-loader warning, earlier-build Apple speed measurements, and differences from the announcement’s benchmark table make provenance part of the exercise. Retention relative to the source model does not establish parity with a frontier model.

Exercise: draft a comparison sheet for Bonsai and an existing small-model baseline: exact weight/runtime revisions, packing, context and reasoning budget, frozen task completion and regression gates, peak RAM, prefill/TTFT, decode throughput, and total task latency. Mark unmeasured fields unknown; separate inference feasibility from post-training feasibility.

Mastery gate: explain why parameter count, file size, RAM, and latency can rank models differently, and specify a fair same-Mac test. A future experiment requires owner selection, a scoped issue, and bounded resources; this reading entry does not authorize model downloads or runs.

Case study - QORL: parameter-aware query optimization

Added 2026-09-19. Learning queued; experiment proposed, not started.

Reference: QORL.

Read after ground-up Module 10 in the evaluation-and-factory path, with runtime-and-agents for dispatcher overhead and fallback behavior. QORL studies post-training a model to propose faster PostgreSQL plans. The owner-proposed experiment below instead isolates offline search and parameter-based routing; LLM training is explicitly outside its scope.

Exercise: turn the PRD into a frozen comparison protocol. Specify parameter splits, workload weights, repeated-measurement/cache policy, timeout accounting, search budgets, SQL-equivalence checks, and how unfamiliar parameters trigger native fallback. Keep search/selection measurements separate from the fresh held-out evaluation. Compare both search methods against native parameter-aware planning and against each other.

Mastery gate: explain how selection bias, cache state, parameter skew, and routing overhead can erase an apparent speedup. Derive break-even executions as tuning cost divided by positive per-execution savings using consistent units; report no finite break-even when savings are non-positive. Distinguish wall-time amortization from monetary amortization.

Owner PRD: Experimental Parameter-Aware Query Optimization

Goal: Test whether offline plan exploration and parameter-based routing outperform PostgreSQL’s native planner.

Experiment:

1. Use an isolated, representative PostgreSQL snapshot, 1–3 parameterized SELECT queries, and real parameter samples.
2. Compare native parameter-aware planning, structured hint sweeping, and QORL-inspired LLM hint proposals. Give both searches equal execution budgets and hard timeouts; preserve SQL semantics.
3. Benchmark candidates repeatedly. Retain a small plan portfolio and learn a simple parameters → plan dispatcher, falling back to native planning for unfamiliar cases.
4. Evaluate on held-out parameters with fresh measurements after selection. Include dispatcher overhead.

Deliverables: Runnable harness, reusable plan portfolio, dispatcher, and comparison report covering latency, regressions, resource usage, tuning cost, and break-even execution count.

Provisional success criteria: At least 20% lower held-out workload execution time, with no greater than 10% p95 latency regression. Retain the LLM only if it provides worthwhile gains over structured search after accounting for tuning cost.

Non-goals: Production integration, LLM training, SQL rewrites, index/schema changes, or a general-purpose optimizer.

Activation boundary: This entry preserves the owner’s proposed experiment as learning material. Before implementation, choose the isolated snapshot, queries, parameter source, explicit budgets, and acceptance metric definitions in a scoped GitHub Issue. Database-optimizer implementation belongs in a separately scoped project under the repo’s factory boundary; the experimental design and evaluation lessons remain part of this learning roadmap.

Case study - Inside vLLM: inference systems and scaling boundaries

Added 2026-09-19. Study only; no installation or benchmark started.

Source: Aleksa Gordić, Inside vLLM: Anatomy of a High-Throughput LLM Inference System, published August 29, 2025. The article analyzes V1 at commit 42172ad (August 9, 2025); treat implementation names and setup details as a historical snapshot.

Read after attention and transformer foundations, in runtime-and-agents, with architecture-and-kernels for execution and memory tradeoffs. Study the request lifecycle, paged KV storage, continuous batching, chunked prefill, prefix reuse, constrained/speculative decoding, and the progression to multi-GPU and multi-node serving.

Exercise, in three passes:

1. Draw a request from admission through scheduling, prefill, decode, and completion. Hand-simulate three requests of different lengths under a fixed token budget and KV-block capacity; show allocation and release.
2. Write a prediction table for which changes help TTFT, inter-token latency, throughput, or memory, and where improving one may hurt another. Connect the predictions to the existing KV-cache notes and inference guide.
3. Separate ideas transferable to Mac-local serving from CUDA-specific execution mechanisms and cluster coordination. Explain what a second machine buys and what communication costs it introduces; do not assume an MLX implementation has vLLM’s exact behavior or features.

Mastery gate: explain why high throughput need not mean low interactive latency, trace block ownership without confusing storage layout with attention math, and justify a single-Mac versus distributed serving choice for a stated workload. Record the trace and predictions in the learning tracker. Refresh upstream documentation before any future implementation; this is boundary study, not a request to recreate or deploy a serving stack.

Case study - Splash: model-specific Mac inference

Added 2026-09-19. Study only; no installation or benchmark started.

Sources: Inco AI’s launch article, the Apache-2.0 repository, and its development guide. The published measurements use an M5 Pro with 48 GB of unified memory and compare complete engine configurations. Treat the results as vendor-reported end-to-end evidence, not an independent benchmark or an ablation of any one technique.

Read after transformer and training-mechanics foundations in runtime-and-agents, with architecture-and-kernels and quantization-and-packaging. Splash trades generic model support for a package built around each supported model: shape-specific fused Metal kernels, a model-specific DFlash 2 draft, packed 4-bit weights, prefix-indexed paged cache, and a startup memory plan. The shared runtime still owns scheduling, cache, and OpenAI/Anthropic-compatible APIs. This is a concrete Mac-local example of when specialization can beat a general engine and what flexibility it gives up.

The article reports, on its stated setup, 74 tok/s short-prompt decode for Qwen3.8-27B, 282 ms cached TTFT on a 32K replay, and 170 tok/s aggregate decode for four concurrent short prompts. Those numbers combine kernels, draft, scheduler, cache, packaging, and recommended settings. They do not isolate the gain from DFlash, kernel generation, quantization, or memory planning, and they do not establish output-quality parity by themselves.

Exercise: design a same-Mac comparison between Splash and one general-purpose engine. Pin the Mac, OS, model revision, runtime commits, quantization, prompt set, output/reasoning limits, and API behavior. Measure cold and cached TTFT, prefill, decode, four-request aggregate throughput, peak process and Metal memory, cache-hit rate, output validity, and task completion. Record packaging and per-model tuning cost separately. Predict where a specialized engine should win and define a case that favors the general engine.

Mastery gate: explain why faster decode alone does not prove a faster agent, how prefix reuse changes repeated-turn latency, why concurrency can widen the advantage of batching, and what must be re-engineered to add a new model. A future install or benchmark requires a scoped issue, an exact-source protocol, and a bounded resource budget; this reading entry authorizes none of them.

Running source queue

Read in this order when updating the roadmap:

1. CS336
2. SmolLM / FineWeb-Edu
3. Tulu 3, then Savante / Aryabhata after ground-up Modules 9–10
4. Anthropic agents
5. OpenAI evals, then QORL for search, routing, and held-out runtime evaluation
6. Poolside Laguna
7. Apple MLX, then Inside vLLM for inference systems and scaling boundaries, Splash for model-specific Mac serving, and Bonsai 2 in the quantization/runtime paths
8. Llama/Qwen reports
9. DeepSeek-R1
10. Lamina/video references

Each time a new source is added, update this file with one of:

• Adopt now: exact owner-authorized issue and frozen experiment contract.
• Adopt later: exact trigger.
• Study only: why it is context, not roadmap.
• Skip: why it does not fit posttrainllm.