Skip to content
posttrainllm docs
Esc
↑↓navigate↵open⌘Jpreview
On this page

Industry learning roadmap

Industry learning roadmap

This is the external learning track for posttrainllm. Use it after the repo-local archive/learning_roadmap.md: CS336 is the spine, and company docs/blogs are the applied case studies.

The goal is not to copy frontier-scale infrastructure. The goal is to extract small, testable ideas that fit posttrainllm: better data, cleaner evals, stronger specialist training, and Mac-first runtime discipline.

How to read

  1. Read the source.
  2. Write the one sentence lesson.
  3. Map it to a posttrainllm artifact: code, doc, eval, or explicit skip.
  4. Only implement if it improves a current Tier A/B item in PLAN.md.

Module 0 - Course spine: Stanford CS336

Source: Stanford CS336 - Language Modeling from Scratch

Why it matters: CS336 is almost exactly posttrainllm’s educational contract. It walks through tokenizer/model/optimizer basics, systems profiling, FlashAttention, distributed memory efficiency, scaling laws, data filtering and deduplication, and SFT/RL-style post-training.

posttrainllm mapping:

CS336 piece posttrainllm anchor
Assignment 1: basics python_ref/, tests/test_phase1.py
Assignment 2: systems wasm/, webgpu/, FA2 notes
Assignment 3: scaling configs/, bench/, docs/benchmark_harness_design.md
Assignment 4: data posttrainllm download-dataset, dedupe, dataset registry
Assignment 5: alignment/reasoning RL sft, dpo, future RLVR/Tier 5 reasoning

Action: add CS336 as the default external course for anyone learning the repo. Do not import assignments wholesale; use it as a reading and audit checklist.

Module 1 - Small model data recipes

Sources:

Lesson: small models do not win by architecture alone. They need unusually good data: educational-quality text, code subsets, deduplication, and scale-aware evaluation.

posttrainllm actions:

  • Keep B10: quality classifier on pretrain data.
  • Keep dedupe and MinHash dedupe in the default data path.
  • Add every future training run to a manifest with corpus hash, filter settings, and eval set.

Module 2 - Open post-training recipes

Sources:

Lesson: post-training is a recipe, not a single dataset. The useful shape is SFT -> preference tuning -> verifiable-reward RL, with explicit data mixtures and evaluation.

posttrainllm actions:

  • Keep docs/training/ as the canonical pretrain/SFT/DPO pipeline.
  • Add held-out task evals before claiming any specialist win.
  • Treat RLVR as a Tier 5 learning experiment until SFT/DPO specialists are real.

Module 3 - Reasoning and RLVR

Source: DeepSeek-R1 official repo/report

Lesson: reasoning gains come from verifiable rewards and long rollouts, but this is only meaningful after the base model and SFT path are stable.

posttrainllm actions:

  • Use math/code tasks with exact checkers before any LLM judge.
  • Start with GRPO/DAPO as mental models, not immediate production features.
  • Log full trajectories, rewards, and token ids if any RL-style run happens.

Module 4 - Agent design

Sources:

Lesson: most useful agent systems are simple workflows with good tools. Multi- agent handoffs help only when the boundary is crisp.

posttrainllm actions:

  • Prefer tool quality and eval harnesses over more agent layers.
  • Keep the router/specialist boundary explicit: one specialist per task family.
  • Add handoff only after one specialist can prove it should delegate.

Module 5 - Agentic coding and eval discipline

Source: Poolside Laguna deep dive

Lesson: the stealable pieces are not 30T tokens or 6,144 GPUs. They are data mixing discipline, repeated agent evals, token-preserving trajectories, and careful sandbox budgets.

posttrainllm actions:

  • Keep B21: micro-AutoMixer for specialist data mixes.
  • Keep B22: token-preserving agent trajectory recorder.
  • Keep B23: repeated pass@1 agent eval protocol.
  • Keep B24: Muon only after a large/proxy-scale re-benchmark.

Module 6 - Evals as product infrastructure

Sources:

Lesson: evals should be scenario-shaped and rubric-shaped, not just aggregate leaderboard numbers. Structured-output tasks need schema validity plus semantic grading.

posttrainllm actions:

  • Add custom evals for each specialist before training the specialist.
  • For JSON/tool/storyboard outputs, score both schema validity and task success.
  • Keep public benchmark scores separate from repo-local product evals.

Module 7 - General foundation model reports

Sources:

Lesson: foundation-model reports are useful for phase structure, eval breadth, tokenizer and multilingual choices, and safety/post-training taxonomy. They are not directly actionable at posttrainllm scale.

posttrainllm actions:

  • Read these for vocabulary and comparison tables.
  • Do not chase 100B-scale architecture changes unless a tiny proxy can falsify the idea first.

Module 8 - Mac/local runtime

Sources:

Lesson: posttrainllm’s differentiator is not beating CUDA. It is making local, inspectable model training and inference work well on Apple Silicon.

posttrainllm actions:

  • Keep Mac-native MLX paths first-class.
  • Measure TTFT, throughput, memory, and energy before adding runtime tricks.
  • Treat CoreML/ANE/GPU changes as measured runtime work, not roadmap glamour.

Module 9 - Specialized visual/video systems

Sources:

Lesson: for posttrainllm, the feasible first step is a structured explainer compiler, not pixel-native video generation.

posttrainllm actions:

  • Keep Tier 5.7 scoped to prompt/doc -> script -> storyboard DSL -> deterministic render.
  • Train a visual-planner specialist only after storyboard data and evals exist.

Running source queue

Read in this order when updating the roadmap:

  1. CS336
  2. SmolLM / FineWeb-Edu
  3. Tulu 3
  4. Anthropic agents
  5. OpenAI evals
  6. Poolside Laguna
  7. Apple MLX
  8. Llama/Qwen reports
  9. DeepSeek-R1
  10. Lamina/video references

Each time a new source is added, update this file with one of:

  • Adopt now: exact Tier A/B task.
  • Adopt later: exact trigger.
  • Study only: why it is context, not roadmap.
  • Skip: why it does not fit posttrainllm.

Was this page helpful?