Roadmap — open-source datasets
Roadmap — open-source datasets
Verified URLs, sizes, and licenses for everything in the pretrain → SFT → DPO pipeline. Picked for “this fits on a 48 GB Mac” or “we can stream a slice.”
Pretraining corpora
| Dataset | Size | License | Notes |
|---|---|---|---|
| HuggingFaceFW/fineweb-edu | 1.3 T tokens (default), sample-10BT / 100BT / 350BT subsets, 114 total |
ODC-BY-1.0 | Our pick. Educational-quality web text; each doc has a 2.5-5.06 quality score. CC-MAIN snapshots from 2013-2025. |
| HuggingFaceFW/fineweb | 15 T tokens | ODC-BY-1.0 | Broader, less-filtered parent of FineWeb-edu. Use when fineweb-edu becomes a bottleneck. |
| togethercomputer/RedPajama-Data-V2 | 30 T+ tokens | Apache 2.0 + various | RedPajama’s open Llama-equivalent. Mostly Common Crawl. |
| cerebras/SlimPajama-627B | 627 B tokens | various | Deduplicated subset of RedPajama-V1. |
| allenai/c4 | 750 GB | ODC-BY | Cleaned Common Crawl (T5’s pretraining data). |
| wikimedia/wikipedia | ~20 GB English | CC-BY-SA | Clean encyclopedic prose. Pairs well with web corpora. |
| HuggingFaceTB/cosmopedia | ~25 B tokens | Apache 2.0 | Synthetic educational text generated by Mixtral. Used to train SmolLM. |
| HuggingFaceTB/smollm-corpus | ~600 B tokens | Apache 2.0 | SmolLM’s pretraining mix — Cosmopedia + FineWeb-edu + Stack. |
| roneneldan/TinyStories | ~2 GB | CDLA-Permissive-1.0 | Designed for sub-100M-param models specifically. Vocabulary capped at ~1500 words; the 1M-param coherence threshold. |
| monology/pile-uncopyrighted | ~825 GB | various | The Pile minus copyrighted portions. Pythia’s training data. |
| bigscience/roots | 1.6 T tokens | varies | BLOOM’s pretraining corpus. Multilingual. |
| allenai/dolma | 3 T tokens | various | OLMo’s training data. Web + academic (arXiv + Semantic Scholar) + code (StarCoder) + Reddit + StackExchange + books. More diverse mix than FineWeb (which is pure web). Smaller (3T vs FineWeb’s 15T) but higher per-token diversity. |
SFT (supervised fine-tuning)
| Dataset | Size | License | Notes |
|---|---|---|---|
| databricks/databricks-dolly-15k | 15 K pairs | CC-BY-SA-3.0 | Hand-written, high quality. Our first-run pick. |
| HuggingFaceH4/no_robots | 10 K pairs | CC-BY-NC-4.0 | Hand-written, broader topic coverage than Dolly. |
| allenai/tulu-3-sft-mixture | 939 K pairs | ODC-BY-1.0 + mixed | 18 datasets blended. Includes Persona-MATH, WildChat-GPT4, FLAN-v2, Evol-CodeAlpaca, OpenAssistant-Guanaco, WildGuardMix, etc. The kitchen-sink SFT set. |
| teknium/OpenHermes-2.5 | ~1 M conversations | mixed | Distilled from GPT-4. Higher quality per pair than Tulu. |
| Open-Orca/SlimOrca | ~518 K | MIT | Slimmer Orca; distilled instructions. |
| tatsu-lab/alpaca | 52 K | CC-BY-NC-4.0 | GPT-3.5-generated; classic. Lower quality per pair. |
| OpenAssistant/oasst1 | ~10 K conversations | Apache 2.0 | Multi-turn, human-labeled. |
| HuggingFaceH4/ultrachat_200k | 200 K conversations | MIT | Large-scale multi-turn chat. |
Preference data (DPO / KTO / SimPO / ORPO)
| Dataset | Size | License | Notes |
|---|---|---|---|
| HuggingFaceH4/ultrafeedback_binarized | 187 K paired (61K train_prefs split) | MIT | Our DPO pick. Six splits: train_sft / train_prefs / train_gen × test/train. GPT-4 judgments. TruthfulQA-decontaminated. |
| argilla/dpo-mix-7k | 7 K paired | mixed | Smaller, cleaner. Good for small DPO runs. |
| argilla/ultrafeedback-binarized-preferences-cleaned | 60 K paired | MIT | UltraFeedback with corrected labels (~hundred fixed). |
| allenai/llama-3.1-tulu-3-8b-preference-mixture | 270 K paired | ODC-BY | Tulu-3’s preference mix; uses on-policy DPO data. |
| anthropic/hh-rlhf | 160 K paired | MIT | Human-labeled “helpful” and “harmless” preferences. Slow to grow but human-grade. |
| nvidia/HelpSteer3 | 40 K | CC-BY-4.0 | Single-label good/bad (use with KTO). |
| argilla/distilabel-capybara-dpo-7k-binarized | 7 K paired | mixed | Smaller curated set. |
Code data
| Dataset | Size | License | Notes |
|---|---|---|---|
| bigcode/the-stack-v2 | 3.1 B files (~67 TB raw) | various | Largest permissive-license code corpus. Use subsets. |
| codeparrot/github-code-clean | ~110 GB | various | Pre-filtered Github code. |
| bigcode/the-stack-dedup | ~3 TB | various | Deduplicated v1. |
| nickrosh/Evol-Instruct-Code-80k-v1 | 80 K | MIT | Code SFT data via Evol-Instruct. |
| HuggingFaceH4/CodeAlpaca_20K | 20 K | CC-BY-4.0 | Hand-cleaned code instructions. |
Math + reasoning
| Dataset | Size | License | Notes |
|---|---|---|---|
| meta-math/MetaMathQA | 395 K | MIT | Math instruction-following. |
| AI-MO/NuminaMath-CoT | 860 K | Apache 2.0 | Olympiad-style math + Chain-of-Thought. |
| openai/gsm8k | 8.5 K | MIT | Grade-school math word problems. The standard math benchmark. |
| hendrycks/competition_math | 12.5 K | MIT | MATH dataset; competition-level math. |
Eval / benchmark datasets
| Benchmark | Source | Notes |
|---|---|---|
| MMLU | cais/mmlu | 57-task knowledge benchmark. Standard for “general capability.” |
| HellaSwag | Rowan/hellaswag | Common-sense continuation. |
| TruthfulQA | truthfulqa/truthful_qa | Truthfulness eval. |
| ARC | allenai/ai2_arc | Grade-school science MC. |
| GSM8K | (see math row above) | Math word problems. |
| MT-Bench | HuggingFaceH4/mt_bench_prompts | LLM-judge instruction-following. |
| AlpacaEval | tatsu-lab/alpaca_eval | Pairwise GPT-4 judgments. |
| IFEval | google/IFEval | Instruction-following with verifiable constraints. Rule-based eval — no LLM judge. |
2025-era datasets
| Dataset | Use | Notes |
|---|---|---|
| open-thoughts/OpenThoughts-114k | Reasoning SFT | Open-source reasoning traces, R1-style |
| deepseek-ai/DeepSeek-R1-Distill-Qwen-7B | Distillation teacher | (And siblings: 1.5B, 14B, 32B). If we want to do reasoning distillation, these are the obvious teachers. |
| agentica-org/DeepScaleR-Preview-Dataset | RLVR / math | Math problems with verifiable answers |
| Magpie-Align/Magpie-Llama-3.3-1M | Synthetic SFT | 1M synthetic SFT pairs from Llama-3.3-70B (Jan 2025). |