# Roadmap — open-source datasets

Verified URLs, sizes, and licenses for everything in the
pretrain → SFT → DPO pipeline. Picked for "this fits on a 48 GB Mac"
or "we can stream a slice."

## Pretraining corpora

| Dataset | Size | License | Notes |
|---|---|---|---|
| [HuggingFaceFW/fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) | 1.3 T tokens (default), `sample-10BT` / `100BT` / `350BT` subsets, 114 total | ODC-BY-1.0 | **Our pick.** Educational-quality web text; each doc has a 2.5-5.06 quality score. CC-MAIN snapshots from 2013-2025. |
| [HuggingFaceFW/fineweb](https://huggingface.co/datasets/HuggingFaceFW/fineweb) | 15 T tokens | ODC-BY-1.0 | Broader, less-filtered parent of FineWeb-edu. Use when fineweb-edu becomes a bottleneck. |
| [togethercomputer/RedPajama-Data-V2](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-V2) | 30 T+ tokens | Apache 2.0 + various | RedPajama's open Llama-equivalent. Mostly Common Crawl. |
| [cerebras/SlimPajama-627B](https://huggingface.co/datasets/cerebras/SlimPajama-627B) | 627 B tokens | various | Deduplicated subset of RedPajama-V1. |
| [allenai/c4](https://huggingface.co/datasets/allenai/c4) | 750 GB | ODC-BY | Cleaned Common Crawl (T5's pretraining data). |
| [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) | ~20 GB English | CC-BY-SA | Clean encyclopedic prose. Pairs well with web corpora. |
| [HuggingFaceTB/cosmopedia](https://huggingface.co/datasets/HuggingFaceTB/cosmopedia) | ~25 B tokens | Apache 2.0 | Synthetic educational text generated by Mixtral. Used to train SmolLM. |
| [HuggingFaceTB/smollm-corpus](https://huggingface.co/datasets/HuggingFaceTB/smollm-corpus) | ~600 B tokens | Apache 2.0 | SmolLM's pretraining mix — Cosmopedia + FineWeb-edu + Stack. |
| [roneneldan/TinyStories](https://huggingface.co/datasets/roneneldan/TinyStories) | ~2 GB | CDLA-Permissive-1.0 | Designed for sub-100M-param models specifically. Vocabulary capped at ~1500 words; the 1M-param coherence threshold. |
| [monology/pile-uncopyrighted](https://huggingface.co/datasets/monology/pile-uncopyrighted) | ~825 GB | various | The Pile minus copyrighted portions. Pythia's training data. |
| [bigscience/roots](https://huggingface.co/datasets/bigscience/roots) | 1.6 T tokens | varies | BLOOM's pretraining corpus. Multilingual. |
| [allenai/dolma](https://huggingface.co/datasets/allenai/dolma) | 3 T tokens | various | OLMo's training data. Web + academic (arXiv + Semantic Scholar) + code (StarCoder) + Reddit + StackExchange + books. More diverse mix than FineWeb (which is pure web). Smaller (3T vs FineWeb's 15T) but higher per-token diversity. |

## SFT (supervised fine-tuning)

| Dataset | Size | License | Notes |
|---|---|---|---|
| [databricks/databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k) | 15 K pairs | CC-BY-SA-3.0 | Hand-written, high quality. Our first-run pick. |
| [HuggingFaceH4/no_robots](https://huggingface.co/datasets/HuggingFaceH4/no_robots) | 10 K pairs | CC-BY-NC-4.0 | Hand-written, broader topic coverage than Dolly. |
| [allenai/tulu-3-sft-mixture](https://huggingface.co/datasets/allenai/tulu-3-sft-mixture) | 939 K pairs | ODC-BY-1.0 + mixed | 18 datasets blended. Includes Persona-MATH, WildChat-GPT4, FLAN-v2, Evol-CodeAlpaca, OpenAssistant-Guanaco, WildGuardMix, etc. **The kitchen-sink SFT set.** |
| [teknium/OpenHermes-2.5](https://huggingface.co/datasets/teknium/OpenHermes-2.5) | ~1 M conversations | mixed | Distilled from GPT-4. Higher quality per pair than Tulu. |
| [Open-Orca/SlimOrca](https://huggingface.co/datasets/Open-Orca/SlimOrca) | ~518 K | MIT | Slimmer Orca; distilled instructions. |
| [tatsu-lab/alpaca](https://huggingface.co/datasets/tatsu-lab/alpaca) | 52 K | CC-BY-NC-4.0 | GPT-3.5-generated; classic. Lower quality per pair. |
| [OpenAssistant/oasst1](https://huggingface.co/datasets/OpenAssistant/oasst1) | ~10 K conversations | Apache 2.0 | Multi-turn, human-labeled. |
| [HuggingFaceH4/ultrachat_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k) | 200 K conversations | MIT | Large-scale multi-turn chat. |

## Preference data (DPO / KTO / SimPO / ORPO)

| Dataset | Size | License | Notes |
|---|---|---|---|
| [HuggingFaceH4/ultrafeedback_binarized](https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized) | 187 K paired (61K train_prefs split) | MIT | **Our DPO pick.** Six splits: train_sft / train_prefs / train_gen × test/train. GPT-4 judgments. TruthfulQA-decontaminated. |
| [argilla/dpo-mix-7k](https://huggingface.co/datasets/argilla/dpo-mix-7k) | 7 K paired | mixed | Smaller, cleaner. Good for small DPO runs. |
| [argilla/ultrafeedback-binarized-preferences-cleaned](https://huggingface.co/datasets/argilla/ultrafeedback-binarized-preferences-cleaned) | 60 K paired | MIT | UltraFeedback with corrected labels (~hundred fixed). |
| [allenai/llama-3.1-tulu-3-8b-preference-mixture](https://huggingface.co/datasets/allenai/llama-3.1-tulu-3-8b-preference-mixture) | 270 K paired | ODC-BY | Tulu-3's preference mix; uses on-policy DPO data. |
| [anthropic/hh-rlhf](https://huggingface.co/datasets/Anthropic/hh-rlhf) | 160 K paired | MIT | Human-labeled "helpful" and "harmless" preferences. Slow to grow but human-grade. |
| [nvidia/HelpSteer3](https://huggingface.co/datasets/nvidia/HelpSteer3) | 40 K | CC-BY-4.0 | Single-label good/bad (use with KTO). |
| [argilla/distilabel-capybara-dpo-7k-binarized](https://huggingface.co/datasets/argilla/distilabel-capybara-dpo-7k-binarized) | 7 K paired | mixed | Smaller curated set. |

## Code data

| Dataset | Size | License | Notes |
|---|---|---|---|
| [bigcode/the-stack-v2](https://huggingface.co/datasets/bigcode/the-stack-v2) | 3.1 B files (~67 TB raw) | various | Largest permissive-license code corpus. Use subsets. |
| [codeparrot/github-code-clean](https://huggingface.co/datasets/codeparrot/github-code-clean) | ~110 GB | various | Pre-filtered Github code. |
| [bigcode/the-stack-dedup](https://huggingface.co/datasets/bigcode/the-stack-dedup) | ~3 TB | various | Deduplicated v1. |
| [nickrosh/Evol-Instruct-Code-80k-v1](https://huggingface.co/datasets/nickrosh/Evol-Instruct-Code-80k-v1) | 80 K | MIT | Code SFT data via Evol-Instruct. |
| [HuggingFaceH4/CodeAlpaca_20K](https://huggingface.co/datasets/HuggingFaceH4/CodeAlpaca_20K) | 20 K | CC-BY-4.0 | Hand-cleaned code instructions. |

## Math + reasoning

| Dataset | Size | License | Notes |
|---|---|---|---|
| [meta-math/MetaMathQA](https://huggingface.co/datasets/meta-math/MetaMathQA) | 395 K | MIT | Math instruction-following. |
| [AI-MO/NuminaMath-CoT](https://huggingface.co/datasets/AI-MO/NuminaMath-CoT) | 860 K | Apache 2.0 | Olympiad-style math + Chain-of-Thought. |
| [openai/gsm8k](https://huggingface.co/datasets/openai/gsm8k) | 8.5 K | MIT | Grade-school math word problems. The standard math benchmark. |
| [hendrycks/competition_math](https://huggingface.co/datasets/hendrycks/competition_math) | 12.5 K | MIT | MATH dataset; competition-level math. |

## Eval / benchmark datasets

| Benchmark | Source | Notes |
|---|---|---|
| **MMLU** | [cais/mmlu](https://huggingface.co/datasets/cais/mmlu) | 57-task knowledge benchmark. Standard for "general capability." |
| **HellaSwag** | [Rowan/hellaswag](https://huggingface.co/datasets/Rowan/hellaswag) | Common-sense continuation. |
| **TruthfulQA** | [truthfulqa/truthful_qa](https://huggingface.co/datasets/truthfulqa/truthful_qa) | Truthfulness eval. |
| **ARC** | [allenai/ai2_arc](https://huggingface.co/datasets/allenai/ai2_arc) | Grade-school science MC. |
| **GSM8K** | (see math row above) | Math word problems. |
| **MT-Bench** | [HuggingFaceH4/mt_bench_prompts](https://huggingface.co/datasets/HuggingFaceH4/mt_bench_prompts) | LLM-judge instruction-following. |
| **AlpacaEval** | [tatsu-lab/alpaca_eval](https://huggingface.co/datasets/tatsu-lab/alpaca_eval) | Pairwise GPT-4 judgments. |
| **IFEval** | [google/IFEval](https://huggingface.co/datasets/google/IFEval) | Instruction-following with verifiable constraints. Rule-based eval — no LLM judge. |

## 2025-era datasets

| Dataset | Use | Notes |
|---|---|---|
| [open-thoughts/OpenThoughts-114k](https://huggingface.co/datasets/open-thoughts/OpenThoughts-114k) | Reasoning SFT | Open-source reasoning traces, R1-style |
| [deepseek-ai/DeepSeek-R1-Distill-Qwen-7B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B) | Distillation teacher | (And siblings: 1.5B, 14B, 32B). If we want to do reasoning distillation, these are the obvious teachers. |
| [agentica-org/DeepScaleR-Preview-Dataset](https://huggingface.co/datasets/agentica-org/DeepScaleR-Preview-Dataset) | RLVR / math | Math problems with verifiable answers |
| [Magpie-Align/Magpie-Llama-3.3-1M](https://huggingface.co/Magpie-Align) | Synthetic SFT | 1M synthetic SFT pairs from Llama-3.3-70B (Jan 2025). |