Dataset inventory — what's available, sizes, schemas

Reference doc for every dataset wired into posttrainllm — registry entries, what they're for, how to pull them, what the records look like after conversion, and known gotchas (gated datasets, parquet decode).

Dataset inventory

This is the working reference for “what dataset should I use for X.” Every entry is in the curated list-datasets registry; this doc adds the practical bits (downloadable today? what’s in the file? known gotchas?) the registry doesn’t cover.

Quick map:

• Browse: posttrainllm list-datasets [--specialist kind | --format sft|dpo|plain]
• Pull: posttrainllm download-dataset hf://datasets/owner/name --out path.jsonl
• Convert: posttrainllm extractor-data (BFCL/τ-bench → router pairs)
• Default cache: ~/.cache/posttrainllm/datasets/

Cache snapshot — 2026-06-02

Verified by pulling each Tier A foundational set. Sizes from du -sh:

External (git-cloned, not HF Hub)

Three of the four ready-for-training sets (alpaca-cleaned, hermes-fc-v1, orca_dpo_pairs, MetaMathQA) are immediately usable with posttrainllm sft / posttrainllm dpo. Parquet-only sets stage on disk but need a python-side decode pass; see “Known gotchas” §2 below.

GitHub corpus path verified unauthenticated on sindresorhus/is (4 issue→PR records in ~30 s; real corpus build needs GITHUB_TOKEN for the 5,000 req/h limit).

Tool-calling (north-star primary)

Verified pull (commit f566023): hermes-function-calling-v1 schema-sniffed as sft (confidence 75%, chat array → conversations), 11,230 records / 8.4M tokens / 6.5M scored tokens. The response format is <tool_call>{"name": ..., "arguments": ...}</tool_call> — not raw JSON. Trainees must learn this XML-wrap to score on BFCL metrics.

Code + debugger

For the debugger specialist, the natural starting corpus is SWE-bench_Verified + python_code_instructions_18k_alpaca (~62 MB total), with issue→PR pairs from posttrainllm fetch-github added on top for repo-specific context. SWE-bench Verified is also the canonical eval target.

Math + reasoning

Instruct (general)

Preference (DPO)

General pretrain corpora

Indic / multilingual evals (not training data)

Special pipelines (not HF Datasets)

Known gotchas

1. Gated datasets need HF_TOKEN. The CLI surfaces the accept-license URL — copy it, click through, then export HF_TOKEN=hf_xxx. xlam-function-calling-60k is the most prominent example.

2. Parquet shards aren’t decoded yet. Some datasets only ship as .parquet (e.g., Locutusque/function-calling-chatml). The posttrainllm download-dataset CLI surfaces this with a clear error and the cache path. Decode manually via Python pandas / pyarrow until upstream support lands.

3. JSONL vs ChatML wrap is a real footgun. posttrainllm sft --template chatml wraps everything in <|im_start|>user\n{instruction}<|im_end|>\n<|im_start|>assistant\n{response}. The hermes records already prefix "system: ..." inline, so all of that ends up in the user turn at training time. Test prompts at inference must match this shape, NOT the proper <|im_start|>system\n...\n<|im_start|>user\n...\n<|im_start|>assistant you might expect. See docs/specialist_v1_findings.md.

4. macOS reaps /tmp. Long-lived training caches should go to ~/.cache/posttrainllm/ or a stable project directory. /tmp gets cleaned aggressively (saw this mid-session on 2026-05-31).

5. The 22-entry registry isn’t exhaustive. It’s the curated slice that’s been tested with posttrainllm download-dataset’s schema sniffer. Other HF datasets work if you pass the field names manually via --map.

Recommended starting bundles

How to extend this doc

When a new dataset becomes interesting, add a row to the right table

• note any gotchas in the “Known gotchas” section. Keep entries short (one-line schema, one-line status, one-line gotcha). The canonical “what models can train on what” doc is docs/capability_matrix.md.