5.6 TTS toy (audio-token GPT)
status: not-started
owner: unassigned
created: 2026-06-13
parent_plan: docs/PLAN.md §3 Tier 5 (5.6, detailed scoping already present in PLAN.md §5.6)
related_prds: 5.3-vision-language-toy.md — sibling research arc; different modality, same shape (external pretrained encoder + decoder fusion)
PRD — Audio-token GPT (TTS) toy on consumer hardware
Goal
Train a from-scratch autoregressive decoder over discrete audio tokens (EnCodec codebook IDs) conditioned on text, à la VALL-E / MusicGen. Single-speaker (LJSpeech), recognizable-but-not-natural speech as V1. The transformer is shipped posttrainllm; the new pieces are codec integration, text→audio conditioning, vocoder decode, and the audio data pipeline.
Tier-5 research arc. ~2–4 weeks. PLAN.md already carries detailed
scoping (§5.6 detailed scoping); this PRD lifts that scoping into
the executable shape.
Why now (and ordering)
- Comes after A1-B8 specialists AND after 5.3 VL. Documented ordering in PLAN.md: the specialist track validates the platform thesis; 5.3 stress-tests the “external pretrained encoder + fuse into decoder” pattern that TTS reuses. Shipping 5.6 first would do that integration speculatively; shipping it after 5.3 means inheriting a validated pattern.
- Publishable as “smallest from-scratch audio-token GPT on consumer hardware” alongside the 5.3 framing.
Scope — in
Pieces already shipped (reused):
- Transformer decoder, KV cache, sampling, MTP heads (for K-codebook prediction)
- Training loop (
posttrainllm train) + the PEFT bundle for downstream fine-tunes CrossAttention.swift(currently used for YOCO; adapt for text- encoder K/V source)
New pieces:
| Piece | Effort |
|---|---|
| EnCodec encode/decode (Swift port of HF EnCodec weights) | ~3-5 days |
| Text→conditioning surface (text encoder + cross-attention OR text-as-prefix) | ~2-3 days |
| Audio data pipeline (LJSpeech / LibriTTS pre-tokenization to codec IDs) | ~2-3 days |
| Eval (WER via Whisper transcription, MOS estimator) | ~2 days |
| First training run on LJSpeech → intelligible speech | 2-4 days wall |
- New CLI:
posttrainllm train-tts <text-corpus> <audio-corpus> --out tts.tinygpt,posttrainllm tts "say something" --out hello.wav.
Scope — out
- Multi-speaker / voice cloning. V1 single-speaker.
- Singing / music generation. Speech only.
- High-fidelity (MusicGen-class). V1 targets intelligible, not natural.
- Streaming TTS (chunked low-latency). Batch generation.
Files to touch
| File | Change |
|---|---|
Sources/TinyGPTModel/Encodec.swift |
new — codec encode/decode |
Sources/TinyGPTModel/AudioConditioner.swift |
new — text-encoder + cross-attn to decoder |
Sources/TinyGPT/TrainTTS.swift |
new — recipe |
Sources/TinyGPT/TTSSample.swift |
new — generate |
Sources/TinyGPT/EvalTTS.swift |
new — WER (Whisper) + MOS proxy |
Sources/TinyGPT/TinyGPT.swift |
three new cases |
docs/research/tts-toy-results.md |
new — paper-shaped report |
evals/tts-smoke.sh |
new — load encodec, encode 1s, decode, file round-trip |
docs/PLAN.md |
5.6 ⬜ → ✅ on ship |
Don’t touch
CrossAttention.swiftoutside of its existing YOCO contract. Add the audio-conditioner use as a callsite, not a refactor.- Anything in
TinyGPTServe/— TTS is offline-only V1.
Acceptance criteria
- EnCodec roundtrip:
posttrainllm encode --in clip.wav --out ids.json && posttrainllm decode --in ids.json --out clip2.wavproduces audio with WER < 5% relative to original (Whisper transcription reference). -
posttrainllm train-tts --text ljspeech-text.txt --audio ljspeech-codec.jsonl --steps 50000 --out tts.tinygptruns in < 4 days on M5 Pro. -
posttrainllm tts "the quick brown fox jumps" --out fox.wavproduces a WAV that, when fed to Whisper, transcribes to a string with WER ≤ 30% vs the original prompt. -
docs/research/tts-toy-results.mdreports WER + MOS proxy- comparison to MusicGen-small (300M) baseline size.
Reference patterns
- 5.3 VL PRD pattern (external encoder + fuse-into-decoder). 5.6 is the audio counterpart.
- VALL-E paper and MusicGen — the recipe templates.
- EnCodec, Défossez et al. 2022 — the codec we use. HF release available.
Open questions
- Whether to use EnCodec’s RVQ codebooks 1:1 (8 codebooks → MTP heads with K=8) or a smaller subset for V1. Recommendation: 4 codebooks for V1 — same shape, less to train, intelligibility preserved per the VALL-E ablations.
- Whether to publish the trained model. Recommendation: yes — consistent with 5.3.