---
title: "5.6 TTS toy (audio-token GPT)"
sidebar:
  order: 506
---

<!-- PRD metadata (moved from frontmatter for Blume schema compatibility) -->
**status:** not-started  
**owner:** unassigned  
**created:** 2026-06-13  
**parent_plan:** docs/PLAN.md §3 Tier 5 (5.6, detailed scoping already present in PLAN.md §5.6)  
**related_prds:** 5.3-vision-language-toy.md — sibling research arc; different modality, same shape (external pretrained encoder + decoder fusion)

# PRD — Audio-token GPT (TTS) toy on consumer hardware

## Goal

Train a from-scratch autoregressive decoder over discrete audio
tokens (EnCodec codebook IDs) conditioned on text, à la VALL-E /
MusicGen. Single-speaker (LJSpeech), recognizable-but-not-natural
speech as V1. The transformer is shipped posttrainllm; the new pieces
are codec integration, text→audio conditioning, vocoder decode, and
the audio data pipeline.

Tier-5 research arc. ~2–4 weeks. PLAN.md already carries detailed
scoping (`§5.6 detailed scoping`); this PRD lifts that scoping into
the executable shape.

## Why now (and ordering)

- **Comes after A1-B8 specialists AND after 5.3 VL.** Documented
  ordering in PLAN.md: the specialist track validates the platform
  thesis; 5.3 stress-tests the "external pretrained encoder + fuse
  into decoder" pattern that TTS reuses. Shipping 5.6 first would
  do that integration *speculatively*; shipping it after 5.3 means
  inheriting a validated pattern.
- Publishable as "smallest from-scratch audio-token GPT on consumer
  hardware" alongside the 5.3 framing.

## Scope — in

Pieces already shipped (reused):
- Transformer decoder, KV cache, sampling, MTP heads (for K-codebook
  prediction)
- Training loop (`posttrainllm train`) + the PEFT bundle for downstream
  fine-tunes
- `CrossAttention.swift` (currently used for YOCO; adapt for text-
  encoder K/V source)

New pieces:
| Piece | Effort |
|---|---|
| EnCodec encode/decode (Swift port of HF EnCodec weights) | ~3-5 days |
| Text→conditioning surface (text encoder + cross-attention OR text-as-prefix) | ~2-3 days |
| Audio data pipeline (LJSpeech / LibriTTS pre-tokenization to codec IDs) | ~2-3 days |
| Eval (WER via Whisper transcription, MOS estimator) | ~2 days |
| First training run on LJSpeech → intelligible speech | 2-4 days wall |

- **New CLI:** `posttrainllm train-tts <text-corpus> <audio-corpus> --out
  tts.tinygpt`, `posttrainllm tts "say something" --out hello.wav`.

## Scope — out

- **Multi-speaker** / voice cloning. V1 single-speaker.
- **Singing / music generation.** Speech only.
- **High-fidelity** (MusicGen-class). V1 targets intelligible, not
  natural.
- **Streaming TTS** (chunked low-latency). Batch generation.

## Files to touch

| File | Change |
|---|---|
| `Sources/TinyGPTModel/Encodec.swift` | new — codec encode/decode |
| `Sources/TinyGPTModel/AudioConditioner.swift` | new — text-encoder + cross-attn to decoder |
| `Sources/TinyGPT/TrainTTS.swift` | new — recipe |
| `Sources/TinyGPT/TTSSample.swift` | new — generate |
| `Sources/TinyGPT/EvalTTS.swift` | new — WER (Whisper) + MOS proxy |
| `Sources/TinyGPT/TinyGPT.swift` | three new `case`s |
| `docs/research/tts-toy-results.md` | new — paper-shaped report |
| `evals/tts-smoke.sh` | new — load encodec, encode 1s, decode, file round-trip |
| `docs/PLAN.md` | 5.6 ⬜ → ✅ on ship |

## Don't touch

- `CrossAttention.swift` outside of its existing YOCO contract.
  Add the audio-conditioner use as a *callsite*, not a refactor.
- Anything in `TinyGPTServe/` — TTS is offline-only V1.

## Acceptance criteria

- [ ] EnCodec roundtrip: `posttrainllm encode --in clip.wav --out
  ids.json && posttrainllm decode --in ids.json --out clip2.wav` produces
  audio with WER < 5% relative to original (Whisper transcription
  reference).
- [ ] `posttrainllm train-tts --text ljspeech-text.txt --audio
  ljspeech-codec.jsonl --steps 50000 --out tts.tinygpt` runs in
  < 4 days on M5 Pro.
- [ ] `posttrainllm tts "the quick brown fox jumps" --out fox.wav`
  produces a WAV that, when fed to Whisper, transcribes to a
  string with WER ≤ 30% vs the original prompt.
- [ ] `docs/research/tts-toy-results.md` reports WER + MOS proxy
  + comparison to MusicGen-small (300M) baseline size.

## Reference patterns

- 5.3 VL PRD pattern (external encoder + fuse-into-decoder). 5.6
  is the audio counterpart.
- [VALL-E paper](https://arxiv.org/abs/2301.02111) and
  [MusicGen](https://arxiv.org/abs/2306.05284) — the recipe templates.
- [EnCodec, Défossez et al. 2022](https://arxiv.org/abs/2210.13438)
  — the codec we use. HF release available.

## Open questions

- Whether to use EnCodec's RVQ codebooks 1:1 (8 codebooks → MTP
  heads with K=8) or a smaller subset for V1. **Recommendation:**
  4 codebooks for V1 — same shape, less to train, intelligibility
  preserved per the VALL-E ablations.
- Whether to publish the trained model. **Recommendation:** yes —
  consistent with 5.3.