--- title: "5.7 specialized explainer-video model (Lamina-like)" sidebar: order: 507 --- **status:** not-started **owner:** unassigned **created:** 2026-06-13 **parent_plan:** docs/PLAN.md §3 Tier 5 (5.7, detailed scoping already present in PLAN.md §5.7) **related_prds:** 5.3-vision-language-toy.md (5.7 ordered after 5.3 — needs the text↔visual bridge validated), A1-first-specialist-tool-caller.md (5.7 also depends on the specialist-training discipline) # PRD — Structured explainer compiler + visual-planner specialist ## Goal Build the smallest useful "explainer video" system: prompt or document → script → storyboard DSL → deterministic SVG/Canvas renderer → MP4 with captions + voiceover. NOT a competitor to Sora/Runway. The first useful version is a **visual-planning specialist** plus a renderer, not a pixel-generative model. Tier-5 research arc. **Ordered after A1-B8 (specialist track) and 5.3 (VL).** Comes last because it depends on both: the specialist training discipline + the text↔visual integration pattern. ## Why now (in this PRD: when 5.7 is finally next) - The explainer-video product shape is asymmetric: huge demand signal (Lamina Labs / Simi product traction), tractable when decomposed into "the model writes a DSL; the renderer paints." Avoids the long-horizon-visual-consistency problem that plagues end-to-end video models. - The publishable artifact is the *compiler* — a model that reliably converts a lesson topic into a storyboard DSL — plus a renderer. Both pieces are recipe-shaped specialist work that the platform already supports. - A real product can ride on top: the V1 system is small enough to run locally and would already help solo educators / course creators / customer-training teams. ## Scope — in The compiler architecture: ``` source document / prompt → lesson script (LLM-generated) → storyboard scenes (constrained JSON / DSL) → visual DSL (objects, labels, arrows, equations, timeline) → deterministic renderer (SVG / Canvas / Manim-style) → captions + voiceover + MP4 ``` Concrete artifacts: | Piece | What | |---|---| | **Storyboard DSL** | JSON schema with concepts, equations, diagrams, timings, camera/stroke actions (constrained by grammar) | | **Renderer (V1)** | SVG/Canvas frames → ffmpeg → MP4. Manim-style API later. | | **Visual-planner specialist** | SFT/LoRA model: prompt/doc → storyboard DSL. This is the first trainable model. | | **Asset library** | Shapes, arrows, axes, code blocks, graph layouts, simple physics/math primitives | | **Data pipeline** | Pair open lessons/transcripts/docs with generated or human-edited storyboards | | **Eval set** | Held-out concepts; rubric for factual correctness, visual grounding, pacing, label consistency, equation validity | | **Editing loop** | User can regenerate one scene, lock script, lock diagrams, export MP4 | Model ladder (V1 = level 1; V2 = level 2; level 3+ = future): 1. **No learned video model:** use a strong text model (or cloud model) to produce the DSL; render deterministically. Validates product + schema fast. 2. **Tiny visual-planner specialist:** finetune posttrainllm or HF-loaded base on prompt/doc → DSL. 3. **Visual critic / evaluator:** model scores whether scene frames match the script, flags bad labels / missing objects / impossible diagrams. 4. **Optional diffusion/image/video model:** decorative assets only, after deterministic explainer path works. ## Scope — out (for V1 + V2) - **Photorealistic video.** Explicitly not the goal. - **Long-horizon visual consistency** across hundreds of frames. - **Real-time generation.** Batch is fine. - **Voice cloning / multi-speaker** in the voiceover. Single TTS voice from the existing TTS arc (5.6) or a system TTS shim. - **Closed-source asset packs.** Open-license shapes only. ## Files to touch | File | Change | |---|---| | `Sources/TinyGPTModel/StoryboardSchema.swift` | new — JSON schema definition + validator | | `Sources/TinyGPT/Storyboard.swift` | new — prompt-doc → DSL CLI (V1: uses a cloud or strong-local model; V2: a trained specialist) | | `Sources/TinyGPT/RenderScene.swift` | new — DSL → SVG → MP4 | | `Sources/TinyGPT/TrainVisualPlanner.swift` | new (V2) — finetune recipe for the visual-planner specialist | | `assets/explainer-primitives/` | new directory — open-license shapes / icons / equations | | `web/src/pages/storyboard-preview.astro` | new (V1) — drag-drop the DSL JSON, render preview frames in-browser | | `docs/research/explainer-video-results.md` | new — V1 + V2 results | | `docs/PLAN.md` | 5.7 ⬜ → ✅ on V2 ship | ## Don't touch - The TTS arc (5.6). 5.7 consumes whatever voice is available (system TTS, MacWhisper-Speak shim, or 5.6 once it lands); no coupling. - Anything in `TinyGPTServe/` — 5.7 is offline-only. ## Acceptance criteria (V1 — no learned visual planner) - [ ] `posttrainllm storyboard "the Pythagorean theorem" --out pyth-board.json --planner cloud-or-strong-local` produces a valid DSL JSON. - [ ] `posttrainllm render-scene pyth-board.json --out pyth.mp4` produces an MP4 that scrubs cleanly and renders all visual elements. - [ ] On a 10-concept eval set, the rubric (manual: factual / visual grounding / pacing / labels / equations) averages ≥ 7/10. - [ ] `web/src/pages/storyboard-preview.astro` renders DSL JSON in the browser for fast iteration. ## Acceptance criteria (V2 — learned visual planner) - [ ] `posttrainllm train-visual-planner --base qwen3-4b-instruct-2507 --data storyboards.jsonl --out vp.lora` runs. - [ ] `posttrainllm storyboard "X" --planner local-vp` produces a DSL that, on the 10-concept eval set, averages ≥ 6/10 (slightly lower bar than V1 cloud-baseline — V2 trades quality for being local). - [ ] Editing loop (regenerate scene 3 while locking scenes 1, 2, 4–6) works. ## Reference patterns - The Lamina / Simi product framing — cited in PLAN.md §5.7. - `factory-pace-planner-v6_1.md` — the constrained-DSL output pattern (Pace's `{intent, payload}` schema is the closest analog). - Manim — the renderer-target pattern. We may not adopt Manim itself, but its DSL shape is a useful reference for V1's schema. ## Open questions - Whether the V1 visual-planner is a cloud-API call or a strong-local model (Qwen3-9B). **Recommendation:** strong-local for V1; cloud is a fallback for the editing loop. - Whether to ship the asset library independently. **Recommendation:** yes — `assets/explainer-primitives/` becomes its own MIT-licensed surface usable beyond posttrainllm.