B25 ScaleDown Challenge specialist

status: not-started
owner: unassigned
created: 2026-06-13
parent_plan: docs/PLAN.md §3 Tier B (B25)
related_prds: A1-first-specialist-tool-caller.md (sibling specialist; different domain), E6 posttrainllm eval-scaledown (the harness — still ⬜, doc-only at docs/recipes/b25-scaledown.md)

PRD — Extractive context-compression specialist for ScaleDown leaderboard

Goal

Train a Mac-runnable specialist that takes (query, long_context) and returns the subset of sentences relevant to the query — i.e. extractive compression. Submit to the public ScaleDown Challenge leaderboard as “competitive task SLM trained from scratch on a Mac.” A second publicly-scored proof-point alongside A1’s BFCL.

The architectural trick is a token-level relevance classifier head on the residual stream → sentence-level aggregation → threshold-keep (no autoregressive generation). Different from A1’s tool-caller in shape: not a generic chat model, a domain-shaped specialist.

Why now

• A1 (tool-caller) is the first specialist. B25 is the second, and the only one that ships with a public external scoreboard — the exact “shipped specialist trained on a Mac” proof-point the platform needs for credibility beyond our docs.
• The harness (E6 posttrainllm eval-scaledown) is the smaller PRD — it’s one of the still-⬜ Tier-E items. Once it lands, B25 has its scoring.
• The training cost is small (~half-week wall-clock on M5 Pro per the recipe at docs/recipes/b25-scaledown.md); the leaderboard submission cycle is short.

Scope — in

• Base model: qwen3-4b-instruct-2507 (consistent with A1; same Mac runtime story).
• New module: Sources/TinyGPTModel/RelevanceHead.swift — a small Linear(d_model → 2) classification head trained on per-token labels (relevant / not). Attaches at the final residual stream (post-norm).
• New subcommand: posttrainllm compress <query> --doc <doc.txt> --model <base+adapter> --threshold 0.5 --out compressed.txt. Token-level scores → sentence-level mean → threshold filter.
• Training data: MS-MARCO + Natural Questions (D3 pulls these for B25). Teacher-label per-sentence relevance using a stronger local model (Qwen3-9B or cloud-escalate Gemma-3-12B); seeds saved to docs/research/data/scaledown-seed.jsonl.
• Loss: token-level BCE on the relevance head; mask-out non- context tokens (query and instruction) so we score document tokens only.
• Recipe: scripts/recipes/b25-scaledown.sh — data prep, SFT with the new head (frozen base + trained head + LoRA on a few blocks), eval, ScaleBench submission.
• Eval: the E6 posttrainllm eval-scaledown harness scores downstream F1/EM after compression vs full-context.
• Submission: the leaderboard accepts a model + a small wrapper script; we package both.

Scope — out

• Abstractive compression (rewriting / summarizing). Extractive is the V1 path; abstractive is a separate specialist arc.
• Multi-document compression (long-context retrieval across N docs). V1 takes one doc.
• Speed / latency optimization for the compression itself — V1 uses the same base’s forward + the head, with no special inference path. Optimization is V2.

Files to touch

Don’t touch

• A1 specialist artifacts — orthogonal recipe.
• The OpenAI surface in posttrainllm serve — V1 ships compress as a CLI, not a server endpoint.

Acceptance criteria

• posttrainllm compress "what is RoPE?" --doc rope_paper.txt --model qwen3-4b-instruct-2507+b25.lora --threshold 0.5 returns a ≤ 30%-original-length excerpt that, on a held-out QA eval, scores within 5pp F1 of the full document.
• posttrainllm eval-scaledown <model+adapter> (E6) produces a row in the shared E0 schema.
• Submission to the ScaleDown leaderboard succeeds; we land a public rank, regardless of where.
• docs/specialists/b25-scaledown.md includes our rank + the full recipe.
• evals/b25-acceptance.sh re-runs the gate from a clean checkout.

Reference patterns

• Sources/TinyGPTModel/LinearProbe.swift — the small classification- head pattern. Relevance head is the same shape, different label.
• A1 specialist’s recipe (docs/specialists/a1-tool-caller.md) — the user-facing brief template.
• ScaleDown blog
  • their public methodology — cite, don’t redocument.

Open questions

• Whether to share the relevance head across LoRA layers vs train one independently. Recommendation: one shared head + LoRA on the top 4 blocks. Simplest path that still adapts; more aggressive is V2.
• Whether to publish the teacher-labeled training data on HF. Recommendation: yes if licensing of the source corpora allows; open-data submission is more credible than “we trained on something undisclosed.”