Development pilot · 3 paired seeds · no tools

Watch the policy break.

Can a 30–50M specialist compress game intelligence from a frontier general LLM? Every frame below is a prerecorded model decision—not an animation of algorithmic play.

Failed benchmark artifact—retained. The expected intelligence advantage did not reproduce: Sonnet matched random legal play, while the Opus cohort remained incomplete.

Gate 0 · before training

First prove intelligence helps.

If a pinned frontier model cannot outperform random legal moves, we stop and choose a different game. No 30–50M specialist is trained merely to beat a broken ruler.

Failed · artifact retained

Strict invalid decisionsmust be 0
Paired win rate vs randommust be ≥60%
Mean score vs randommust be ≥1.10×
Bootstrap lower boundmust be positive
Frozen paired games30 seeds

Sonnet: 0.995× random over four complete games. Opus: 1.058× over three complete games—below the 1.10× bar—then one provider disconnect. No 30-seed run; no specialist training.

Recorded output

Decision replay

Load replay JSON

Score0

Max tile2

Move0

Speed

Paired development slice

What happened

128-move cap

Policy	Role	Mean score	Mean max tile	Moves	Invalid	Median decision
Qwen3 0.6B0.6B	small base	7	3	4.0	3	113 ms
Qwen3 4B Instruct4B	general LLM	845	85	100.7	0	227 ms
Qwen3.5 9B9B class	general LLM	271	27	54.3	2	403 ms
Our 2048 specialist30–50M (≤50M)	specialist	Not trained — benchmark rejected at Gate 0

Decision: do not train on this benchmark. Sonnet matched random legal play; Opus showed only a small, incomplete advantage below the admission threshold. Keep this artifact as a documented failure, then require the next candidate to demonstrate a robust frontier-versus-random capability gradient.

Engine sanity checks

Not model competitors.

Never training labels

random-legal1009 mean score

Random legal moves

Uniformly samples only from actions that change the current board.

Built-in advantage: It can never choose an illegal or no-op direction, so it measures a valid-executor floor rather than intelligence.

greedy-one-ply1319 mean score

One-move heuristic

Simulates every legal move once and scores merge gain, empty cells, monotonicity, and corner position.

Built-in advantage: It receives the exact transition function; it is an engine sanity check, not an LLM competitor or teacher.

Reproduce locally

Same state. Same parser. Your Mac.

Same character observation, greedy decoding, two output tokens maximum, no tools/search/code/rollouts.

python3.12 scripts/game_2048_mlx_pilot.py --model <MODEL_REF> --policy-id <ID> --track strict --seed 2048000000 --seed 2048000001 --seed 2048000002 --max-moves 128 --output runs/game-2048/model-pilot/<ID>.json

ObservationB=<16 chars>;S=…;M=…;L=UDLR

Strict outputRaw one character: U, D, L, or R

RuntimeMLX-LM 0.31.3 on Apple Silicon

Constrained diagnosticModel chooses only among currently legal character tokens

What this does not prove

This artifact is intentionally retained as a failed benchmark attempt, not presented as a successful or complete reproduction.
Three development seeds; this is not frozen benchmark evidence.
The 9B-class local model is the closest installed general model to the requested 8B comparison.
No 30–50M custom specialist was trained because the benchmark failed before training.
Pinned Sonnet matched random legal play on the valid four-game screen (0.995× mean score).
Pinned Opus completed three games at 1.058× random, below the 1.10× threshold, before a provider disconnect invalidated its fourth game.
These prerecorded local results are strict-track only; the legal-constrained diagnostic is not yet recorded.
Algorithmic policies are diagnostics only and never determine the LLM proof gate.