# Character 2048 LLM benchmark — posttrainllm > Inspect the retained Character 2048 benchmark failure, replay every local model move, and see why the expected intelligence advantage did not reproduce. Canonical: https://posttrainllm.com/benchmarks/game-2048 [Benchmarks](https://posttrainllm.com/benchmarks)/ Character 2048 Development pilot · 3 paired seeds · no tools # Watch the policy break. Can a 30–50M specialist compress game intelligence from a frontier general LLM? Every frame below is a prerecorded model decision—not an animation of algorithmic play. **Failed benchmark artifact—retained.** The expected intelligence advantage did not reproduce: Sonnet matched random legal play, while the Opus cohort remained incomplete. Gate 0 · before training ## First prove intelligence helps. If a pinned frontier model cannot outperform random legal moves, we stop and choose a different game. No 30–50M specialist is trained merely to beat a broken ruler. **Failed · artifact retained** - Strict invalid decisions **must be 0** - Paired win rate vs random **must be ≥60%** - Mean score vs random **must be ≥1.10×** - Bootstrap lower bound **must be positive** - Frozen paired games **30 seeds** Sonnet: 0.995× random over four complete games. Opus: 1.058× over three complete games—below the 1.10× bar—then one provider disconnect. No 30-seed run; no specialist training. Recorded output ## Decision replay Load replay JSON Score **0** Max tile **2** Move **0** → Speed 0.5×1×2×4× Model **—** Seed **—** Exact character input `—` Legal **—** Raw output **—** Parsed move **—** • Choose a run to inspect its first decision. Episode score — Terminal reason — Trace — Paired development slice ## What happened 128-move cap | Policy | Role | Mean score | Mean max tile | Moves | Invalid | Median decision | | --- | --- | --- | --- | --- | --- | --- | | Qwen3 0.6B · 0.6B | small base | 7 | 3 | 4.0 | 3 | 113 ms | | Qwen3 4B Instruct · 4B | general LLM | 845 | 85 | 100.7 | 0 | 227 ms | | Qwen3.5 9B · 9B class | general LLM | 271 | 27 | 54.3 | 2 | 403 ms | | Our 2048 specialist | specialist | Not trained — benchmark rejected at Gate 0 | | | | | **Decision:** do not train on this benchmark. Sonnet matched random legal play; Opus showed only a small, incomplete advantage below the admission threshold. Keep this artifact as a documented failure, then require the next candidate to demonstrate a robust frontier-versus-random capability gradient. Engine sanity checks ## Not model competitors. Never training labels ### Random legal moves random-legal 1009 mean score Uniformly samples only from actions that change the current board. **Built-in advantage:** It can never choose an illegal or no-op direction, so it measures a valid-executor floor rather than intelligence. ### One-move heuristic greedy-one-ply 1319 mean score Simulates every legal move once and scores merge gain, empty cells, monotonicity, and corner position. **Built-in advantage:** It receives the exact transition function; it is an engine sanity check, not an LLM competitor or teacher. Reproduce locally ## Same state. Same parser. Your Mac. Same character observation, greedy decoding, two output tokens maximum, no tools/search/code/rollouts. `python3.12 scripts/games/game_2048_mlx_pilot.py --model --policy-id --track strict --seed 2048000000 --seed 2048000001 --seed 2048000002 --max-moves 128 --output runs/game-2048/model-pilot/.json`Copy command Observation **`B=<16 chars>;S=…;M=…;L=UDLR`** Strict output **Raw one character: U, D, L, or R** Runtime **MLX-LM 0.31.3 on Apple Silicon** Constrained diagnostic **Model chooses only among currently legal character tokens** Download prerecorded outputs ## What this does not prove - This artifact is intentionally retained as a failed benchmark attempt, not presented as a successful or complete reproduction. - Three development seeds; this is not frozen benchmark evidence. - The 9B-class local model is the closest installed general model to the requested 8B comparison. - No 30–50M custom specialist was trained because the benchmark failed before training. - Pinned Sonnet matched random legal play on the valid four-game screen (0.995× mean score). - Pinned Opus completed three games at 1.058× random, below the 1.10× threshold, before a provider disconnect invalidated its fourth game. - These prerecorded local results are strict-track only; the legal-constrained diagnostic is not yet recorded. - Algorithmic policies are diagnostics only and never determine the LLM proof gate.