Local-model arena — turn-based strategy, self-play RLVR (PRD)

Pit Mac-local models against each other and against frontier models in turn-based strategy games, then self-play-RL a local model until it beats a frontier model playing zero-shot. The match win/lose IS a verifiable reward — RLVR with no hand-authored golds or checker.

status: proposed

Local-model arena (self-play RLVR on turn-based strategy games)

Pit Mac-local models against each other — and against a frontier model — in turn-based strategy games, then RL a local model in-environment until it beats the frontier model that plays the same game zero-shot.

Why this is the sharpest version of the north star

• It collapses the whole thesis (see AGENTS.md “Eval philosophy”) into one demoable claim: a 4B RL’d inside a game beats a frontier model playing that game cold. The frontier model is smarter in general but never specialized here — that asymmetry is the entire point, and it’s reachable on a Mac.
• The match outcome is a verifiable reward. No authored checker, no golds — the game decides win/lose. Contrast the BFCL multi-turn gate (multi-turn-agentic-eval.md), which needed a hand-built checker. Cheaper signal, and open-ended so it won’t saturate the way single-turn RL did (the argument is made in game-rl-environment-poc.md — don’t re-litigate it).

Relationship to existing work (DRY — link, don’t re-explain)

• game-rl-environment-poc.md — single NPC improves at one behavior in the fleet’s own game. This PRD is the competitive, multi-model sibling on off-the-shelf games. Same GRPO loop, different environment + an adversary.
• rl-multi-agent-roadmap.md — the parked multi-agent blueprint this realizes.
• tool-calling-frontier-parity.md §5 — the validated from-scratch MLX GRPO loop (group-normalized advantage, KL-to-ref, grad-accum) we reuse via an (env.reset, env.step, reward) interface.
• Reward + trajectory plumbing: B28 composite-reward, B22 trajectory-recorder (shipped).

Steal first (do NOT build an arena from scratch)

• TextArena (Guertler et al., 2025; github LeonGuertler/TextArena) — open-source LLM-vs-LLM framework: ~50+ turn-based text games, a reset/step env API, string observations/actions, a player interface, and TrueSkill leaderboards. Plug a local mlx player and a frontier player in as two agents. This is the load-bearing steal.
• Project Vend (Anthropic, 2025) — LLM runs a vending-machine business; reference for “economic/strategy game as a benchmark.”
• Generative Agents / Smallville (Park et al., 2023, arXiv:2304.03442) — the town/shop flavor, for the later “arena world” direction.

Game selection (the one real design call)

Pick for: turn-based, crisp + cheap-to-check win condition, RL-improvable (learned heuristics beat brute reasoning), cheap to simulate many rollouts, low observation/action token cost.

• Avoid: real-time action / “button-mashing” (LLMs have no reflex or latency edge); deep pure-reasoning games where frontier + search dominate and a 4B can’t catch up (chess, Go); trivially solved games (tic-tac-toe).
• Sweet spot (turn-based strategy): a small resource/territory game (simplified Catan/Risk-like), a negotiation/auction game, a card/deduction game (simplified poker, Liar’s Dice), or positional games with a branching factor that rewards learned heuristics. TextArena ships many of these off the shelf.
• PoC pick: ONE such TextArena title with a clean win condition and short transcripts. Default to a 2-player negotiation or simple-strategy game.

Design

1. Player adapter — wrap mlx_lm generate as a TextArena agent (observation string → legal action string). Frontier agent reuses the OpenAI-style client from scripts/bfcl_multiturn_deepseek.py.
2. Baseline tournament (“before”) — local-stock vs frontier-zero-shot vs scripted/random; record win rates (TrueSkill).
3. Self-play RLVR — GRPO on the local policy; reward = game outcome (win +1 / lose −1, optional shaping). Opponent = a frozen snapshot / past-self pool so the policy isn’t chasing a moving target; refresh the pool periodically.
4. Re-tournament (“after”) — does the RL’d local model beat frontier-zero-shot? Track the win-rate trend during training + the final head-to-head.

Acceptance criteria

• Local + frontier players run in the arena on one turn-based strategy game; baseline win rates recorded.
• Self-play GRPO run; local win rate vs a fixed opponent rises over training (stable, KL-bounded — the loop is already proven).
• Headline: RL’d-local win rate vs frontier-zero-shot ≥ 50% on that game (a real “beat frontier cheaply” artifact) — or a documented honest negative with the reason (frontier too strong / game too reasoning-bound / reward hacked).

Risks / notes

• Illegal-move loops / reward hacking — enforce legal-move masking (grammar- constrained decoding; see factory-serve-grammar, shipped) and penalize illegal moves; keep the reward strictly the game outcome.
• Self-play collapse / cycling — keep an opponent pool of past snapshots; always evaluate against a fixed external baseline (frontier-zero-shot / scripted), not just self-win-rate.
• Frontier API cost — train against local self-play; use the frontier opponent only for periodic evaluation, cached + capped.
• Sample efficiency — GRPO is rollout-hungry and one Mac bounds scale; the PoC targets a trend on one game, not a general game-player.
• Sequencing — phase-2. Trigger only after the tool-calling 4B distill→GRPO loop lands; don’t let it derail frontier-parity work.

Future — the actual “arena product”

Many games + a public Mac local-model leaderboard (TrueSkill), sibling of the agentic leaderboard. Fold the viewer into the eval-leaderboard viewer (shipped) / B31 gallery.