Game-as-RL-environment PoC (PRD)
Turn the fleet's AI game (autonomous characters in a world) into a Mac-local RL environment and train a self-improving NPC with GRPO. Environments are the scarcest RL ingredient — we own a live one. The parked trigger (distillation closed + a working GRPO loop) is now MET.
status: proposed
Game-as-RL-environment PoC
Makes actionable the parked blueprint in
docs/learn/rl-multi-agent-roadmap.md.
Trigger condition is now met: the distillation thread closed and a from-scratch
MLX GRPO loop exists and is validated (GSM8K + tool-calling — see
tool-calling-frontier-parity.md §5).
Why now (and why this, not more single-turn RL)
- Environments are the scarcest RL ingredient (Prime Intellect built a 2,500-environment Hub precisely because of that). The fleet’s AI game is a live multi-agent environment — an ownable asset most can’t get.
- Single-turn RL saturated on a strong base (GRPO on the 4B was neutral: 87.3→86.7, because a fixed verifiable task has a ceiling). An open-ended game environment won’t saturate — there’s always a better policy. That’s exactly why the game, not another single-turn benchmark, is the right next RL target.
- It’s the on-mission direction: learn everything about agentic/multi-agent RL, on a Mac, in an environment we own.
Goal (PoC scope — deliberately one behavior)
Show one distilled small model, as one NPC’s brain, measurably improve at one in-game behavior via GRPO, trained entirely on the Mac.
| RL ingredient | In the PoC |
|---|---|
| Policy | a distilled small local model (the cost-compression lever), LoRA |
| Environment | the game world (one scenario) |
| Reward | a measurable in-game outcome (goal reached / interaction won) — clean ⇒ RLVR |
Design
- Trajectory recorder (game side): log per turn
(observation → action → outcome → reward). This is the roadmap’s “B22 recorder.” Define the observation serialization + the reward signal for the one chosen behavior. - Reward = a verifiable in-game outcome (binary or graded), so it’s RLVR — the tractable kind. Avoid LLM-judge rewards for the PoC.
- GRPO trainer: reuse the validated loop (group-normalized advantage, KL-to-ref,
grad-accumulation — the same structure as our tool-calling GRPO). Generalize it to
take an
(env.reset, env.step, reward)interface instead of the BFCL prompt set. Skeleton:scripts/archive/game_rl_poc.py(this PR). - Loop: sample K rollouts of the NPC acting in the scenario → reward each → group-normalize → policy-gradient on the LoRA → repeat. Watch the behavior’s success rate trend up.
Acceptance criteria
- One scenario instrumented with a clean reward; trajectories logged.
- GRPO run on a distilled small model as the NPC brain; the chosen behavior’s success rate rises vs the pre-RL baseline over training.
- Stable (KL-bounded, no blowup) — the tool-calling GRPO already proved the loop.
Risks / notes
- Game integration is the dependency — needs the recorder + a programmatic step/reset/reward interface from the game. Scope the PoC to one scenario to bound it.
- Reward hacking / degenerate policies — keep the reward verifiable and the scenario simple; KL-to-ref guards against drift.
- Sample efficiency: GRPO is rollout-hungry; one Mac limits scale. PoC targets a trend, not a finished agent.
Relationship to the other PRs
- The multi-turn eval PRD measures whether small models can hold agentic conversations; this PRD is how to improve them via RL in an open-ended env. Eval first (measure the cliff), then this (train the climb).