GRPO-Clarify v1 — RL on the ambig dimension

GRPO-Clarify v1 — RL on the ambig dimension

Status: PRD (pre-experiment). Owner: Sarthak. Date: 2026-06-12.

Goal

Break the 22% ambig ceiling. The historical 12-config drilldown (docs/DRILLDOWN.md) showed every local model — including the new champion Gemma-3-12B — tops out at ~22% on clarify fixtures while oos (82%) and destructive (77%) are workable. Clarify is the unsolved frontier and it’s a behavior gap, not a knowledge gap: models guess instead of asking. That’s exactly the shape RL-against-a-verifier fixes.

Success = a Qwen3-4B LoRA that beats 22% ambig on held-out h2-ext with zero regression on oos/destructive. Anything else gets discarded.

Base model

Qwen3-4B-Instruct (the locked QLoRA base from the planner-candidates drill — passes 3/5 dims zero-shot, fits training + rollouts on M5 Pro 48GB). LoRA adapters only; no full fine-tune.

Reward = strict scorer

scripts/eval_pace_unhappy.py --strict is the reward function. The lenient scorer is exploitable within minutes of GRPO (topic-word stuffing, prompt echo, target stuffed in payload, degenerate spokenText) — strict mode closes those holes and has an adversarial self-test (--self-test, pure python, no HTTP) that must pass before every training run.

Shaped reward (proposal)

Notes:

• The −0.5 anti-over-asking penalty is mandatory. A previous two-stage shim died from exactly this failure mode (clarify-on-everything); GRPO will rediscover it instantly if every training prompt is ambig. Mix ~30-40% non-ambig prompts (action/answer/oos) into every batch.
• Partial credit (+0.3) exists for gradient signal early on; consider annealing toward strict-only in late training so the policy can’t farm intent-only reward (see risk register).
• Caveat: ultra-short clarify answers like "which app?" (2 words) fail strict’s degenerate-spokenText check and earn only +0.3. That’s acceptable shaping pressure toward fuller questions, but watch for it pinning reward.

Training framework (researched 2026-06-12)

Recommendation: mlx-lm-lora (v2.1.0, Apr 2026). Native MLX GRPO on Apple Silicon, and — the decisive feature — custom reward functions plug in as a python file: --reward-functions-file ./my_rewards.py --reward-functions "clarify_strict". That means the strict scorer drops in directly. Supports Qwen3 (Qwen3-4B-Instruct appears in its examples), LoRA + 4/6/8-bit quantized training, GRPO variants (Dr. GRPO, GSPO, DAPO), and since v0.9.9 its GRPO backend uses native mlx-lm batch generation for rollouts. Also on PyPI.

Alternatives considered:

• trl GRPOTrainer on MPS — transformers auto-detects MPS and recent TRL releases fixed MPS-specific bugs, but the GRPO rollout path is built around vLLM/CUDA; on Mac it’s the slow, less-tested road. Fallback only.
• Doriandarko/MLX-GRPO — pure-MLX GRPO demo pipeline (GSM8K-oriented). Good reference code, less maintained than mlx-lm-lora.
• mlx-examples PR #1233 — the upstream GRPO PR (same author as mlx-lm-lora); the standalone package is ahead of it.

Rollout token-budget math (M5 Pro 48GB)

Assumptions: 4B model ≈ 8 GB bf16 (or ~3 GB 4-bit), ~50 tok/s decode at inference; budget training-loop decode at ~0.5-0.7× that (LoRA-attached policy, batched rollouts, no serve-grade KV tricks). Response length ~120 tok (v11 JSON), prompt ~1.5k tok (system prompt + elements).

Per GRPO step (1 prompt, group size G):

• G=4: ~480 decode tok ≈ 10-19 s
• G=8: ~960 decode tok ≈ 19-38 s

One epoch over 200 training prompts at G=8: _{192k decode tokens ≈ 65 min pure decode, call it **}2 h wall-clock** with prefill + backward

• optimizer. G=4 halves that at the cost of noisier advantages — start G=4 for shakeout, G=8 for the real run.

Eval cost (lenient + strict both run): n=40 ambig prompts × _{120 tok ≈ 5k tok ≈ 2-3 min; full 3-dim gate (}130 fixtures) ≈ 10 min. Cheap enough to gate every checkpoint.

Memory: model + LoRA grads + optimizer + G concurrent KV fits in 48 GB with room. Operational rule: LM Studio’s 8 GB model must be unloaded first — one large model at a time. Run under caffeinate.

Train/eval contamination policy

• The h2 and h2-ext fixture suites (evals/fm-fixtures-{ambig,oos,destructive}-h2{,-ext}) are eval-only. NEVER training prompts. No paraphrases of them either — paraphrase leakage is still leakage.
• Training prompts are generated fresh: new ambiguous scenarios (missing recipient/time/content/quantity, multi-candidate elements) plus non-ambig distractors, from templates + teacher paraphrase.
• Seed material: 149 DPO pairs at ~/.cache/posttrainllm/datasets/clarify-dpo-v1.jsonl — use the prompts as scenario seeds and the chosen/rejected pairs to sanity-check the reward (chosen should score ≥ rejected). Audit first: some chosen completions ("which app?") only earn partial credit under strict.
• h2 = checkpoint-selection eval; h2-ext = held-out ship gate, looked at once, at the end.

Ship gate

1. Self-test green: python3 scripts/eval_pace_unhappy.py --self-test.
2. Ambig on held-out h2-ext (lenient scorer, for comparability with the drilldown numbers) > 22% (Gemma-3-12B champion).
3. oos and destructive on h2-ext: zero regression vs the same Qwen3-4B base + system prompt before RL.
4. Strict-mode ambig reported alongside (the honest number).
5. Any gate fails → discard the adapter. No partial ships.

Reward-hacking risk register

Out of scope (v1)

• Full fine-tune, >4B bases, multi-dim reward beyond the shaped table, distillation from the 30B teacher (separate track), any cloud compute.