B35 local-agent vertical PoC — code reviewer on a Mac

status: not-started
owner: unassigned
created: 2026-06-17
parent_plan: docs/PLAN.md §3 Tier B (B35 — future)
related_prds: B22-token-preserving-agent-trajectory.md, B26-deferred-tools.md, B28-composite-reward-framework.md, B29-trace-to-training-data.md, B30-prompt-reasoning-classifier.md, B31-gallery-and-project-pins.md, B32-eval-ci-gate.md
context: docs/sessions/2026-06-17-vercel-eve-and-local-agents.md

PRD — Local-agent vertical PoC (code reviewer on a Mac)

Goal

Prove the local-only specialist agent thesis on one concrete vertical: a code reviewer that runs entirely on the user’s Mac on a posttrainllm fine-tuned 12B model, with zero cloud dependency, and beats (or matches within ε) a frontier-cloud baseline on a fixed code-review benchmark.

If this PoC clears the gate, the wedge — “agents Eve can’t reach because they’re cloud-native” — becomes a real product surface. If it doesn’t, we learn what the gap is before committing more roadmap area.

Why now (post-Eve)

Vercel’s Eve (June 2026) validates the agent-platform thesis at the infra layer, but Eve and all its cloud-native peers (Cursor, Replit Agent, Devin, Cognition) burn frontier-API tokens per invocation. None serves the buyer who needs:

• Zero data leaves the machine (regulated industries: legal, healthcare, defense, finance, classified codebases).
• Zero marginal cost after training (high-volume internal use: large-scale code review on monorepos, batch refactor across services).
• Offline operation (air-gapped environments, intermittent connectivity).

posttrainllm already owns ~70% of the stack for this: QLoRA on 4-14B (posttrainllm sft), serve (posttrainllm serve with B26 deferred tools), trace recorder (B22), trace-to-training (B29), composite reward (B28), eval gate (B32). What’s missing is one shipped vertical that proves the loop runs.

See [[feedback_posttrainllm_north_star]] — formula is (speed × accuracy) / cost. Cost = 0 dominates as long as accuracy is in the ballpark. This PoC is the experiment that measures the ballpark.

Scope — in

Phase 1 — baseline + benchmark (1 week)

• Pick the eval: SWE-bench-Verified code-review subset, or a derived code-review-specific harness over the same repos. Choose based on whether SWE-bench’s review signal is actually informative (it’s primarily a fix-the-bug task). Fallback: build a derived benchmark from princeton-nlp/SWE-bench_Verified rows where the task is “given diff, predict whether the PR was approved + flag the issues the reviewer flagged.”
• Frontier baseline: Claude Opus 4.7 + gemini-3 + gpt-oss-120b on the same eval. Three frontier numbers establish the ceiling.
• Open-baseline: gemma-3-12b-it-qat-4bit (current champion per [[project_drilldown_2026_06_12]]) zero-shot. This is the lower bar the PoC must beat after fine-tuning.

Phase 2 — specialist (1-2 weeks)

• Training data: PR-review pairs from public repos (the-stack-smol’s cousin: PR + review-comment + accept/reject signal from GitHub Archive via B22 style trajectory recording).
• Base: Gemma-3-12B-it-qat-4bit or Qwen3-Coder-14B (decision deferred to Phase 1 once we see baseline numbers).
• Approach: QLoRA + composite reward (B28 dims: identifies-bug, identifies-style, doesn’t-hallucinate, format-compliant). Trained via posttrainllm sft then DPO on (preferred-review, dispreferred-review) pairs.
• Eval-gate (B32) blocks ship until the specialist matches the open baseline within ε on the held-out review eval AND beats it on the unhappy-path slice (PRs the reviewer rejected — the harder signal).

Phase 3 — agent runtime (1 week)

• Thin agent harness: posttrainllm agent subcommand, with the verb model derived from B26’s deferred-tools contract. Tools: read_file, list_diff, comment_inline, summarize_review. A 3-hop loop with composite-reward in-loop self-critique.
• No new infra — reuse posttrainllm serve for the runtime, B26 for the tool surface, B22 for trace recording (which then feeds the next training iteration via B29).
• Optional: VSCode extension or git-pre-push hook for distribution.

Phase 4 — the loop closes (ongoing)

• Every real user review run → B22 trace → B29 trains the next iteration.
• B32 gate blocks regressions; B30 keeps the training-mix balanced.
• This is the self-improving part: the more reviews users do, the better the model gets, all on their Mac.

Scope — out

• Multi-agent orchestration — that’s Eve’s pitch; we don’t compete.
• TypeScript SDK / cloud deployment — Vercel’s home turf.
• Slack/Discord channels — Eve has them; not a local-agent moat.
• Other verticals (legal review, SQL agent, refactor agent) — V2 after this PoC validates the loop.

Acceptance criteria

The PoC ships if all of these clear:

• Frontier baseline number recorded on the chosen eval (Phase 1).
• Open-baseline (zero-shot Gemma-3-12B) number recorded.
• posttrainllm-specialist beats open-baseline by ≥ 5pp on accuracy (matches the [[feedback_posttrainllm_north_star]] 5%/2-weeks bar).
• Specialist runs end-to-end on M5 Pro / 48 GB at ≥ 15 tok/s decode (informal target — the bar is “feels responsive”).
• One real codebase (posttrainllm itself? Or a public OSS repo) has a PR review generated by the agent that a human reviewer rates ≥ 3/5 on usefulness.
• Trace recording (B22) actually fires per agent step; the loop is provably closeable.

The kill criterion (be honest about the negative result)

If, after 4 weeks of focused work, the specialist can’t beat the zero-shot open baseline by 5pp on the chosen eval even on the slice we picked for friendliness, the local-only vertical thesis is weakened enough to deprioritize. Either:

• The QLoRA-on-12B / distilled specialist loop genuinely doesn’t move the needle vs zero-shot strong open models at our compute budget.
• We picked the wrong vertical and code review is too judgment-heavy.
• Frontier pulled too far ahead in 2026-H1 for a 12B specialist to matter.

In any of those cases, the answer isn’t “build harder” — it’s “publish the negative result and keep posttrainllm narrow as a model factory, not an agent company.” Per [[feedback_research_first_doctrine]].

Reference shape

Open questions

• Should the agent ship Pace-style as a Mac app or CLI-first? Mac app gives distribution, CLI is faster to ship and matches the early-adopter buyer (developers).
• License story for training data. Hermes-FC + Apache-licensed PR corpora; do not let an APIGen-MT-style CC-BY-NC license through per [[project_specialists_research_2026_06_09]].
• Distribution. If the PoC succeeds, is it a public model in the gallery (B31), a paid Pace-style app, an OSS release with sponsorship arm, or all three?

What this PRD is NOT

A commitment to ship. It’s a kill-or-validate experiment. If the 4-week timebox runs out without clearing the bar, B35 is closed and the learning is published as a session retrospective per the user’s documentation-as-first-class doctrine.