GEPA prompt evolution — automated system-prompt iteration

Date: 2026-06-09 Status: PARKED — activate after v11 ships Trigger: when a v12 planner is needed AND v11 ship-gate eval suite is stable

What

Apply GEPA (Genetic-Pareto Prompt Evolution, ICLR 2026 Oral) to automate the system-prompt + action-description iteration loop. Stop hand-tuning prompts; let evolutionary search find the Pareto-front.

Reference application: NousResearch/hermes-agent-self-evolution — a Hermes-specific wrapper. The wrapper is illustrative; we’d use GEPA directly, not the wrapper.

Why now (motivation)

10 versions of Pace planner system prompts (v3 → v10) were all hand-tuned. Each iteration cost a person-day of analysis + writing. GEPA’s claim: $2-10 per optimization run with trace-driven mutations against a fixed eval suite. That replaces 4-6 of our cycles with one API-driven loop.

Prerequisite — already done as of 2026-06-09:

✅ Locked eval suite (v11 ship gate)
✅ 6 dimensions, fixed thresholds, immutable
✅ Multi-objective fitness available (accuracy_v2, BFCL, abstention, disambiguation, schema, safety, plus prompt-length as cost axis)

Without those, GEPA = noise. With them, GEPA = the right tool.

How

read current pace-system-prompt-v10-actions.txt
       │
       ▼
GEPA: propose N=8 mutations targeting observed failure patterns
       │  (uses execution traces from v9/v10 runs against the eval suite)
       ▼
serve each candidate prompt against fm-fixtures-v2, BFCL-12, OOS, AMBIG, DESTRUCT
       │
       ▼
compute Pareto front: (accuracy, prompt_length, TTFW)
       │
       ▼
human PR review on top 2 candidates
       │
       ▼
ship if all 6 ship-gate thresholds clear AND prompt ≤ existing length × 1.2

GEPA reads why each candidate fails (not just that it fails) — failures inform the next mutation cycle. This is reflective prompt evolution, not blind random search.

What it optimizes for Pace

Target	Why GEPA helps
`pace-system-prompt-v10-actions.txt`	Direct hand-tuning replacement
`grammars/v10-actions/registry.json` action descriptions	These shape tool selection — GEPA can tighten descriptions for ambiguity reduction
Future v11/v12 system prompts	First-time prompts get a Pareto-front exploration instead of “engineer’s first guess”

License notes

Component	License	Use
GEPA (`gepa-ai/gepa`)	MIT	✅ Direct dependency
DSPy (`stanfordnlp/dspy`)	MIT	✅ Optional, GEPA-adjacent
Hermes-Agent-Self-Evolution wrapper	MIT	⚠️ Reference only, don’t import
Darwinian Evolver (`imbue-ai/darwinian_evolver`)	AGPL v3	❌ DO NOT integrate; license-incompatible with commercial Pace

Cost / runtime

GEPA’s API-driven loop calls an LLM (we’d use local Qwen3-14B + thinking, no $$$). Self-hosted: ~3-5 hours wall per optimization run on Qwen3-14B local teacher with no API spend. The published “$2-10/run” assumes paid API.

ROI

	Estimate
Integration effort	~12-16h
P(meaningful accuracy gain)	~60%
Quality of gain	5-10pp on prompt-sensitive dimensions, with NO new training
ROI	~0.2-0.3 (below most queued tasks)
Multiplier	Compounds with v11 training. After v11 trains, run GEPA on v11’s prompt for a cheap second derivative gain.

Activation gate

Two conditions must hold:

v11 has shipped OR v11 has failed and we’re planning v12
The v11 ship-gate eval suite remains the canonical fitness function (do not invent a new one for GEPA)

If both → activate. If either fails → wait.

What NOT to do with GEPA

Don’t use it to chase a single metric (the Pareto formulation prevents this naturally; don’t break it)
Don’t apply it to training data — that’s not what GEPA is for
Don’t pull the Hermes wrapper as a dependency — read it for ideas, then write our own ~150-line integration
Don’t run it during a training run — GPU contention

pace-planner-v11-ship-gate.md — the eval suite GEPA optimizes against
pace-planner-v11-training-data.md — data work is orthogonal; GEPA evolves prompts not weights
Memory: [[feedback-research-first-doctrine]] — we verified this is ICLR 2026 Oral, not pre-print noise