Gepa

---
name: gepa
description: Use when a bounded textual artifact (prompt, rubric, tool description, extraction instruction) keeps underperforming and success can be measured with an evaluator, dataset, or trace set. GEPA proposes evaluator-backed candidate rewrites through a normal PR/proposal adoption gate. Do not use for vague behavior changes, governance/persona/core-memory edits, fake metrics, or problems whose first honest task is defining the evaluator or collecting data.
requires_extras: [gepa]
---

<!-- desc: Optimize bounded textual artifacts with GEPA only when evaluator-backed examples, budget, and a normal adoption gate exist. -->

# GEPA

GEPA is a reusable optimization workflow for **bounded textual artifacts**:
prompts, rubrics, tool descriptions, extraction instructions, review checklists,
or similar text whose quality can be measured on examples.

GEPA is not reflection. Reflection and Five Whys may notice a GEPA-shaped problem,
but GEPA starts only after the target artifact, evaluator, dataset, budget, and
adoption gate are concrete.

## Contract

**Trigger:** Use GEPA when a prompt/rubric/tool description/extraction instruction
keeps underperforming, the artifact is narrow enough to mutate safely, and better
vs. worse can be measured on a trace set, labeled examples, or evaluator.

**Requires:**

1. **Target artifact** — exact file/path or prompt block plus frozen baseline text.
2. **Evaluator** — explicit score and diagnostic feedback per example. Numeric-only
   scores are not enough; the evaluator must return Actionable Side Information
   (ASI): what failed, where, and why.
3. **Dataset / trace set** — representative examples, with a holdout split when
   possible.
4. **Budget / stopping rule** — max metric calls, wall-clock, and minimum adoption
   threshold.
5. **Adoption gate** — optimized text lands as a proposed diff / PR / proposal;
   GEPA never auto-replaces a production prompt.

**Produces:** baseline-vs-candidate comparison, evaluator results, ASI failure
analysis, known tradeoffs, and an adopt/reject recommendation.

**Does not:** optimize identity/persona/core values, replace reflection, repair
missing instrumentation, invent a fake evaluator, or mutate code semantics hidden
behind text.

## Fit checklist

Use GEPA only when all are true:

- The target is one bounded textual artifact.
- The failure recurs or affects an important recurring path.
- A meaningful evaluator or labeled trace set exists now.
- The evaluator can explain failures with ASI, not just return a scalar.
- The artifact can be changed and reviewed independently of broad governance.
- A normal adoption gate can reject the candidate if it overfits or regresses.

If any item is false, do not run GEPA yet. File the prerequisite instead: build the
dataset, write the evaluator, add instrumentation, or run Five Whys/reflection on
the framing.

## Anti-fit checklist

Do not use GEPA when any are true:

- The real problem is ambiguous goals, operator alignment, persona, or governance.
- The artifact is `memory/core/*`, a persona/value block, or a policy boundary.
- The first honest action is "define the evaluator" or "collect examples".
- The score is easy to Goodhart or cannot cite per-example failure reasons.
- The measured outcome depends mainly on non-text knobs GEPA cannot move.
- The proposed change would silently ship without human/operator review.

Rule of thumb: if a competent reviewer could not tell whether candidate text won
for the right reasons, GEPA is premature.

## Standard workflow

### 1. Freeze the task

Create or update a chainlink issue / spec that records:

- artifact path and exact baseline text or commit SHA
- dataset / trace paths
- evaluator command or scoring rubric
- metric priorities and failure modes the metric misses
- `max_metric_calls` / cost cap / wall-clock cap
- adoption threshold and review path

For codebase prompts, the adoption path is a normal feature branch and PR. For
protected memory or prompt-template surfaces, use the protected-surface proposal
workflow; do not edit live surfaces directly.

### 2. Verify evaluator quality before optimizing

Run the evaluator on the baseline and inspect examples. It must return ASI that is
specific enough to guide rewrites, e.g.:

- source example id
- expected behavior / gold label
- actual extraction or decision
- false-positive / false-negative / compression / hallucination reason
- exact source text or trace span that supports the diagnosis

If the evaluator cannot produce this, stop. Improve the evaluator first.

### 3. Run a bounded optimization pass

Use the standalone `gepa` library rather than reimplementing the optimizer. It is
not installed by default — it ships as the opt-in `gepa` extra (the optimizer is
only needed while a pilot runs, so the base runtime stays lean):

- PyPI deployments: `pip install 'mimir-agent[gepa]'`
- workspace / uv deployments: `uv sync --extra gepa`

gepa has no `langchain` extra. Route its reflection LM (and the task model) through
mimir's already-configured ChatModel via `mimir.gepa_support` rather than a separate
LiteLLM / OpenAI key — `reflection_lm` takes a plain `(prompt: str) -> str` callable:

```python
import gepa
from mimir.gepa_support import reflection_lm_from_config

result = gepa.optimize(
    seed_candidate={"instructions": baseline_prompt},
    trainset=train_examples,
    valset=holdout_examples,
    adapter=my_pilot_adapter,           # evaluator + ASI; see step 2 and chainlink #404
    reflection_lm=reflection_lm_from_config(),
    max_metric_calls=100,               # explicit cost cap; raise only after a useful pilot
)
```

Adapt the exact API to the installed `gepa` version, but keep the invariants:
explicit baseline, explicit examples, ASI-rich evaluator, explicit metric-call cap,
and no auto-application of `result.best_candidate`.

### 4. Compare on holdout and write the decision record

Record:

- baseline score and candidate score
- per-failure ASI themes
- examples improved, unchanged, and regressed
- whether gains hold outside the optimization slice
- any new failure mode introduced by the candidate
- recommendation: adopt, reject, or revise evaluator/dataset and rerun

Reject candidates that only improve the training slice, trade precision for
unacceptable hallucinations, or win by exploiting evaluator blind spots.

### 5. Ship through the normal gate

If adopting:

- create a PR/proposal with the candidate text diff
- include the baseline-vs-candidate results in the body
- keep parser/schema/version bumps explicit when the artifact has one
- run the ordinary tests for that subsystem

Never write GEPA output directly into production prompts or protected memory.

## Good first targets

- **Commitment extraction prompt** — bounded artifact in
  `mimir/commitments/extractor.py`; chainlink #137 already provides a 30-session
  corpus and precision/recall / over-compression metrics.
- **Session-summary / unfinished extraction prompt** — high impact, but only after
  a labeled corpus and evaluator exist.
- **Tool descriptions** — useful when there is a probe set with known desired tool
  choices; target one tool or one small family, not the whole catalog.
- **Review skill prompt compliance** — plausible after isolating text behavior from
  harness/workflow failures; success must include actual `gh pr review` submission.

## Anti-target examples

- **Weekly reflection as a whole** — too broad and governance-heavy; reflection can
  recommend GEPA but should not become GEPA.
- **Core identity / persona / values blocks** — not local evaluator-backed prompt
  artifacts; narrow metrics would distort policy.
- **"Be less sycophantic" globally** — useful concern, bad GEPA target unless
  reduced to one artifact plus a labeled evaluation set.
- **Recall/simhi bench prompt tweaks** when scores are dominated by non-text knobs
  such as thresholds, triple toggles, and retrieval strategy.

## Pilot template

Copy this into the chainlink issue or spec for a real run:

```markdown
## GEPA pilot: <artifact>

- Target artifact: `<path>` / `<symbol>`
- Baseline commit/text: <sha or frozen block>
- Dataset / traces: `<path>`
- Evaluator command: `<command>`
- Primary metric: <precision/recall/etc.>
- ASI fields: <what diagnostic text must cite>
- Budget: max_metric_calls=<N>, wall-clock=<duration>
- Adoption threshold: <minimum holdout improvement and non-regression rules>
- Adoption gate: PR/proposal reviewed by <who/surface>
- Known blind spots: <metric limitations>
```

If you cannot fill this in, do not run GEPA yet.