Graph Eval

---
name: graph-eval
description: Offline, code-gated A/B harness that ranks candidate finder briefs by
  how many synthetic planted defects each catches — a relative prompt-selection
  signal, never a recall estimate. Off the critical path.
argument-hint: <task-class>
disable-model-invocation: true
---

# Graph-eval: pick the better brief with a code-measured A/B, honestly

Rank candidate finder briefs (or DAG shapes) for a task class by **relative**
performance on style-matched synthetic plants. This is a disciplined prompt-A/B
harness — **not** a categorical capability and **not** a measurement of how well
`/graph` will do on the user's real repo. Run it offline; never on the critical
path of a real goal.

Task class: **$ARGUMENTS**

## What this is (and the honest limits, which are load-bearing)

- Output is: *"brief A caught more synthetic plants than brief B — a relative
  selection signal."* It is **never** "brief A has X% recall on your repo."
  Correlated same-model finders with heterogeneous catchability cannot estimate
  their own real-world recall; do not compute Lincoln-Petersen / capture-recapture
  bands — they launder a wide-variance guess as an instrument reading.
- Because plants are synthetic, this is Goodhart-prone. The guardrails below are
  the point of the skill, not decoration.

## Procedure

1. **Mutually-blind plant / hunt split.** A **plant** agent (spawn with
   `isolation: 'worktree'`) injects K defects into a copy of the target and returns
   an **answer key that only orchestrating code ever sees** — the key is never
   concatenated into any finder/hunter prompt. Require **K ≥ 30** so a win clears
   noise.
2. **Held-out archetype split (anti-Goodhart).** Keep an `escape_catalog` of defect
   archetypes. The archetypes used to *test* a brief are **never** the archetypes
   named in that brief. A brief that only finds what it was told to look for is not
   better; it's overfit.
3. **Run each candidate brief** as a fan-out over the planted copy (same seed set,
   same budget). Each hunter returns found locations; **code** scores them against
   the held answer key. No agent grades itself.
4. **Promotion is code-gated.** Compute the delta (plants caught A − B) in code.
   Promote a champion only if the delta is positive across the K ≥ 30 plants and
   the held-out archetypes — never on an agent's self-report. Ties or noise-level
   deltas promote nothing.
5. **Persist the champion** to `.claude/graph-memory/<task-class>.md` (a
   git-committable brief), with a one-line record of the measured delta and the
   plant count. The `/graph` skill may read this file as a *prior* for that task
   class — clearly labeled as a synthetic-plant selection signal, not a guarantee.

## Output

A short report: candidate briefs ranked by code-measured plants-caught, the delta,
K, the held-out archetypes used, and the promoted champion path (or "no promotion —
delta within noise"). State plainly: *"relative prompt-selection on synthetic
plants; not a recall figure for any real repo."*