Prompt Diff

---
name: prompt-diff
description: Compare two versions of the system prompt (or any other
  prompt) across the golden case set and produce a per-case, per-metric
  diff. Use when the user is A/B-testing a prompt change before merging.
---

# prompt-diff

Purpose: give the user evidence — not vibes — that prompt A is better
than prompt B for the workload the evals actually cover.

## When to fire

- The user says: "which prompt is better", "A/B this prompt", "diff
  prompts", "compare prompt versions", "should I use prompt X or Y".
- The user pasted two prompt strings, or pointed at two prompt files, or
  is comparing HEAD vs a previous commit of `./agent/prompts/system.ts`.

## Inputs you need

Before starting, confirm:

1. **Prompt A** — the baseline. Usually the current `SYSTEM_PROMPT`
   export from `./agent/prompts/system.ts`.
2. **Prompt B** — the candidate. Either a string the user provided or
   another commit / file to read.
3. **Case set** — default to everything under `./evals/cases/`. If the
   user wants a subset, filter by `tags` on each case.

Ask exactly one clarifying question if any input is missing.

## Steps

1. **Save both prompts** into temp files under `eval-results/prompts/`:
   - `prompt.A.txt`
   - `prompt.B.txt`
   Keep them in the repo tree so future runs can be re-diffed.

2. **Run the suite twice**, once per prompt. The runner reads
   `SYSTEM_PROMPT` at import time, so patch the import via an env
   override rather than editing the file. Use this pattern:
   ```bash
   AGENT_SYSTEM_PROMPT_FILE=eval-results/prompts/prompt.A.txt \
     npm run eval:ci
   mv eval-results/summary.json eval-results/summary.A.json

   AGENT_SYSTEM_PROMPT_FILE=eval-results/prompts/prompt.B.txt \
     npm run eval:ci
   mv eval-results/summary.json eval-results/summary.B.json
   ```
   (If the env override isn't wired yet, tell the user — do NOT edit
   `system.ts` and commit it silently.)

3. **Produce the diff.** For each case ID, report:
   - Pass status: A ✓ vs B ✗ (regression), A ✗ vs B ✓ (improvement),
     equal (skip in report).
   - Step count delta.
   - Duration delta (informational only — don't optimize on it).
   - Tool-call set delta (did B start using or stop using a tool?).

4. **Summarize** in this shape:
   ```
   Prompt A: <passed>/<total>   avg steps <n>
   Prompt B: <passed>/<total>   avg steps <n>

   B wins on:      <case ids>
   A wins on:      <case ids>
   No difference:  <count>
   ```

5. **Recommend.** State a recommendation (adopt B, keep A, mixed —
   need more cases) and back it with the numbers, not adjectives.

## Definition of done

- Both prompts stored in `eval-results/prompts/` for reproducibility.
- Both summary JSONs kept, not overwritten.
- Report includes at least one concrete case per "wins on" bucket, with
  the failure message quoted.
- You did NOT overwrite `./agent/prompts/system.ts` without an explicit
  user instruction.