Prompt Eval And Regression

---
name: prompt-eval-and-regression
description: "Build the eval/regression set that gates prompt changes — labeled input/expected pairs over the hard cases, a scoring method (exact / schema-valid / rubric / LLM-judge with its caveat), a pass threshold, a CI gate with the model pinned, and injection cases. Reach for this before shipping a prompt, when a tweak silently broke other cases, or to defend against prompt injection. Pairs with structured-output-design."
---

# Skill: Prompt eval & regression

Turn "it worked when I tried it" into "it is verified to still work." A prompt is
**unverified until the regression set is green.**

## Step 0 — One opinion up front
**Prompts are code.** They live in version control, change in reviewable diffs, and
must pass an eval before merge — exactly like any other code.

## Step 1 — Build the regression set
Collect labeled `input → expected` pairs covering:
- The **known-hard/edge cases**.
- **Every past failure** (each production bug becomes a permanent test).
- **Injection/jailbreak cases** (see §4 of the decision-trees doc).
Keep it in the repo next to the prompt.

## Step 2 — Choose a scoring method
Match the method to the task:
- **Exact / schema-valid** — closed-form or structured tasks. Cheapest, most
  reproducible; prefer it wherever possible.
- **Rubric** — explicit criteria, human or model graded.
- **LLM-as-judge** — scalable but the judge is a fallible prompt. **Judge the
  judge:** validate against human labels first; never let a model grade its own
  output unaudited; watch for position/verbosity/self-preference bias.
Set a **pass threshold** a change must clear.

## Step 3 — Pin for reproducibility
Pin the **model + version + temperature (+ seed where available)**. Handle
nondeterminism: temperature 0 where the task allows; multiple samples + a tolerance
where it doesn't. An eval against an unpinned model proves nothing tomorrow.

## Step 4 — Wire the CI gate
Run the regression set on every change to a prompt file; **fail the build on a
regression.** State honestly what the set does and doesn't cover — a green gate
over a thin set is false safety.

## Step 5 — Version & roll out
Prompts carry a version (semver or content hash). Roll changes out shadow → canary
→ full, with a watched metric and a rollback trigger.

## Step 6 — Hand off
- The **large offline eval program / benchmark design** → `llm-evaluation-engineering`.
- **Adversarial attacks on the running system** → `ai-red-teaming`.
- The **fix** for what the evals catch → `prompt-implementation-engineer` / `prompt-architect`.

## Output
A regression set + scoring method + threshold, a CI gate with the model pinned,
injection cases included, an honest coverage statement, and a versioning + canary
rollout plan.