Ai Evaluation Patterns

---
name: ai-evaluation-patterns
description: |
  Guides expert-level ai evaluation patterns implementation: ai-ml and testing decision frameworks, production-ready patterns, and concrete templates for ai evaluation patterns workflows.
  Use when the user asks about ai evaluation patterns, ai evaluation patterns configuration, or ai-ml best practices for ai projects.
  Do NOT use when the user needs a different ai ml engineering capability -- check sibling skills in the ai ml engineering subcategory.
license: Apache-2.0
metadata:
  author: foundry-skills
  version: "1.0.0"
  tags: "ai-ml testing best-practices"
  category: "ai-machine-learning"
  subcategory: "ai-ml-engineering"
  depends: ""
  disclaimer: "none"
  difficulty: "advanced"
---
# AI Evaluation Patterns

## When to Use

**Use this skill when:**
- User asks how to measure the quality of an LLM-powered feature -- classification accuracy, generation fluency, RAG answer faithfulness, agent task completion rate
- User wants to build or improve an evaluation harness for a model before deploying to production
- User needs to detect quality regressions after a prompt change, model upgrade, or retrieval config change
- User is designing an A/B testing or shadow deployment strategy to compare two model versions safely in production
- User wants to implement LLM-as-judge scoring, human evaluation rubrics, or automated metric pipelines
- User needs to define what "good output" means for a system that produces open-ended text, structured data, or tool calls
- User is building a golden dataset or benchmark suite for a specific AI application domain
- User needs to set quality gates in CI/CD to prevent deploying a prompt regression

**Do NOT use this skill when:**
- User needs guidance on prompt engineering techniques themselves -- use the prompt engineering skill instead
- User is asking about fine-tuning or RLHF training pipelines -- those involve training-loop evaluation, which differs from inference evaluation
- User needs infrastructure for logging/observability only -- use the AI observability skill if one exists
- User wants to benchmark raw model capability (MMLU, HellaSwag, etc.) -- this skill covers application-level evaluation, not academic benchmarks
- User is asking about data labeling workflows at scale without a deployed model to evaluate -- that is a data annotation task

---

## Process

### 1. Define What You Are Evaluating

Before writing a single line of evaluation code, establish the evaluation contract.

- Identify the **task type** precisely: text generation, structured extraction, classification, retrieval-augmented answering, multi-step reasoning, or tool-calling agent behavior. Each type has different canonical metrics.
- Write down the **failure modes** you most care about. For a customer support bot: hallucinated policy details and offensive tone. For a code generator: syntactically invalid output and incorrect logic. List these before picking metrics -- metrics should be derived from failure modes, not the other way around.
- Define the **evaluation unit**: is it one LLM call, one multi-turn conversation, or one end-to-end workflow? Avoid evaluating sub-components in isolation if the user experience depends on the whole pipeline.
- Establish **acceptance thresholds** before any data is collected. Example: faithfulness >= 0.85, answer relevance >= 0.80, latency p95 < 3s, hallucination rate < 5%. Write these down in a shared doc so thresholds are not set post-hoc to fit current results.
- Assign an **evaluation owner** -- someone responsible for keeping golden datasets fresh and metric definitions stable.

### 2. Build Your Golden Dataset

The quality of your evaluation is bounded by the quality of your test dataset.

- Target a minimum of **50 examples per major use case category** for meaningful signal. Below 20 examples, variance from individual hard cases dominates. For high-stakes production evaluation, aim for 200-500 per category.
- Collect examples from **three sources**: (1) real user queries sampled from production logs after filtering PII, (2) adversarial examples constructed to probe known failure modes, and (3) synthetic examples generated by an LLM and then human-reviewed to fill coverage gaps.
- Store each example as a structured record: `{id, input, expected_output, category, difficulty, created_by, created_at, notes}`. Use semantic versioning on the dataset (v1.0, v1.1) and never mutate existing examples -- only add or deprecate.
- Label difficulty explicitly: `easy`, `medium`, `hard`. Track separate metrics per difficulty tier. A model that scores 90% overall but fails on all hard cases is not production-ready for those hard cases.
- For open-ended generation tasks, provide **reference outputs** that illustrate acceptable responses -- not single ground-truth answers. Multiple valid references improve LLM-as-judge calibration dramatically.
- Implement **dataset refresh triggers**: add examples whenever a production incident occurs, whenever a user complaint is logged, or when a new topic cluster emerges in query analytics. Stale golden datasets give false confidence.

### 3. Choose Your Metrics Toolkit

Select metrics based on task type and what failure modes you identified in Step 1.

**For RAG / Question Answering:**
- **Faithfulness** (0--1): Does the answer contain only claims that are grounded in the retrieved context? Measure using LLM-as-judge with a decompose-then-verify approach. Score = (grounded claims) / (total claims). RAGAS framework provides a production-ready implementation.
- **Answer Relevance** (0--1): Is the generated answer addressing the actual question? Use embedding cosine similarity between the question and the answer, or an LLM judge.
- **Context Precision** (0--1): Of the retrieved chunks, what fraction are actually relevant to answering the question? Signals retrieval quality.
- **Context Recall** (0--1): What fraction of the necessary information to answer the question was present in the retrieved chunks? Requires ground-truth reference answers.

**For Classification / Extraction:**
- **Precision, Recall, F1** per class -- always report per-class metrics, not macro averages only, because class imbalance hides real failures.
- **Confusion matrix analysis** -- look for systematic confusions between specific label pairs, not just overall accuracy.
- **Schema compliance rate** for structured extraction: what fraction of outputs parse successfully into the expected JSON schema? Target 99.5%+ in production.

**For Open-Ended Generation:**
- **Coherence** (1--5 Likert): Is the text logically structured and internally consistent?
- **Groundedness** (0--1): Are factual claims traceable to source material?
- **Tone compliance** (binary + categorical): Does output match the brand voice / style guide?
- **Instruction following rate**: Did the model follow all explicit constraints in the prompt? Count constraint violations per output.
- Avoid BLEU and ROUGE for production evaluation of LLM outputs -- they correlate poorly with human judgment for modern generation quality. Use them only for comparing near-identical text variations.

**For Agents / Tool-Calling:**
- **Task success rate**: Did the agent complete the user goal? Requires an oracle or environment simulation.
- **Tool call accuracy**: Precision/recall on which tools were invoked and with what arguments.
- **Step efficiency**: How many tool calls did it take vs. the minimum required? Track average steps per task category.
- **Trajectory correctness**: Was the reasoning chain to the final answer valid even if the final answer is correct? Correct answers via wrong paths are brittle.

### 4. Implement the Evaluation Pipeline

Build the evaluation runner as a first-class software component, not a one-off script.

- Structure the pipeline as: `load_dataset -> run_model -> score_outputs -> aggregate_metrics -> report`. Each stage should be independently testable and its outputs inspectable.
- Use **async execution** for LLM calls during evaluation. Run evaluations in parallel with a configurable concurrency limit (typically 10--20 concurrent requests for API-based models) to keep wall-clock time under 10 minutes for a 200-example suite.
- Cache model outputs aggressively. Use a content-addressed cache keyed on `hash(model_id + prompt_template_version + input)`. Re-running evaluation after only a metric change should cost zero API calls.
- For LLM-as-judge scoring, use a **separate, stronger model** than the model being evaluated. If you are evaluating GPT-4o-mini outputs, use GPT-4o or Claude 3.5 Sonnet as the judge. Never use the same model to judge itself -- it will exhibit self-preference bias.
- Implement **judge reliability checks**: run each example through the judge twice with temperature > 0 and measure agreement rate. Judges with < 85% self-agreement on your rubric indicate the rubric is ambiguous and needs clarification.
- Store all intermediate artifacts: the exact prompt sent to the model, the raw model response, the judge prompt, the judge response, and the parsed score. Without these artifacts, debugging score anomalies is impossible.
- Implement a `--sample N` flag for fast iteration (run on 20 random examples) and a `--full` flag for complete evaluation. Developers should be able to run a quick eval in under 2 minutes.

### 5. Design Your LLM-as-Judge Prompts

LLM-as-judge is the most scalable evaluation method but requires careful prompt engineering to be reliable.

- Use a **criterion-specific judge** rather than a single "overall quality" judge. Score faithfulness with one prompt, coherence with another. Composite rubrics in a single prompt reduce judge reliability because the model's attention is split.
- Provide a **scoring rubric with behavioral anchors**. Instead of "Score 1--5 for coherence," write: "Score 5 if the response is perfectly logical with clear transitions between ideas and no contradictions. Score 3 if the main point is clear but there are minor logical jumps. Score 1 if the response is internally contradictory or the main point is impossible to determine."
- Use **Chain-of-Thought reasoning** in the judge prompt: instruct the judge to reason before scoring. This improves calibration significantly. End the prompt with "First explain your reasoning in 2--3 sentences, then output SCORE: [1-5]". Parse the score from the final line.
- Force **structured output** from the judge: use JSON mode or function calling to get `{reasoning: string, score: number, flags: string[]}` rather than free text you have to parse. This eliminates parsing failures.
- Run a **calibration pass** before deploying a new judge: score 20--30 human-labeled examples with the judge and compute correlation with human scores. Accept judge prompts with Pearson r >= 0.75 or Cohen's kappa >= 0.60 vs. human raters.
- Implement **positional bias correction** for pairwise comparisons: run A-vs-B and B-vs-A and average the scores. Positional bias in LLM judges can be as large as 10--15 percentage points.

### 6. Run Regression Testing in CI/CD

Connect evaluation to your deployment pipeline so regressions are caught before they reach users.

- Define a **gating policy**: the pipeline fails if any metric drops more than X points from the baseline recorded for the current production prompt version. Common thresholds: faithfulness drop > 0.03, task success rate drop > 2 percentage points, schema compliance drop > 0.5%.
- Maintain a **metric history database** -- a simple append-only table: `{run_id, timestamp, git_commit, prompt_version, metric_name, metric_value, dataset_version}`. Query this for trend analysis and to understand when a regression was introduced.
- Run evaluations on **three dataset slices simultaneously**: the full golden set (comprehensive quality), the hard-examples subset (robustness), and the latest 20 production incidents (known failure modes). A change that improves overall metrics while degrading hard examples should be flagged for human review.
- Implement a **canary evaluation pattern**: when deploying a new prompt or model version, run both old and new on the same 50-example batch in the CI pipeline and compute a delta report. A table showing metric-by-metric changes (green for improvement, red for regression) is actionable. A single aggregate pass/fail is not.
- Integrate evaluation results into pull request comments automatically. Engineers should see the evaluation delta before merging, not after deploying.

### 7. Implement Human Evaluation Where Automated Metrics Are Insufficient

Some quality dimensions require human judgment and cannot be reliably automated.

- Use human evaluation for: brand tone compliance on sensitive customer-facing content, nuanced helpfulness judgments in complex conversations, evaluation of creative or subjective tasks, and calibration of new LLM judges.
- Implement a **dual-rater protocol** with disagreement resolution. Assign each example to two independent raters. Compute inter-rater reliability (Cohen's kappa or Fleiss' kappa). If kappa < 0.6, the rubric is ambiguous -- revise it before continuing. Resolve disagreements via a third rater or adjudication discussion.
- Use **relative preference scoring** (A is better than B) rather than absolute Likert scales whenever the goal is comparing two versions. Humans are much more reliable at relative comparisons than absolute ones. The Bradley-Terry model converts pairwise preferences into a single quality score.
- Schedule **human evaluation sprints** after every major model or prompt change and monthly for production systems. Do not rely on continuous human evaluation of every output -- it is not scalable and annotator fatigue degrades quality quickly.
- Track **human evaluation cost** explicitly: typical annotation rates are $0.10--$0.50 per example for crowdworkers and $1--$5 per example for domain experts. Budget accordingly. A 200-example expert evaluation for a medical application can cost $600--$1000.

### 8. Monitor Evaluation Metrics in Production

Offline evaluation on golden datasets is necessary but not sufficient. Production data distribution differs from your test set.

- Implement **online evaluation sampling**: score a random 5--10% of production requests using LLM-as-judge in near-real-time. This provides continuous quality signal without requiring ground truth labels.
- Build **anomaly detection on metric time series**: alert when a rolling 1-hour metric average drops more than 2 standard deviations below the 30-day baseline. This catches prompt injection attacks, sudden topic distribution shifts, and model provider degradations.
- Track the **"no-answer" or "refusal" rate** separately from quality metrics. A spike in refusals often precedes a quality degradation event and is easier to detect in real time.
- Implement **stratified production monitoring**: track metrics by query category, user segment, and input length bucket. System-wide averages hide quality problems that affect a specific user segment.
- Feed production samples that receive poor automated scores or negative user feedback into the golden dataset as new test cases. Close the loop between production monitoring and offline evaluation.

---

## Output Format

### Evaluation Configuration Document

```yaml
# eval_config.yaml
evaluation_id: customer-support-bot-v2
created_at: 2024-01-15
owner: ml-platform-team@company.com

task_type: rag_question_answering
model_under_evaluation:
  model_id: gpt-4o-mini
  prompt_template_version: v3.2.1
  temperature: 0.1

dataset:
  path: datasets/customer_support_golden_v4.jsonl
  version: 4.0
  total_examples: 312
  categories:
    - returns_policy: 85
    - shipping_questions: 92
    - account_issues: 71
    - product_information: 64

judge:
  model_id: gpt-4o
  temperature: 0.2
  rubric_version: v2.1

metrics:
  faithfulness:
    threshold: 0.85
    weight: 0.35
  answer_relevance:
    threshold: 0.80
    weight: 0.25
  context_precision:
    threshold: 0.75
    weight: 0.20
  instruction_following:
    threshold: 0.92
    weight: 0.20

gating_policy:
  fail_on_any_metric_below_threshold: true
  fail_on_regression_from_baseline_greater_than: 0.03
  require_human_review_if_delta_exceeds: 0.05
```

### Evaluation Results Report

```
╔══════════════════════════════════════════════════════════════╗
║  EVALUATION REPORT                                           ║
║  Run ID: eval-2024-01-15-14:32:09                            ║
║  Model: gpt-4o-mini | Prompt: v3.2.1 | Dataset: v4.0        ║
╠══════════════════════════════════════════════════════════════╣
║  OVERALL STATUS: ✓ PASS                                      ║
╠══════════════════════════════════════════════════════════════╣

METRIC SUMMARY
┌─────────────────────┬───────────┬───────────┬────────┬────────┐
│ Metric              │ Current   │ Baseline  │ Delta  │ Status │
├─────────────────────┼───────────┼───────────┼────────┼────────┤
│ Faithfulness        │ 0.891     │ 0.874     │ +0.017 │ ✓ PASS │
│ Answer Relevance    │ 0.843     │ 0.851     │ -0.008 │ ✓ PASS │
│ Context Precision   │ 0.779     │ 0.762     │ +0.017 │ ✓ PASS │
│ Instruction Follow  │ 0.948     │ 0.941     │ +0.007 │ ✓ PASS │
│ Schema Compliance   │ 0.997     │ 0.994     │ +0.003 │ ✓ PASS │
└─────────────────────┴───────────┴───────────┴────────┴────────┘

BREAKDOWN BY CATEGORY
┌──────────────────────┬──────┬──────────────┬────────────────┐
│ Category             │ N    │ Faithfulness │ Ans. Relevance │
├──────────────────────┼──────┼──────────────┼────────────────┤
│ returns_policy       │ 85   │ 0.923        │ 0.871          │
│ shipping_questions   │ 92   │ 0.881        │ 0.839          │
│ account_issues       │ 71   │ 0.854        │ 0.812          │  ← review
│ product_information  │ 64   │ 0.908        │ 0.851          │
└──────────────────────┴──────┴──────────────┴────────────────┘

BREAKDOWN BY DIFFICULTY
┌──────────────┬──────┬──────────────┬────────────────┐
│ Difficulty   │ N    │ Faithfulness │ Ans. Relevance │
├──────────────┼──────┼──────────────┼────────────────┤
│ easy         │ 142  │ 0.941        │ 0.893          │
│ medium       │ 118  │ 0.883        │ 0.836          │
│ hard         │ 52   │ 0.801        │ 0.771          │  ← monitor
└──────────────┴──────┴──────────────┴────────────────┘

FAILURE ANALYSIS (bottom 10 examples by faithfulness)
┌─────────┬──────────────────────────────┬────────────┬───────────────────┐
│ ID      │ Category                     │ Score      │ Failure Pattern   │
├─────────┼──────────────────────────────┼────────────┼───────────────────┤
│ ex-0041 │ account_issues               │ 0.51       │ date hallucination│
│ ex-0187 │ shipping_questions           │ 0.58       │ added policy info │
│ ex-0223 │ account_issues               │ 0.62       │ date hallucination│
└─────────┴──────────────────────────────┴────────────┴───────────────────┘

NOTE: 3 of 10 worst failures involve date hallucination in account_issues.
      Recommend adding date grounding instruction to prompt.

COST SUMMARY
  Judge API calls: 624 (312 examples × 2 metrics using LLM judge)
  Estimated cost: $0.87
  Wall clock time: 4m 22s (concurrency: 15)
```

### Per-Example Evaluation Record Schema

```python
@dataclass
class EvaluationRecord:
    # Identity
    example_id: str          # e.g. "ex-0041"
    run_id: str              # e.g. "eval-2024-01-15-14:32:09"
    timestamp: datetime

    # Input
    question: str
    retrieved_contexts: list[str]
    prompt_sent: str         # exact string sent to model

    # Output
    model_response: str
    response_latency_ms: int
    tokens_input: int
    tokens_output: int

    # Scores
    faithfulness_score: float        # 0.0-1.0
    faithfulness_reasoning: str      # judge chain-of-thought
    answer_relevance_score: float
    answer_relevance_reasoning: str
    instruction_following_score: float
    schema_compliant: bool

    # Metadata
    category: str
    difficulty: str          # easy | medium | hard
    failure_flags: list[str] # e.g. ["date_hallucination", "out_of_scope"]
```

---

## Rules

1. **Never set quality thresholds after seeing the results.** Thresholds must be defined before running evaluation. Post-hoc threshold setting is the most common form of evaluation fraud and produces false confidence.

2. **Never use the same model as both subject and judge.** GPT-4o-mini cannot reliably judge GPT-4o-mini outputs -- it exhibits self-preference bias of 5--15 percentage points. Always use a stronger or different model family as judge.

3. **Never evaluate a single aggregate metric in isolation.** A faithfulness score of 0.88 aggregated across categories hides a 0.54 on the most complex category. Always report per-category and per-difficulty breakdowns alongside the aggregate.

4. **Never skip schema compliance measurement for structured outputs.** An extraction system that produces valid JSON 97% of the time fails 30 times per 1000 calls -- each failure is a production bug. Track this separately from quality metrics and treat < 99% as a blocker.

5. **Always cache model outputs during evaluation runs.** Re-scoring the same inputs after changing only a metric definition should cost $0. Without caching, evaluation cost discourages frequent iteration, which causes quality problems to go undetected.

6. **Never evaluate without versioning both the dataset and the prompt template.** An evaluation score is only meaningful relative to a specific `(dataset_version, prompt_version, model_version)` triple. Without version tracking, you cannot determine what changed when metrics shift.

7. **Always compute inter-rater reliability before trusting human evaluation data.** Kappa < 0.6 means raters are applying the rubric inconsistently, and the resulting labels are not reliable. Fix the rubric and re-annotate rather than averaging noisy labels.

8. **Never use BLEU or ROUGE as primary quality metrics for open-ended LLM outputs.** These n-gram overlap metrics correlate with human judgment at r ≈ 0.2--0.4 for modern generation quality. They produce misleading rankings. Use them only for verifying near-exact format matches.

9. **Always include adversarial examples in the golden dataset.** A system that scores 95% on benign examples but has never been tested on prompt injection, ambiguous queries, or out-of-distribution inputs will fail unpredictably in production. At minimum, 15--20% of golden examples should be adversarial.

10. **Never treat evaluation as a one-time activity.** Model provider behavior drifts over time even with no changes on your side. Run the full evaluation suite on a weekly scheduled job against production to detect silent degradations. A production model that passed evaluation 3 months ago may be failing today.

---

## Edge Cases

### Model Provider Updates Change Behavior Silently

Major LLM API providers occasionally update model behavior without versioning the model ID. GPT-4o behavior on 2024-01-01 is not guaranteed to match 2024-06-01 even at the same temperature. This causes evaluation scores to drift over time with no change in your code.

**Handling:** Pin specific model snapshot versions wherever providers offer them (e.g., `gpt-4o-2024-08-06` instead of `gpt-4o`). Run a weekly scheduled evaluation against production on a 50-example sentinel set. Alert when faithfulness or task success drops > 0.05 from the week-prior baseline. When drift is detected, compare outputs on 10 specific examples side-by-side to characterize the behavioral change before updating prompts.

### Small Dataset -- High Variance on Rare Categories

When a task category has fewer than 30 examples in the golden dataset, a single hard example can swing the category metric by 3--5 percentage points. This causes the CI gate to fail or pass inconsistently based on which examples happen to be in a small sample.

**Handling:** Do not gate on categories with fewer than 30 examples. Instead, flag them as "monitored but not gating." Report confidence intervals alongside point estimates -- for N=20 with mean=0.85, the 95% confidence interval is approximately ±0.16, which means the metric is nearly meaningless for gating. Prioritize expanding those categories before adding them to the gating policy. Use Wilson score intervals for proportion metrics rather than normal approximation, as they are more accurate for small N.

### Conflicting Metrics Signal Different Optimal Prompts

A prompt change that improves faithfulness by 0.04 may decrease answer relevance by 0.03. This is especially common when adding strict grounding instructions -- the model becomes more faithful but more terse, reducing perceived helpfulness.

**Handling:** Use weighted composite scoring only when the metric weights have been validated by human preference data. Without that validation, present the delta table to a human decision-maker rather than computing a single score. For the faithfulness-vs-relevance tradeoff specifically, collect 20 human preference judgments on examples where the two versions diverge. The human preference distribution reveals which direction matters more to users than any automated metric.

### Evaluation Dataset Contamination from Production Data

When golden examples are sampled from production logs, there is a risk that similar inputs appear in the training data or few-shot examples of the model under test, inflating scores artificially. This is especially problematic when using an LLM-generated synthetic dataset for evaluation.

**Handling:** Maintain a strict temporal split: examples collected before a model or prompt version was deployed cannot be used to evaluate that version. Store `created_at` timestamps on all golden examples and filter by deployment date. For synthetic datasets, use a different model to generate evaluation examples than the one being evaluated. Run a semantic deduplication pass (cosine similarity > 0.92) between your golden dataset and your prompt's few-shot examples to detect leakage.

### LLM Judge Refuses to Score or Returns Malformed Output

When the content of the model-under-test output is toxic, off-topic, or triggers the judge model's safety filters, the judge may refuse to score or return a non-parseable response. Naive implementations crash or silently skip these examples.

**Handling:** Implement a three-tier fallback for judge failures: (1) retry with a simplified scoring prompt that wraps the content more carefully, (2) if retry fails, score as 0 on quality dimensions and 1 on a separate `judge_failure` flag, (3) log all judge failures with the full context for manual review. Judge failure rate should be tracked as its own metric -- if it exceeds 2% of examples, the judge prompt needs hardening. Never silently drop failed judge calls from the aggregate metric denominator.

### Evaluating Multi-Turn Conversations

Single-turn evaluation metrics are straightforward. Multi-turn conversations introduce dependencies between turns: a response that is faithful in isolation may be inconsistent with context established three turns ago.

**Handling:** Define the evaluation unit as the full conversation, not the individual turn. Add conversation-level metrics: `context_consistency` (do responses in later turns contradict earlier turns?) and `goal_progress` (is the conversation moving toward resolving the user's original intent?). For faithfulness in multi-turn contexts, concatenate all retrieved context across turns before running the faithfulness judge. Store the full conversation history alongside each evaluation record. A conversation-level evaluation suite typically needs to be 2--3x larger than a single-turn suite to achieve equivalent statistical power, because conversation outcomes have higher variance.

### Evaluation of Agent Trajectories

For agents that execute multi-step tool-calling plans, the final output is not the only thing to evaluate. An agent can arrive at a correct answer via a flawed reasoning chain that will fail on slightly different inputs.

**Handling:** Implement trajectory-level evaluation in addition to outcome evaluation. Record the complete sequence of tool calls, their arguments, and their results. Score: (1) `outcome_correctness` -- did the agent produce the right final answer?, (2) `tool_selection_efficiency` -- were unnecessary tool calls made? (compare to minimum-step reference), (3) `argument_correctness` -- were tool arguments well-formed and semantically correct?, (4) `error_recovery` -- when a tool call failed, did the agent handle it gracefully? Use a replay-based evaluation environment where tool calls are intercepted and their responses are controlled, allowing deterministic re-evaluation of the same trajectory.

---

## Example

**Input:** "We built a RAG-based legal research assistant that helps lawyers find relevant case law and summarizes holdings. We're getting complaints that the summaries sometimes contain claims that aren't in the source documents. How do we build an evaluation system to measure and fix this?"

**Output:**

The problem you're describing is a faithfulness failure -- the model is generating hallucinated legal claims not grounded in the retrieved case documents. This is a critical failure mode for a legal application. Here is a complete evaluation system to measure it, diagnose the root cause, and gate future changes against it.

---

### Step 1: Define the Evaluation Contract

The primary metric you need is **faithfulness**: the fraction of claims in the generated summary that are directly traceable to the retrieved source documents.

Acceptance threshold: **faithfulness >= 0.92** for a legal application. Legal errors cause professional liability. This is stricter than a general-purpose application's 0.85 threshold.

Secondary metrics to track alongside faithfulness:
- **Answer completeness** (0--1): Did the summary cover all material holdings in the source? (A summary can be faithful but miss critical holdings.)
- **Citation accuracy** (binary per claim): When the summary attributes a claim to a specific case, is the citation correct?
- **Schema compliance** (binary): Does the output match the expected `{summary: string, holdings: list[string], citations: list[string]}` structure?

---

### Step 2: Build the Golden Dataset

Collect 200 examples structured as:

```jsonl
{
  "id": "legal-ex-0041",
  "query": "What is the standard for preliminary injunctions in the Ninth Circuit?",
  "retrieved_cases": [
    {
      "case_name": "Winter v. Natural Resources Defense Council",
      "citation": "555 U.S. 7 (2008)",
      "relevant_excerpt": "A plaintiff seeking a preliminary injunction must establish that he is likely to succeed on the merits, that he is likely to suffer irreparable harm in the absence of preliminary relief, that the balance of equities tips in his favor, and that an injunction is in the public interest."
    }
  ],
  "reference_summary": "The Supreme Court established a four-factor test for preliminary injunctions: likelihood of success on the merits, likelihood of irreparable harm absent relief, balance of equities, and public interest. All four factors must be weighed.",
  "category": "injunctive_relief",
  "difficulty": "medium",
  "known_failure_risk": "model may add 'serious questions' sliding scale from Alliance for Wild Rockies without it appearing in retrieved context"
}
```

Distribution:
- 50 examples where the retrieved context fully supports the answer (testing faithfulness on easy cases)
- 80 examples where the retrieved context is incomplete and the model must acknowledge gaps (testing hallucination under uncertainty)
- 40 examples with conflicting holdings across retrieved cases (testing whether model fabricates a false synthesis)
- 30 adversarial examples where a plausible-but-wrong legal claim could be inserted (testing robustness)

---

### Step 3: Implement the Faithfulness Judge

The faithfulness evaluation uses a decompose-then-verify approach:

**Judge Prompt -- Stage 1 (Decompose):**
```
You are a legal text analysis assistant. Given the following case law summary,
extract every factual claim as a separate statement. A claim is any assertion
about what a court held, what a legal standard requires, or what a precedent
established.

Summary: {summary}

Output a JSON array of claims:
{"claims": ["claim 1", "claim 2", ...]}
```

**Judge Prompt -- Stage 2 (Verify each claim):**
```
You are a legal accuracy checker. Determine whether the following claim is
directly supported by the provided source documents.

Claim: {claim}

Source Documents:
{retrieved_context}

A claim is SUPPORTED if the exact information appears in the source documents.
A claim is UNSUPPORTED if it requires information not present in the source documents,
even if the claim is legally accurate in general.

Output:
{"verdict": "SUPPORTED" | "UNSUPPORTED", "evidence": "quote from source or 'not found'", "reasoning": "2-sentence explanation"}
```

**Faithfulness Score Computation:**
```python
def compute_faithfulness(claims: list[dict]) -> float:
    supported = sum(1 for c in claims if c["verdict"] == "SUPPORTED")
    return supported / len(claims) if claims else 0.0
```

Use GPT-4o as the judge (not GPT-4o-mini, which is the model under test). Legal text requires careful reasoning -- do not economize on the judge.

---

### Step 4: Run the Baseline Evaluation

First run reveals the baseline faithfulness distribution:

```
BASELINE EVALUATION RESULTS
Run: legal-rag-baseline-2024-01-15
Dataset: legal-golden-v1.0 (N=200)

Faithfulness:          0.743  ← BELOW THRESHOLD (0.92)
Answer Completeness:   0.821
Schema Compliance:     0.994

FAITHFULNESS BY CATEGORY:
  injunctive_relief:     0.791
  contract_disputes:     0.758
  constitutional_law:    0.701  ← worst
  procedural_rules:      0.762

FAILURE PATTERN ANALYSIS (from 127 unsupported claims):
  47% -- model adds related doctrine not in retrieved context
  31% -- model extrapolates a holding beyond what the case states
  22% -- model confuses holdings from two different cases
```

The failure pattern analysis tells you where to focus prompt engineering: the dominant failure is the model adding related doctrine it "knows" but that isn't in the retrieved documents.

---

### Step 5: Fix and Verify

Add a grounding constraint to the system prompt:

```
CRITICAL INSTRUCTION: You are a legal research assistant operating in strict
grounding mode. You may ONLY include claims in your summary that are explicitly
stated in the provided case excerpts. Do NOT draw on your general legal knowledge
to supplement the summaries. If the retrieved documents do not contain enough
information to answer the query fully, state explicitly what information is missing
rather than filling the gap from your training data.
```

Re-run evaluation. Expected result after this change:

```
POST-FIX EVALUATION RESULTS

Faithfulness:          0.891  (+0.148 vs baseline)
Answer Completeness:   0.784  (-0.037 vs baseline)  ← expected tradeoff
Schema Compliance:     0.997

STATUS: Faithfulness improved significantly but still below 0.92 threshold.
        Completeness dropped as expected -- model is now appropriately
        declining to summarize claims not in context.
```

Faithfulness improved but is still below the 0.92 threshold. Iterate on the retrieval strategy -- context precision is 0.71, meaning 29% of retrieved chunks are noise that confuses the model. Improving retrieval quality is likely the next highest-leverage change.

---

### Step 6: Add to CI/CD

```yaml
# .github/workflows/eval.yml
name: Legal RAG Evaluation Gate

on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'retrieval/**'
      - 'models/**'

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - name: Run evaluation suite
        run: |
          python eval/run_evaluation.py \
            --config eval/legal_rag_config.yaml \
            --sample 50 \
            --baseline-run-id ${{ vars.BASELINE_RUN_ID }}

      - name: Post results to PR
        uses: actions/github-script@v7
        with:
          script: |
            const results = JSON.parse(fs.readFileSync('eval_results.json'));
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              body: results.pr_comment_markdown
            });

      - name: Enforce quality gate
        run: |
          python eval/check_gate.py \
            --results eval_results.json \
            --fail-if faithfulness < 0.92 \
            --fail-if regression > 0.03
```

Any PR that changes prompts, retrieval config, or model selection now automatically runs a 50-example fast evaluation and posts a delta table as a PR comment. The full 200-example suite runs nightly.

---

### Expected Final State After Full Implementation

Within 4--6 weeks of systematic evaluation-driven development:
- Faithfulness target of 0.92 is achievable through combined prompt grounding instruction + retrieval precision improvements (targeting context precision >= 0.85)
- Every prompt change ships with a delta report showing faithfulness impact before merge
- Production monitoring catches regressions within 24 hours via online evaluation on 5% of live traffic
- Golden dataset grows by 10--15 examples per week from production incidents, maintaining evaluation relevance as query distribution evolves