Ai Evaluation

---
name: ai-evaluation
description: Use when building an AI/LLM evaluation harness, golden datasets, quality metrics, judge calibration, regression gates, production monitoring, drift detection, or user-feedback loops; use ai-llm-integration for provider calls.
metadata:
  portable: true
  compatible_with:
  - claude-code
  - codex
---

# AI Evaluation and Monitoring

## Operating contract

## Inputs

| Input | Required | Purpose |
|---|---|---|
| Domain evidence | yes | feature contract, representative dataset, quality dimensions, baseline, thresholds, and risk class |

## Outputs

- Produce: evaluation plan, scored results, confidence and failure slices, release verdict, and monitoring thresholds.

## Capability and permission boundaries

Default to read-only analysis. Read only scoped records; redact secrets and regulated data. Writes, execution, network calls, production configuration, customer communication, billing changes, and delegation require explicit authority and an identified owner. Never widen tenant, time-window, or system scope implicitly.

## Degraded mode

When required telemetry, evidence, execution, network access, or write authority is unavailable, return a partial result with each unassessed item labelled, preserve the safest existing state, and state the evidence or approval needed to continue. Never convert missing evidence into a pass.

## Decision rules

| Condition | Action |
|---|---|
| Scope, owner, or threshold is missing | Stop the affected decision and request it |
| Evidence is incomplete but read-only analysis is safe | Produce a qualified partial result and gap list |
| A mutation exceeds authority or tenant boundary | Block it and route for approval |
| Evidence meets the stated threshold | Issue the output with provenance and owner |

## Anti-Patterns

- Treating absent evidence as success. Fix: mark the check unassessed and name the missing source.
- Expanding one tenant or workflow to all tenants. Fix: enforce supplied scope at every query and action.
- Performing a production write during analysis. Fix: emit a reviewed change plan until authority is explicit.
- Reporting a metric without population, window, or source. Fix: attach all three.
- Hiding a failed threshold inside an average. Fix: report failure slices and the remediation owner.

Acknowledgement: Shared by Peter Bamuhigire, techguypeter.com, +256 784 464178.

<!-- dual-compat-start -->
## Use When

- Use when setting up quality assurance for AI features — defining evaluation criteria, measuring output quality, using AI-as-judge, monitoring production AI, detecting drift, and building user feedback loops

## Evidence Produced

| Category | Artifact | Format | Example |
|----------|----------|--------|---------|
| Correctness | AI evaluation harness configuration | Markdown doc covering eval set, scoring rubric, and acceptance thresholds | `docs/ai/eval-harness-assistant.md` |
| Correctness | Latest AI evaluation results | Markdown doc reporting pass-rate, regression deltas, and outlier examples | `docs/ai/eval-results-2026-04-16.md` |

## References

- Use the links and companion skills already referenced in this file when deeper context is needed.
- Load `ai-llm-integration/references/llm-application-pattern-decision-catalogue.md` (section 4) when the task is the eval-driven development loop or reusing evals as runtime guardrails.
<!-- dual-compat-end -->
## Overview

Evaluation is the biggest bottleneck to successful AI deployment. Define evaluation criteria BEFORE building. Without evaluation, you cannot know if your AI feature is working, degrading, or harming users.

**Core principle:** Evaluation-driven development. Like TDD for AI — define what "good" means first, then build.

## Evaluation Contract

Every production AI feature needs an evaluation contract before release:

- **Task definition**: what the model must do, for whom, in which workflow.
- **Business metric**: time saved, revenue influenced, conversion lift, defect reduction, service quality, risk reduction, or cost avoided.
- **Quality metrics**: correctness, completeness, relevance, tone, actionability, citation quality, and domain compliance.
- **System metrics**: latency, cost per successful task, availability, retry rate, fallback rate, and tool failure rate.
- **Safety metrics**: PII exposure, prompt-injection resistance, harmful output, policy violations, and unauthorized action attempts.
- **Release threshold**: minimum pass rates and maximum regression allowed versus the current production baseline.
- **Rollback trigger**: metrics or incidents that require disabling the feature or reverting prompt/model/tool versions.

---

## Evaluation Dimensions

| Dimension | What to Measure | Method |
|---|---|---|
| **Format** | Is output valid JSON/schema? Correct length? | Automated rules |
| **Factual accuracy** | Does output match the provided context? | AI-as-judge or RAG citation check |
| **Safety** | Toxic, harmful, or brand-risk content? | Classifier or AI-as-judge |
| **Instruction-following** | Did it follow format/tone/language rules? | Automated + AI-as-judge |
| **Relevance** | Does output address the user's question? | AI-as-judge |
| **Cost** | Tokens per request; cost per feature | Logged automatically |
| **Latency** | Time to first token; total response time | Logged automatically |

---

## Evaluation Workflow

### Prompt and agent evaluation addendum

Version the prompt builder, model identifier, context assembly, tool policy,
and evaluation data as one change surface. For each revision, record the
objective, hard constraints, expected output, risk class, and acceptance
grader. Compare against the previous version on representative, edge,
adversarial, multilingual, and low-context cases; report per-slice failures,
not only the aggregate score. Include a regression case for every production
incident. A successful demo or syntax check is not release evidence for an AI
feature.

For agentic workflows, evaluate the whole loop: context selection, plan
quality, tool-argument validity, authorization/approval, side effects,
recovery, final output, latency, cost, and audit trace. Keep deterministic
format/schema/security checks separate from model-judged quality checks. If a
grader, test set, runtime, or account model is unavailable, mark that result
`NOT ASSESSED` rather than treating absence as a pass.

```
1. Define criteria before building
2. Create golden test set (20–50 examples with expected outputs)
3. Run automated format checks on every new model/prompt version
4. Run AI-as-judge for quality checks
5. Compare against previous version — only deploy if metrics hold or improve
6. Monitor production: track live metrics + user feedback
7. Retrain/reprompt when drift detected
```

For features that recommend, generate, rank, automate, or trigger consequential
actions, load [references/ai-human-oversight-evaluation.md](references/ai-human-oversight-evaluation.md).
Evaluate the problem, human, system, model, inputs, and outputs separately. Do not
call a model-quality pass a product-quality pass. Require evidence for correction,
contest, undo or safe fallback, escalation, consent/notice, and the audit event.

When a model, score, rule set, or ranking decides or materially shapes an outcome
for a person (credit, hiring, eligibility, pricing, visibility, risk flags), load
[references/algorithmic-decision-impact-review.md](references/algorithmic-decision-impact-review.md)
for the impact tier, proxy and feedback-loop checks, subgroup error evidence,
fairness-criterion ownership, and contest path required before release.

## Evaluation Dataset Design

- Build separate sets for development, release gating, adversarial testing, and production monitoring.
- Include real examples, expert-created examples, edge cases, multilingual/local examples, low-resource context, and abuse attempts.
- Version each case with task, tenant/domain, source, expected behavior, rubric, and data sensitivity.
- Keep leakage out of evals: do not let the same generated examples drive both prompt tuning and final release scoring.
- Refresh cases after production incidents, new user behavior, platform changes, policy changes, or model migrations.

## Release Gates

| Gate | Required Evidence |
|---|---|
| Prompt/model change | Eval comparison against previous version, cost and latency delta |
| New tool/action | Permission tests, dry-run tests, approval-path tests, failure-mode tests |
| RAG/index change | Retrieval precision samples, citation checks, stale-data checks, tenant isolation checks |
| Fine-tune | Baseline comparison showing prompts/RAG were insufficient, holdout eval results, rollback plan |
| Production launch | Monitoring dashboard, alert thresholds, owner, incident playbook, user feedback channel |

Do not ship an AI feature because a demo looked good. Ship it because it passes the evaluation contract.

---

## Creating a Golden Test Set

```sql
CREATE TABLE ai_eval_cases (
  id              INT AUTO_INCREMENT PRIMARY KEY,
  feature_name    VARCHAR(100),       -- 'invoice_analysis', 'sales_report'
  input           TEXT NOT NULL,      -- the user query or document
  expected_output TEXT,               -- ideal output (or key elements of it)
  eval_criteria   JSON,               -- {"format": "json", "must_contain": ["total", "vendor"]}
  created_by      INT,
  created_at      TIMESTAMP DEFAULT NOW()
);
```

Build test cases from:
1. Real production queries (once live)
2. Domain expert-crafted examples
3. Edge cases: empty input, wrong language, very long input, adversarial input

---

## Automated Evaluation (No LLM Cost)

Run these on every deployment:

```php
class AiEvaluator {
    public function evaluateFormat(string $output, array $criteria): EvalResult {
        $score = 0;
        $issues = [];

        // JSON validity
        if (($criteria['format'] ?? null) === 'json') {
            $decoded = json_decode($output, true);
            if (json_last_error() !== JSON_ERROR_NONE) {
                $issues[] = 'Invalid JSON';
            } else {
                $score += 25;
                // Required keys
                foreach ($criteria['required_keys'] ?? [] as $key) {
                    if (!array_key_exists($key, $decoded)) {
                        $issues[] = "Missing key: $key";
                    } else {
                        $score += 10;
                    }
                }
            }
        }

        // Length constraints
        if (isset($criteria['max_words'])) {
            $wordCount = str_word_count($output);
            if ($wordCount > $criteria['max_words']) {
                $issues[] = "Too long: {$wordCount} words (max {$criteria['max_words']})";
            } else {
                $score += 15;
            }
        }

        // Must-contain terms
        foreach ($criteria['must_contain'] ?? [] as $term) {
            if (stripos($output, $term) === false) {
                $issues[] = "Missing expected term: $term";
            } else {
                $score += 10;
            }
        }

        return new EvalResult($score, $issues);
    }
}
```

---

## AI-as-Judge

Use a strong model to evaluate your AI feature's outputs. Reliable for quality, relevance, and tone.

```php
function judgeAiOutput(string $input, string $output, string $criteria): array {
    $judgePrompt = <<<PROMPT
You are evaluating the quality of an AI assistant's response.

Evaluation criteria:
{$criteria}

User input:
---
{$input}
---

AI response to evaluate:
---
{$output}
---

Score the response on each criterion from 1–5 (5 = excellent).
Explain your reasoning briefly, then give an overall score (1–5).
Format your response as JSON:
{
  "relevance": {"score": X, "reason": "..."},
  "accuracy": {"score": X, "reason": "..."},
  "tone": {"score": X, "reason": "..."},
  "overall": X
}
PROMPT;

    return callLLM('gpt-4o', $judgePrompt, temperature: 0.1);
}
```

**AI-judge best practices:**
- Use a stronger model as judge than the one being evaluated
- Ask for reasoning BEFORE score (reduces positional bias)
- Use pairwise comparison (A vs B) for relative quality rather than absolute scores
- Multiple judges + average for high-stakes decisions
- Watch for self-serving bias (GPT-4 favours GPT-4 outputs)

---

## Production Monitoring

### Metrics to Track Per Feature

```sql
CREATE TABLE ai_quality_metrics (
  id             BIGINT AUTO_INCREMENT PRIMARY KEY,
  tenant_id      INT NOT NULL,
  feature_name   VARCHAR(100),
  prompt_version VARCHAR(10),
  model          VARCHAR(50),
  format_valid   BOOLEAN,
  latency_ms     INT,
  tokens_in      INT,
  tokens_out     INT,
  judge_score    DECIMAL(3,2),         -- 1.00–5.00 from AI judge (async)
  user_rating    TINYINT,              -- 1–5 from explicit feedback
  thumbs_up      BOOLEAN,              -- quick user feedback
  created_at     TIMESTAMP DEFAULT NOW(),
  INDEX idx_feature_date (feature_name, created_at),
  INDEX idx_tenant_date (tenant_id, created_at)
);
```

### Key Metrics by Priority

1. **Format failure rate** — % of responses failing JSON/schema validation
2. **User thumbs-down rate** — explicit negative feedback
3. **Early termination rate** — user stops generation mid-way
4. **Average judge score** — from async AI-as-judge on random sample
5. **p50/p90/p99 latency** — track at percentiles, not average
6. **Cost per request** — tokens × price per token

### Alerting Thresholds

```php
$alerts = [
    'format_failure_rate' => 0.05,    // Alert if > 5% of responses fail format
    'thumbs_down_rate'    => 0.15,    // Alert if > 15% negative feedback
    'p99_latency_ms'      => 8000,    // Alert if p99 latency > 8 seconds
    'cost_per_request'    => 0.05,    // Alert if avg cost > $0.05 per request
];
```

---

## Drift Detection

Drift = your AI feature is silently getting worse. Causes:
1. **Model API updates** — providers silently update model versions
2. **System prompt edits** — even small changes change behaviour
3. **User behaviour shift** — users learn to write differently over time
4. **Data drift** — RAG documents become stale

### Detection

```sql
-- Weekly average quality score — watch for downward trend
SELECT
    YEARWEEK(created_at) AS week,
    feature_name,
    AVG(judge_score) AS avg_quality,
    AVG(CASE WHEN thumbs_up = FALSE THEN 1 ELSE 0 END) AS negative_rate,
    AVG(latency_ms) AS avg_latency
FROM ai_quality_metrics
WHERE tenant_id = ? AND created_at > DATE_SUB(NOW(), INTERVAL 8 WEEK)
GROUP BY week, feature_name
ORDER BY week;
```

**Act when:**
- Average quality score drops > 0.5 points vs last 4-week average
- Format failure rate doubles vs baseline
- User negative feedback rate increases > 5% week-over-week

---

## User Feedback Signals

| Signal | Type | Strength |
|---|---|---|
| Thumbs up / down | Explicit | Medium |
| Star rating | Explicit | Medium |
| "That's wrong" in chat | Implicit | High |
| User edits output | Implicit | Very high |
| Early generation stop | Implicit | Medium |
| Rephrases same question | Implicit | High |
| Regenerates response | Implicit | Medium |

**Collect user edits as preference data:** original output = rejected, edited version = preferred.

---

## Evaluation Before vs After Deployment

| Phase | What to Evaluate | How |
|---|---|---|
| **Pre-deploy** | New prompt version vs old | A/B on golden test set |
| **Pre-deploy** | New model vs old | Same test set, compare scores |
| **Post-deploy** | Production quality | Sample 5% of requests → AI judge |
| **Post-deploy** | User satisfaction | Feedback collection |
| **Ongoing** | Drift detection | Weekly metric trend |

**Never deploy a new prompt or model without running the golden test set first.**

---

## Anti-Patterns

- **No golden test set** — you cannot measure regression
- **Only measuring average latency** — track p90/p99; outliers hurt users
- **Skipping evaluation to ship faster** — silent quality degradation is worse than delay
- **No prompt versioning** — you cannot roll back a broken prompt
- **Judge uses same model as evaluated** — self-serving bias inflates scores
- **No user feedback mechanism** — your most valuable signal goes uncollected

---

## Sources
Chip Huyen — *AI Engineering* (2025) Ch.3–4,10; Chip Huyen — *Designing ML Systems* (2022) Ch.8

## Multi-Tenant Eval Harness

This skill covers evaluation concepts. The engineering of the eval harness as a control-plane service in a multi-tenant SaaS — per-tenant golden datasets, CI gate that blocks regressing prompt/model changes, judge-LLM calibration, production sampling, drift detection, and pre-promotion gates for flagship tenants — lives in `ai-eval-harness`.

Cross-references:
- `ai-eval-harness` — the harness as a service.
- `ai-hallucination-slo-and-grounding` — uses eval signals to budget hallucination.
- `ai-feature-rollout-and-experimentation` — gates rollouts on eval.
- `ai-on-saas-architecture` — control-plane positioning.
## Consolidated Child References

- Load [references/routing.md](references/routing.md) to map retired AI child skill slugs to their reference modules.