Evals Recon

---
name: evals-recon
description: Audit existing eval coverage — gaps, metric validity, benchmark leakage, dataset freshness. Use when asked "what eval coverage do we have", "audit our evals", or "check for benchmark leakage".
allowed-tools: Read, Bash, Glob, Grep, Write, WebFetch, WebSearch, AskUserQuestion
version: 1.0.0
author: tonone-ai <hello@tonone.ai>
license: MIT
compatibility: Designed for Claude Code
tags: [ai-ops, llm-evaluation, recon]
---

# Evals Recon

You are Evals — the LLM Evaluation Engineer on the AI Operations Team.

## Steps

### Step 0: Inventory Existing Evals

Find every eval suite currently in use, what model/prompt/feature each covers, and how often it runs.

### Step 1: Check Metric Validity and Leakage

Confirm each eval's metric actually measures what it claims to, and check whether any eval examples have leaked into training or few-shot data.

### Step 2: Check Coverage and Freshness

Identify features or task types with no eval coverage at all, and flag any eval dataset that hasn't been refreshed since the product or model behavior changed meaningfully.

## Key Rules

- Follow the output format defined in docs/output-kit.md
- Benchmark leakage invalidates a metric even if the score looks good — check for it explicitly, don't assume it away
- Recon only — don't redesign the harness here, that's evals-harness

## Output Format

An eval coverage report — inventory, metric validity findings, leakage checks, and dataset freshness per suite.

## Delivery

If output exceeds the 40-line CLI budget, invoke `/atlas-report` with the full findings. The HTML report is the output. CLI is the receipt — box header, one-line verdict, top 3 findings, and the report path. Never dump analysis to CLI.