Evaluate And Harden Agent

---
name: evaluate-and-harden-agent
description: "Evaluate an agent and harden its failure modes before it touches real traffic — an offline eval harness scored on task-completion, trajectory, and tool-use correctness against a fixed task set, plus loop hardening (step/tool-call caps, timeouts + retries, stop conditions, human-in-the-loop on irreversible actions) and tracing so every step/tool-call/token-cost is observable, with cost and latency reported alongside quality. Reach for this when the user asks 'how do I know this agent works?', 'set up agent evals', or 'the agent loops / does something dangerous'. Used by `agent-implementation-engineer` (primary)."
---

# Skill: evaluate-and-harden-agent

> **Invoked by:** `agent-implementation-engineer` (primary). Also consulted by `agentic-systems-architect` when a go/no-go needs an evidence bar defined before build.
>
> **When to invoke:** "how do I know this agent actually works?"; "set up evals for the agent"; "the agent loops forever / burns tokens"; "the agent did something irreversible it shouldn't have"; "it works in the demo but fails in the tail".
>
> **Output:** an offline agent-eval harness (task-completion + trajectory + tool-use, over a fixed task set) and a hardening plan (caps, timeouts/retries, stop conditions, human-in-the-loop, tracing) — with cost and latency reported next to quality.

## Procedure

1. **Build a fixed task set before scoring anything.** Collect representative tasks with known-good outcomes (start with real or realistic inputs, include the hard/tail cases, not just the happy path). This set is the regression harness — freeze it and grow it as failures surface.
2. **Score three dimensions, not one.**
   - **Task-completion:** did the agent achieve the goal? (the outcome, judged against the known-good result — exact-match, rubric, or LLM-judge as fits.)
   - **Trajectory:** did it take a *sane path*, or flail and get lucky? Check step count, redundant/looping calls, and whether the plan was coherent — a right answer via a 40-step random walk is a latent failure.
   - **Tool-use correctness:** right tools, right arguments, recovered from errors? Wrong-tool and bad-argument rates are the leading indicators of agent quality.
3. **Run it offline against fixtures/mocks first.** Mock the tools/systems so the eval is deterministic, free, and fast — catch regressions before spending on live calls. Only after offline is green do you evaluate against live dependencies.
4. **Report cost and latency alongside quality — always.** Per-task **token cost** and **wall-clock**, plus the distribution (the p95, not just the mean). A correct agent that costs $2 and takes 90s per run may be unshippable; the number needs a price tag.
5. **Harden the loop against its failure modes.**
   - **Caps:** a hard **step/tool-call limit** so a stuck agent stops instead of billing forever.
   - **Timeouts + retries:** per-tool timeout with bounded backoff; a hung tool must not hang the agent.
   - **Stop conditions:** explicit goal-reached / no-progress / cap-hit exits.
   - **Human-in-the-loop:** a confirmation gate on **every irreversible or high-blast action** (send, pay, delete, write-to-prod) — the loop pauses for approval.
   - **Tracing:** every step, tool call, argument, result, and token cost recorded, so any failure is replayable.
6. **Feed production back into the eval set.** Treat early live runs as data — every new failure mode becomes a fixture in the frozen task set, so the agent can't regress on it again.

## Output format

- **Eval harness:** the task set + the three scorers (task-completion / trajectory / tool-use) + offline-fixture setup.
- **Quality + cost/latency report:** per-dimension scores with token-cost and wall-clock (mean + p95).
- **Hardening plan:** caps, timeouts/retries, stop conditions, human-in-the-loop gates, tracing wiring.
- **Volatile facts** (model IDs, pricing, framework tracing APIs): dated + `[verify-at-use]`.

## Guardrails

- **A demo is not an eval** — score the tail on a frozen task set, not the happy path once.
- **A right answer via a bad trajectory is a latent failure** — always score the path, not just the outcome.
- **Never ship an agent with a real-world write and no human gate** on the irreversible step.
- **Never report quality without cost and latency** next to it.