Agent Eval Framework

---
name: omcustom:agent-eval-framework
description: Quantitative agent evaluation using 4-metric framework (correctness/step_ratio/tool_call_ratio/latency_ratio) with ideal trajectory annotation and capability-categorical taxonomy. Use when measuring agent efficiency, comparing agent variants, or gating new agents through correctness→efficiency phases. Complements harness-eval (SE benchmarks) and evaluator-optimizer (qualitative rubric).
scope: harness
version: 1.0.0
user-invocable: true
argument-hint: "<measure|compare|gate> <agent-name> [task-id]"
---

# Agent Eval Framework

## Purpose

Provides **quantitative, trajectory-based evaluation** for oh-my-customcode agents. Fills the measurement gap not covered by existing skills:

| Skill | Coverage | Gap |
|-------|----------|-----|
| harness-eval | SE benchmark task quality (15 tasks) | No efficiency metrics |
| evaluator-optimizer | Qualitative rubric loop | No quantitative gate |
| deep-verify | Release quality (structural/correctness) | No step/latency ratios |
| multi-model-verification | Code correctness across models | No trajectory comparison |

This skill adds **efficiency measurement** — not just "did the agent succeed?" but "how efficiently did it succeed relative to an ideal trajectory?"

## The 4-Metric Framework

Derived from LangChain's deep agent evaluation methodology.

| Metric | Formula | Direction | Use |
|--------|---------|-----------|-----|
| correctness | pass / fail per task | binary | Phase 1 gate — must pass before efficiency |
| step_ratio | observed_steps / ideal_steps | lower is better | Measures unnecessary reasoning hops |
| tool_call_ratio | observed_tool_calls / ideal_tool_calls | lower is better | Measures redundant tool invocations |
| latency_ratio | observed_latency_s / ideal_latency_s | lower is better | Measures wall-clock efficiency |

**Thresholds (recommended starting points)**:
- step_ratio ≤ 1.5 — acceptable
- tool_call_ratio ≤ 1.3 — acceptable
- latency_ratio ≤ 2.0 — acceptable (parallelism headroom)

## Ideal Trajectory Annotation

Each task requires a hand-annotated ideal trajectory stored in `.claude/outputs/evals/trajectories/`.

```yaml
task_id: example-001
capability: file_operations
ideal:
  steps: 4
  tool_calls: 4
  latency_seconds: 8
description: "Refactor user.py — read, parse, edit, verify"
```

**Annotation guidelines**:
- `steps`: count of distinct reasoning/action steps in an expert run
- `tool_calls`: minimum tool invocations required (no redundant reads)
- `latency_seconds`: median of 3 expert runs
- `capability`: one of the six taxonomy categories below

## Capability-Categorical Taxonomy

Maps LangChain capability categories to oh-my-customcode tools and task types.

| Capability | Tools | Example Tasks |
|------------|-------|---------------|
| file_operations | Write, Edit | Refactor, create files, patch configs |
| retrieval | Glob, Grep, Read | Code search, dependency analysis, symbol lookup |
| tool_use | Agent, Skill, Bash | Multi-tool workflows, pipeline execution |
| memory | Read/Write to `.claude/agent-memory*/` | Context recall, R011 patterns, cross-session refs |
| conversation | routing skills (secretary/dev-lead/de-lead/qa-lead) | Multi-turn user interaction, intent routing |
| summarization | result-aggregation skill | Multi-agent synthesis, parallel result merge |

Use this taxonomy to **select representative tasks per category** when building an eval suite. Aim for ≥3 tasks per capability category.

### Tool: Writing artifacts under .claude/outputs/

Under `mode: "bypassPermissions"`, direct Write/Edit/Bash on `.claude/**` paths is permitted (CC v2.1.121+, #1101) — no `/tmp/*.sh` wrapping is needed.

To write eval trajectories or result reports under `.claude/outputs/evals/`:

1. Write the artifact body directly to `.claude/outputs/evals/{trajectories,sessions}/...` with the Write tool (every Agent tool call includes `mode: "bypassPermissions"`, R010)
2. Read-only Bash on `.claude/outputs/` (e.g., `cat`, `head`, `wc`) is allowed for verification

Reference: R006/R010 sensitive-path handling (CC v2.1.121+), #1101.

## Phased Opt-in Gate Workflow

**Phase 1: Correctness Gate** (MUST pass before Phase 2)

1. Run agent against task set
2. Score each task pass/fail
3. Require correctness ≥ 0.80 (80%) before proceeding
4. If below threshold: diagnose failure modes, fix agent, re-run

**Phase 2: Efficiency Comparison**

1. Compute step_ratio, tool_call_ratio, latency_ratio per task
2. Aggregate by capability category
3. Compare against baseline (previous agent version or reference agent)
4. Flag regressions (ratio increase > 20%)

**Consumers of this workflow**:
- **mgr-creator**: candidate consumer — opt-in for new agents requiring quantitative gating before deployment
- **worker-reviewer-pipeline**: gate review cycles — efficiency regression = re-review trigger
- **deep-verify**: optional quantitative dimension for release quality checks

## Tracing Infrastructure (LangSmith Alternative)

oh-my-customcode uses existing infrastructure for trajectory capture:

| Component | Role | How |
|-----------|------|-----|
| native MEMORY.md (R011) | Persist eval metrics across sessions | Write `## Metrics` section with task_id + observed metrics to agent-memory MEMORY.md |
| statusline.sh (R012) | Real-time step counter during eval runs | Extend statusline with `STEPS=n` segment |
| `.claude/outputs/evals/` | Artifact storage for eval results | Per-session eval reports in `sessions/{YYYY-MM-DD}/` |

**Trace capture pattern**:
```
task start → record tool_calls[] + timestamps → task end
→ compute ratios against ideal trajectory
→ write to native MEMORY.md ## Metrics: {task_id, capability, correctness, step_ratio, tool_call_ratio, latency_ratio}
```

### Persistent Storage (added v0.116.0, #1036)

Baseline annotations and observed trajectories can be persisted to eval-core's SQLite database (`evalBaselines` + `agentTrajectories` tables). This complements the YAML file approach for cross-session analysis. The eval-core query API for the `evalBaselines`/`agentTrajectories` tables is not yet implemented — use the YAML annotation approach meanwhile.

## Integration with Existing Skills

| Skill | Integration Mode | How |
|-------|-----------------|-----|
| harness-eval | Additive | After harness-eval runs 15 SE tasks, apply 4-metric layer to each result |
| evaluator-optimizer | Additive | After rubric loop converges, run efficiency gate as final check |
| deep-verify | Optional | Add `--quantitative` flag awareness; deep-verify can invoke this skill |
| mgr-creator | Gate | New agent creation includes Phase 1 correctness check before agent is deployed |

## Usage Pattern

```
/omcustom:agent-eval-framework measure <agent-name> <task-id>
/omcustom:agent-eval-framework compare <variant-a> <variant-b>
/omcustom:agent-eval-framework gate <agent-name>  # correctness → efficiency
```

**measure**: Runs a single agent against a single task, outputs all 4 metrics.

**compare**: Runs two agent variants against the same task set, produces side-by-side ratio table.

**gate**: Full two-phase gate — Phase 1 correctness check, then Phase 2 efficiency if Phase 1 passes. Returns `PASS` or `FAIL` with metric breakdown.

### Example Output

```
[agent-eval-framework] gate lang-golang-expert
Phase 1: correctness = 0.87 (13/15) ✓ threshold: 0.80
Phase 2: efficiency
  step_ratio:      1.12 ✓
  tool_call_ratio: 1.08 ✓
  latency_ratio:   1.31 ✓
Result: PASS — agent approved for deployment
```

## R020 Linkage

Quantitative metrics provide **[Done] gate evidence** beyond binary completion checks (R020 MUST-completion-verification).

| R020 Task Type | 4-Metric Evidence |
|---------------|-------------------|
| Implementation | correctness ≥ threshold + step_ratio ≤ 1.5 |
| Agent/Skill Creation | Phase 1 gate PASS (mgr-creator workflow) |
| Code Review | tool_call_ratio as efficiency signal for review thoroughness |

When declaring `[Done]` for agent creation or major workflow changes, include eval gate results as completion evidence.

See R020 "Optional: Quantitative Evidence" section for the consumer-side advisory pattern.