Eval

---
name: eval
description: >-
  Run the context fitness-function probe suite (.oh/evals/probes/*.sh) against real
  state and write the .oh/evals/RESULTS.md benchmark. Each probe is a deterministic
  3-state oracle (PASS/REGRESSION/SKIPPED); a green→red transition is surfaced as
  a REGRESSION naming the lesson it closes. Tier-B behavioral evals are out of scope.
  TRIGGER when: asked to run evals, check the probe suite, "run /eval", verify a
  lesson's probe is green, benchmark the harness, or before/after editing a
  rule/skill that a probe guards.
---

# Eval

The runner for the harness **fitness function**. It discovers `.oh/evals/probes/*.sh`,
runs each against *real state*, and writes the `.oh/evals/RESULTS.md` scoreboard. A
rectification is provably "done" when its probe is green; a recurrence shows up as
a **REGRESSION** (was-PASS, now-fail) naming the `# source:` lesson. The full
contract — 3-state exit oracle, header convention, correction-surface triage — is
in [`.oh/evals/README.md`](../../../.oh/evals/README.md).

## Usage

```bash
bash .claude/skills/eval/run.sh                 # run the whole suite, rewrite RESULTS.md
bash .claude/skills/eval/run.sh --probe <id>    # run one probe, update only its row
bash .claude/skills/eval/run.sh --tier A        # run only Tier-A probes
```

Exit-code oracle (per probe): `0`=PASS, `1`=REGRESSION, `2`=SKIPPED (not
applicable — excluded from pass-rate), `124`=TIMEOUT, other=ERROR. Each probe is
wrapped in `timeout 30s`. Runner aggregate exit (the process `$?` of `run.sh`
itself): `0` when no new green→red regression occurred this run, `1` when one or
more new regressions were detected (`${#regressions[@]} > 0`). When invoked via the
Bash tool as `bash .claude/skills/eval/run.sh`, the agent caller reads `$?` directly
to gate on success — the printed `REGRESSIONS (...)` stdout block and per-probe stderr
lines remain the human-readable signal. Note: the `eval-weekly` cron is an intentional
legacy caller that appends `|| true` then greps stdout; it does not consume the exit
code by design — this is not a bug.

## What the runner does

1. **Discover + run** every probe matching the filters; extract `# tier:` /
   `# source:` via the exact header grep.
2. **Compute the delta** vs the prior `RESULTS.md` row. **First run** (no prior
   row) emits `new-pass`/`new-fail` and raises NO regression without prior state.
3. **Surface regressions** — any `PASS → (REGRESSION|TIMEOUT|ERROR)` transition is
   printed first, naming the probe's `source`.
4. **Rewrite `RESULTS.md` atomically** — build the full scoreboard into a temp
   sibling file (`RESULTS.md.tmp.$$`) and replace the live file in one `mv -f`
   (never truncate-then-append in place), so a crash or concurrent run can't leave
   a partial scoreboard. Overwrite the row for each probe run; carry prior rows for
   probes not run this invocation from a **pre-write snapshot (`RESULTS_ORIG`)**
   captured before the rewrite — not the live file — so a filtered run never erases
   untouched rows and the scoreboard stays complete.

## When NOT to use

- **Tier-B behavioral evals** (sub-agent + LLM-judge of judgment-call behavior)
  are deferred — `/eval` is deterministic only. Never hard-gate on a noisy metric.
- For *scoring* context files for staleness/budget, that is `/audit context` and
  `/audit skills` — `/eval` checks behavior/state, not prose quality.