Eval

---
name: eval
description: Use when evaluating whether an instruction or workflow change helps agents complete real work, using recorded sessions or bounded comparative tasks.
---

# Instruction experiments

Name the goal and the behavior expected to change. Use the existing experiment
record; a new campaign or separate reporting system is unnecessary. Preserve
the baseline and state what differs in the candidate. Ratification, publication,
installation, and measured usefulness are distinct facts.

For a forward comparison, freeze a small real task, expected outcome, starting
source/environment, required model/harness pairs, and run budget. Give fresh
runners the actual baseline or candidate instructions and the same task, without
the desired conclusion. Confirm the effective instruction sources. An unavailable
required pair stays unavailable; do not silently substitute another model.

Judge delivered behavior and honest evidence first. Record human corrections,
unnecessary work, elapsed time, and available token/cost data alongside results.
Use native transcripts and checks, not model agreement or compliance with a long
rubric. Validate new acceptance checks with a relevant known-broken control.
Decision-only scenarios are smoke tests, not implementation or throughput evidence.

For a session retrospective, sample relevant successes and failures, distinguish
close reading from inventory, and retain source locators. Check harness coverage
before making population claims. Separate missing instructions, ignored rules,
overbearing rules, and broken tools; each may need a different intervention.

Report keep/change/stop/insufficient-evidence. Preserve failed and incomplete
runs. A smaller prompt is not success if behavior, boundaries, or honesty degrade.
A small pilot cannot establish fleet reliability or causal effects of individual
edits in a bundled change. Expand the trial only for a consequential unresolved
question, then check ordinary-work outcomes after authorized adoption.