Evidence Driven Testing

---
name: evidence-driven-testing
description: Capture observable runtime proof with video, screenshots, or measured output. Use when a change needs evidence beyond test reports.
compatibility: Requires Python 3 and FFmpeg for recording, or Node.js 20+ and Chromium for headless capture. Desktop capture also needs OS screen-recording access. Publishing evidence requires network access and authorized destination tooling.
metadata:
  author: michaelshimeles
  version: "1.3.0"
  owner: software-factory
  risk: medium
  capabilities: runtime-evidence,visual-evidence,test-reporting
---

# Evidence-Driven Testing

Record proof of behavior and deliver it to the requested destination.

Choose evidence proportional to the change. Use the recorder for interactive
flows, scripted captures for headless UI checks, and output pairs for CLI or
API changes. Documentation-only edits need content checks and verification of
runnable examples, not a recording. Reuse suitable evidence already captured
in the task. Publish only to destinations included in the requested work.

The recording is the capture of you testing the app via computer use: start the
recorder, then drive the app yourself — click, type, navigate — through each
test target. Every action in the video is the test being performed live; the
recording has no value as evidence unless it shows that interactive session.
If the harness has no computer-use tools but a GUI exists, drive the app with
`cua-driver` instead (see below) — it is still your live session.

The bundled recorder, `scripts/evidence.py`, captures the display with FFmpeg
on Linux, macOS, and Windows, timestamps each annotation you add while
testing, burns them into the video on stop, and verifies the result with
ffprobe. It writes `evidence.mp4`, `report.md`, and `manifest.json` into the
session folder.

## Inputs

- **Test targets** (required): The behaviors/flows to verify, phrased as testable statements.
- **PR / issue** (optional): Where to post the evidence. If omitted, deliver to the requester only.

## The recorder

`EVIDENCE` below means the path to `scripts/evidence.py` inside this skill's
folder (wherever the skill is installed, e.g.
`~/.claude/skills/evidence-driven-testing/scripts/evidence.py` or
`~/.agents/skills/evidence-driven-testing/scripts/evidence.py`). It needs only
Python 3 and FFmpeg.

- **Check first**: `python3 $EVIDENCE doctor` — verifies `ffmpeg`, `ffprobe`,
  `libx264`, and the `ass` filter (`ready`), then which screen-capture source
  works on this machine (`capture_ready` and the source `auto` will pick).
  Exits non-zero only when the toolchain is missing; read `capture_ready`
  before recording.
- **Platforms** (`--source auto` picks the first available):

  | OS | Source | Needs |
  |---|---|---|
  | Linux X11 / XWayland | `x11` (x11grab) | `DISPLAY` set |
  | Linux Wayland | `wayland` (wf-recorder) | `WAYLAND_DISPLAY` set, `wf-recorder` on PATH, and a compositor confirmed to support wlr-screencopy — either a known wlroots one (Sway, Hyprland, river, Wayfire, labwc, dwl, niri) or verified via `wayland-info`. GNOME and KDE Wayland are not capturable this way; `doctor` says so. Use `x11` through XWayland for X11 apps, or a fallback recorder. `--source wayland` still forces it |
  | macOS | `avfoundation` | Screen Recording permission granted to the terminal / agent host app; `doctor` lists screen indexes for `--screen-index` |
  | Windows | `gdigrab` | any standard ffmpeg build; `powershell` for process checks |

  Capture is the full screen by default; `--geometry WxH` and `--offset X,Y`
  crop a region on x11, wayland, and gdigrab. XWayland only sees X11 windows,
  so prefer the `wayland` source when the app under test is Wayland-native.
- **Crash-safe raw capture**: the raw recording is MPEG-TS (`raw.ts`), so if
  the recorder is killed hard or crashes, what was captured still probes and
  renders. `stop` remuxes or re-encodes it into a standard `evidence.mp4`.
- **How stopping works**: on Linux the recorder is signalled through a pidfd.
  On macOS and Windows `start` launches a small supervisor process that owns
  the ffmpeg child, and `stop` asks it (via `stop.request` in the session
  folder) to interrupt, then terminate, then kill; it writes
  `recorder-exit.json` when done. Nothing ever signals a bare PID, so a
  recycled PID can't be hit. If the supervisor dies while the recorder is
  still running, `stop` refuses and tells you which PID to stop by hand. If
  it died before it could even record which process it started, `stop` stays
  blocked until you have checked for a stray recorder yourself and rerun it
  with `--accept-untracked-recorder`; the report then carries that caveat.
- **Fallback recorders** when `doctor` reports no capture source (for
  example Wayland without wf-recorder): `cua-driver recording start <dir>` /
  `stop` (see the cua-driver section), or the OS recorder (macOS:
  `screencapture -v out.mov`). On these paths there is no annotation overlay,
  so keep the annotation protocol as files — an `assertions.md` listing each
  `setup` / `test_start` / `assertion` with its result and the approximate
  video timestamp, exactly as in the headless path.
- **Never** present `--source test` (the synthetic pattern generator) as UI
  evidence. It exists to smoke-test the toolchain; the repo's
  `tests/test_evidence.py` exercises it.

## Instructions

### 1. Prepare the screen

- Maximize the browser/app window; close popups, notifications, and extra panels.
- Navigate to the starting state (logged in, correct page) BEFORE recording, unless setup itself is under test.
- Note the exact revision under test: `git rev-parse HEAD` and
  `git branch --show-current` (or the deployment URL) — the recorder stamps
  them into the report.

### 2. Start recording

- Begin the screen recording before the first meaningful action:

  ```bash
  python3 $EVIDENCE start \
    --output .artifacts/<task-name> \
    --title "<what is being verified>" \
    --commit "$(git rev-parse HEAD)" --branch "$(git branch --show-current)" \
    --environment "<OS / browser / display / deployment>"
  ```

  It prints JSON with a `session` path and the chosen `source`; keep the
  path (`SESSION=...`) for every later command. The source is auto-detected
  and the whole screen is captured; pass `--source`, `--geometry`,
  `--offset`, `--display`/`--xauthority` (X11), `--screen-index` (macOS), or
  `--output-name` (Wayland) only when `doctor` or the situation calls for it.
- Add a `setup` annotation describing the starting context:

  ```bash
  python3 $EVIDENCE annotate "$SESSION" --type setup \
    --message "Logged in, navigating to connectors page"
  ```

### 3. Test via computer use, annotating as you go

- Perform every interaction through computer use on the live app — the
  recording captures your session, so the testing and the evidence are the
  same act. Work at a watchable pace: let the UI settle after each action so
  state changes are visible on video.
- At each named test's start, add a `test_start` annotation in Jest style:

  ```bash
  python3 $EVIDENCE annotate "$SESSION" --type test_start \
    --message "It should execute the tool directly when permission is 'always'"
  ```

- After each check, add an `assertion` annotation with `--result passed`,
  `failed`, or `untested`:

  ```bash
  python3 $EVIDENCE annotate "$SESSION" --type assertion --result passed \
    --message "Tool ran without a permission prompt"
  ```

- Rules for assertions:
  - One assertion per meaningful state change — consolidate, don't annotate per UI label.
  - Use "Precondition: ..." assertions to establish starting state.
  - Keep under 80 characters, high-signal (the recorder rejects longer messages).
  - If a test cannot run (missing prerequisite, expired auth window), mark it `untested` with the reason — never skip silently.
  - The timestamp records when you asserted, not whether it was true — look at
    the screen before choosing `passed`.

### 4. Stop and review

- Stop recording after the final assertion:

  ```bash
  python3 $EVIDENCE stop "$SESSION"
  ```

  This stops the capture (gracefully, so the recorder flushes; escalating
  only if it ignores the request), burns the annotations into `evidence.mp4`,
  probes the result, and writes `report.md` and `manifest.json` next to it.
  It prints `"verified": true` on success; if rendering fails the session is
  marked `finalization_failed` — fix the reported cause and run `stop` again
  (a retry does not signal the recorder twice). If the recorder process can no
  longer be signalled safely (it died, or its PID now belongs to another
  process), the session is marked `recorder_lost`. Running `stop` again
  finalizes whatever video was captured, but only once that recorder process
  is confirmed gone — if it is still alive, stop it first, or the video would
  be rendered while still being written.
- Confirm the recording captured the key moments before sharing: extract a
  frame at each assertion timestamp (`ffmpeg -ss <t> -i evidence.mp4
  -frames:v 1 frame.png`) and check the state and the label are visible.
- Fill in the Caveats section of `report.md`; never leave the placeholder.

### 5. Post the evidence

- `report.md` is the report: what was tested, environment + exact commit,
  pass/fail per test, caveats. Extend it rather than rewriting from scratch.
- Post the video + summary as a PR comment (embed in the PR description if
  it's your PR). `gh pr comment` cannot attach a local video — upload
  `evidence.mp4` through the PR's comment box in an authenticated browser, or
  upload it to a host and link it (for example the `before-and-after` upload
  adapters). Reopen the comment and confirm the video plays before claiming it
  is posted.
- If the task includes a tracker issue and posting there is authorized, attach
  the same video with a one-line result.
- Send the report + recording to the requester.

## Guardrails

- The video must show the actual test session being driven live. Never present
  scripted playback, stitched clips, or synthetic footage as a recording; if
  the harness lacks computer-use tools but a GUI exists, drive via
  `cua-driver`; with no GUI at all, use the headless path instead.
- Never record a half-covered or tiled window — maximize first.
- Never record a screen showing secrets, tokens, customer data, or payment
  details; if a flow requires them, mark it `untested` and say why.
- When verifying a fix, show or reference the old failure alongside the new success.
- Always state the exact commit/branch/deployment tested against.

## No computer-use tools? Drive with cua-driver (GUI available)

When a display exists but the agent has no built-in computer-use capability,
use [cua-driver](https://github.com/trycua/cua) (macOS / Windows / Linux) as
the actuator. It is still you testing the app live — the recording rule holds
unchanged; only the input mechanism differs.

- Verify the setup with `cua-driver doctor` before recording. If a
  `cua-driver` skill is installed, read it and follow its protocol — the
  snapshot-before-action invariant is mandatory.
- Loop per interaction: `launch_app` → `get_window_state` (accessibility tree
  + screenshot) → act via `element_token` (`click`, `type_text`, `press_key`)
  → `verify_state` for the expected postcondition. Each `verify_state` check
  maps 1:1 onto an `assertion` annotation.
- Wherever `doctor` reports `capture_ready: yes`, keep using the bundled
  recorder above for the video and the annotations; cua-driver only supplies
  the input.
- Otherwise, `cua-driver recording start <output-dir>` / `cua-driver recording
  stop` is the recorder (the output directory is required, and the daemon
  must be running: `cua-driver serve`). Video capture is on by default and is
  finalized to `<output-dir>/recording.mp4` on stop — but on Windows/Linux it
  shells out to ffmpeg, so a missing ffmpeg or display yields only the
  per-turn trajectory folders (before/after screenshots, `action.json`,
  `click.png`), no video. After stopping, verify `recording.mp4` exists
  before citing it; if it is absent, fix the recorder or present the
  per-turn before/after screenshots as numbered captures per the headless
  protocol.
- If no annotation overlay is available on this path, keep the protocol as
  files: an `assertions.md` listing each `test_start` / `assertion` with its
  result, exactly as in the headless path.

## Headless path (no GUI available)

When the agent has no desktop to record, keep the same assertion discipline;
swap the recorder for scripted capture:

- Save everything to `.artifacts/<task-name>/` (gitignore it — evidence gets
  uploaded, never committed). Keep the capture script beside the captures so
  the run is repeatable.
- **Screenshots**: the `before-and-after` CLI (`@vercel/before-and-after`)
  captures URLs or elements and its pairs feed PR embeds directly. In
  containers/VMs where Chrome fails with "No usable sandbox", set
  `AGENT_BROWSER_ARGS="--no-sandbox"`.
- **Video / multi-step flows**: a one-off Playwright script, run without
  adding playwright to the project's dependencies:

  ```bash
  bash /path/to/evidence-driven-testing/scripts/run-playwright.sh record.mjs
  ```

  The helper installs the pinned Playwright version from its bundled manifest
  and lockfile with `npm ci` in a temporary directory, copies the script
  beside that installation, installs Chromium if needed, and removes the
  temporary directory on exit. It preserves the caller's working directory,
  so relative artifact paths still point into your project. Node 20+ and npm are
  required; the browser cache remains available for later runs. Keep the script
  self-contained: relative imports and adjacent data files are not copied.
  Use the project's existing Playwright setup for scripts with local imports.
  When upgrading capture tooling, update `scripts/playwright/package.json`,
  regenerate its lockfile with npm, and rerun the capture checks.

  `npx --package=playwright` adds executables to PATH; it does not make the
  package importable from an arbitrary ESM script. Minimal `record.mjs`:

  ```js
  import { chromium } from "playwright";
  const browser = await chromium.launch();
  try {
    const context = await browser.newContext({
      recordVideo: { dir: ".artifacts/<task-name>/" },
    });
    try {
      const page = await context.newPage();
      await page.goto("http://localhost:3000/path-under-test");
      // Drive the flow and check each expected state here.
    } finally {
      await context.close(); // Finalizes the video even if an assertion fails.
    }
  } finally {
    await browser.close();
  }
  ```

  Trim or compress with ffmpeg if the file is large.
- **The annotation protocol becomes files**: number captures in test order
  with the assertion in the name — `01-precondition-signed-in.png`,
  `02-it-saves-on-blur-passed.png` — and keep an `assertions.md` in the
  artifacts folder listing each `test_start` / `assertion` with its result
  (`passed` / `failed` / `untested` + reason).

## Non-UI changes still need evidence

- **API / performance**: a scripted probe with measured numbers — request
  counts per phase, latency before/after — captured to `probe-output.txt`.
- **Rendering / canvas / shader**: rendered frames plus pixel assertions
  (diff values), reviewed by eye and saved as PNGs.
- **Agent behavior**: the relevant transcript excerpt showing the tool call
  and response.
- **Bug fixes**: reproduce and capture the failure **before** writing the
  fix — that capture is the "before" half of a before/after pair.

## Capture hygiene

- Confirm the server you're probing is running *your* code (right port,
  right process), especially when multiple agents share a machine:
  `lsof -i :<port>` — or where `lsof` isn't installed,
  `ss -ltnp "sport = :<port>"` to find the listener's PID, then
  `ps -p <pid> -o args=` to confirm it's yours.
- Evidence complements the repo's checks (typecheck/build/tests); it never
  replaces them.
- Hand before/after media pairs to a before/after tool for the PR embed
  (e.g. `before-and-after before.png after.png --markdown`).