Harness Audit

---
name: harness-audit
description: Agent-driven, read-only self-audit of the harness — run the machine integrity layer once as a dry-run, present a per-check pass/fail table, cite the P1-1 doc-reality result, and for any failure give root-cause + fix + backlog follow-up. Consumes the machine gates; it does not reimplement them. NOT for auditing a consumer project's own codebase (use that project's test suite), and NOT for applying fixes — this skill observes and reports only.
when_to_use: A periodic or on-demand health check of the harness itself — "audit the harness", "/harness-audit", "is the harness passing its own gates", or when you want one interpreted report over the CI/gate layer before a release or after a structural change.
tools: Bash, Read, Grep, Glob
---

# /harness-audit

## Goal

Produce **one interpreted health report** of the harness in a single read-only
dry-run. The machine integrity layer (P1-1 `doc-reality.sh`, the CI jobs, and the
`verify-all.sh` runner) is the *enforcement*; this skill is the *consumer* that
runs it once, reads off a per-check pass/fail table, and turns each failure into a
diagnosis a maintainer can act on.

Concretely the report answers three questions with evidence:

1. **Is the harness passing its own gates right now?** — a per-check PASS / FAIL /
   SKIP table, taken verbatim from `verify-all.sh`.
2. **What does the doc-reality gate (P1-1) say?** — cited explicitly, because it is
   the gate that catches the harness lying about itself (phantom paths + drifted
   counts), and it is the one item most easily missed in a wall of green.
3. **For every non-PASS: why, and what next?** — the check's purpose, the likely
   root cause, the recommended fix, and whether it warrants a new backlog row.

This skill supplies the *interpretation*; it does **not** re-implement any check.
The checks live in `core/tests/` and CI; here you run them and read the results.

## Why this is a layer, not a duplicate of P1-1

P1-1 (`core/tests/doc-reality.sh`) is a **machine gate**: it runs in CI, exits 0/1,
and mechanically blocks doc drift. This skill is the **agent consumer** that sits on
top of that gate — it invokes the whole machine layer in one dry-run and *interprets*
the results (root-cause, fix, backlog follow-up) the way a maintainer would. It reads
doc-reality's verdict; it never re-derives phantom-path or count logic. Deleting this
skill would not weaken any gate (CI still enforces them); deleting doc-reality would.
That asymmetry is the layer relationship: the gate enforces, the audit interprets.
This mirrors the machine-gate-vs-agent-consumer split the plan draws for H-2.

## Steps

### 1. Run the machine integrity layer (one dry-run)

```bash
bash core/tests/verify-all.sh
```

`verify-all.sh` is the single-command runner. It **dynamically discovers** every
`core/tests/*.sh` check (the runner and its own self-test aside, to avoid recursion —
that self-test runs as its own CI step) so no harness check is silently omitted, and
runs, per check, PASS / FAIL / loud SKIP with a final tally. Its discovered set
already includes the machine layer this audit reports on:

- **`doc-reality.sh` (P1-1)** — the doc-drift gate. It also cross-checks the §7
  artifact counts, which include the hook file count, so the historical **"hook
  count" audit item is covered here** (there is no separate hook-count step to run).
- **`sanitize-audit.sh`** — domain-neutrality (no prior-project taint).
- **`supply-chain-scan.sh`** — no injection-style directives in shipped files.
- **`adapter-parity.sh`** — the three adapters normalize one event identically.
- **`registry-drift.sh`** — the registry/manifest gate (plugin/marketplace fields,
  hooks.json → executable core-hook resolution, agent `name:` frontmatter, and
  registry↔agent model agreement).
- **every `*-test.sh` battery** — the per-hook and per-gate fixture tests.
- **`evals:deterministic` and `evals:semantic`** — the two eval layers (same
  invocation CI uses).
- **`gitleaks`** — the secret scan when present; a loud SKIP (never a silent pass)
  when the binary is absent.

Capture the runner's per-check output and its final `=== verify-all: … ===` tally.
This is entirely read-only: the runner and every check it dispatches only read the
repo — no files are written, nothing is committed. Note the runner's exit status: a
**non-zero exit means the harness is failing its own gates** and the report must say
so plainly.

To enumerate the checks without executing them (useful for a coverage sanity-check
of the report), `bash core/tests/verify-all.sh --list` prints the labels only.

### 2. Run the runtime layer (doctor + telemetry digest)

`verify-all.sh` covers the *repo*; two further read-only probes cover the
*runtime this repo is installed into* and the *live gate behavior*:

```bash
bash setup.sh --doctor
bash core/infra/telemetry-digest.sh
```

- **`setup.sh --doctor`** — environment diagnosis (tool availability, exec bits,
  registry/hooks sanity) plus the drift observers: plugin-cache single-version
  (a stale cached version re-exposing retired agents/skills), declared-hook
  manifest vs live settings reconciliation, and the phantom-command scan (a
  runtime `commands/*.md` invoking a script that does not exist on this
  machine). Doctor WARNs are observations, never blockers — report them in the
  same diagnosis format as step 4.
- **`telemetry-digest.sh`** — per-gate fire-rate from the local event sinks;
  surface any DEAD (never fires) or FATIGUE (high-frequency ask) candidates as
  calibration follow-ups, not as failures.

Append the doctor summary line (`doctor: N pass, M warn, K fail`) and any
DEAD/FATIGUE candidates to the report. If either probe's input is absent on
this machine (no runtime install, empty sinks), say so — an unmeasured layer is
reported as unmeasured, not as green.

### 3. Present the table (cite P1-1 explicitly)

Read the runner's output and render a per-check table:

```
check                    result   note
-----------------------  -------  -------------------------------------------
doc-reality.sh (P1-1)    PASS     docs match repo; §7 counts + hook count OK
sanitize-audit.sh        PASS     no prior-project taint
supply-chain-scan.sh     PASS     no injection-style directives
adapter-parity.sh        PASS     3 adapters agree
registry-drift.sh        PASS     manifests/hooks/agents/registry agree
… (batteries) …          PASS     fixture tests green
evals:deterministic      PASS     labeled dataset, Pass^3
evals:semantic           PASS     test-meaningfulness judge, Pass^3
gitleaks                 SKIP     binary not installed on this host
```

Call out the **doc-reality (P1-1)** line by name — its verdict is the headline of
this audit. State the overall tally (N passed / M failed / K skipped) and the
runner's exit status.

### 4. Diagnose each non-PASS

For every FAIL, every doctor WARN, and every SKIP that matters (e.g. a skipped
security scan), write:

- **Purpose** — what the check guards (one line).
- **Root cause** — the most likely reason it is failing, read from the check's own
  failure output (each gate prints the offending item; quote it).
- **Fix** — the concrete corrective step (edit which doc / ship which file / restore
  which exec bit), matching the guidance the gate itself prints.
- **Backlog follow-up** — whether this warrants a new row in
  `docs/internal/harness-improvement-plan.md` (a recurring or structural failure does; a
  one-off local fix does not). Suggest the row; do not add it here.

A SKIP is not a PASS: an absent `gitleaks` is a coverage gap to flag (install it or
rely on the dedicated CI secret-scan job), not a silent green.

### 5. Summarize health + next actions

Close with an overall verdict — healthy (all gates green) or failing (one or more
gates red) — and an ordered list of recommended next actions drawn from step 4. If
the harness is failing its own gates, that is the top-line finding.

## Completion condition

The audit is complete when a single dry-run has produced **(a)** a per-check
pass/fail table and **(b)** an explicit citation of the P1-1 (doc-reality) result.
Anything less (a table with no doc-reality line, or a doc-reality verdict with no
table) is an incomplete audit.

## Hard rules

- **Read-only.** This audit observes; it does not edit, fix, or commit. Diagnosis
  and fix *suggestions* are the output — applying them is a separate, human-directed
  step.
- **Do not reimplement the checks.** The gates in `core/tests/` and CI are the
  source of truth. This skill runs them and interprets their output; if a check's
  logic is wrong, fix the check (and its battery), not this skill.
- **The P1-1 citation is mandatory.** doc-reality is the gate that catches the
  harness lying about itself; its verdict is always named in the report.
- **A non-zero `verify-all.sh` exit is a failing harness.** Report it as such —
  never round a red run up to "mostly green".