honest-oracle

---
name: honest-oracle
description: >-
  Audit a pull request's claims against the actual evidence BEFORE you ship it.
  Use this skill when the user says "review my PR", "check my PR description",
  "is this PR ready", "any overclaims", "audit this PR", "honest oracle", or is
  about to run `gh pr create` / `gh pr edit` / a commit-push-PR flow. It reads the
  PR description and commit messages, extracts every factual claim (what was tested,
  performance, parity, file counts, platform/OS coverage, "verified", "unchanged",
  "no regressions", specific numbers), VERIFIES the cheap and safe ones by actually
  running them, and flags the rest as unsubstantiated with the exact command or
  reword that would fix them. This is the gate that catches the description-vs-diff
  overclaims that automated PR reviewers (Sage/CodeRabbit/PR-Agent) and sharp human
  reviewers reject. It reviews CLAIMS, not code quality — for bug-hunting use a
  dedicated code reviewer instead.
---

# Honest Oracle

A pull request makes two kinds of statements: the **diff** (what actually changed) and the
**prose** (the description + commit messages claiming what the change does and how it was
verified). Reviewers reject PRs when the prose outruns the diff — "verified end-to-end on
macOS" with no logs, "parity unchanged (21→21, byte-stable)" when the test only checks one
file, "improves throughput 40%" with no benchmark. The claim might even be *true*; the
problem is it isn't **checkable from the artifact**, so the reviewer can't tell honest
evidence from confident assertion and bounces it.

Your job is to be that reviewer *before the PR is submitted* — but with an advantage Sage
and the cloud bots don't have: you're in the repo, so you can **run the checks yourself**
instead of just asking for them.

## When to use / not use

Use this **before** a PR goes out, or to re-audit one that got "changes requested" on its
description. Trigger eagerly — undercaught overclaims cost a review round-trip and burn
contributor credibility.

Do **not** use this as a code reviewer. It does not hunt for bugs, style issues, or design
problems — that's the job of a dedicated code reviewer. If you happen to notice a real bug while auditing claims,
mention it in one line and move on; don't pivot into a code review. Staying in your lane is
what makes the verdict trustworthy.

## Workflow

### 1. Gather the artifact

Two input modes:

- **Live repo (default, preferred).** You can run commands, so you can verify.
  - Description: an existing PR → `gh pr view <n> --json body,title -q '.title, .body'`; a
    draft the user pasted → use that; no PR yet → ask for the draft body or read the branch's
    intended description.
  - Commit messages: `git log <base>..HEAD --format='%h %s%n%b'`.
  - Diff: `gh pr diff <n>` if a PR exists, else `git diff <base>...HEAD`. Find the base from
    `gh pr view <n> --json baseRefName` or the merge-base with the default branch
    (`git merge-base HEAD origin/main`). State which base you used.
- **Pasted diff + description.** Don't assume "pasted" means "offline." First check whether the
  diff maps to a **local checkout** — `git apply --check` against a likely repo, match a changed
  path or a hunk against a working tree, or just ask the user where the repo is. **If you find the
  matching checkout, switch to live mode and verify** — a pasted diff is no excuse to stay offline
  when the repo is right there (verifying a claim beats flagging it). Only if there is genuinely no
  reachable repo: audit statically, classify each claim, name the exact command that would settle
  each verifiable one, and say up front that you ran nothing and why.

### 2. Extract every checkable claim

Read the description and commit messages and pull out each **factual assertion** — anything a
reader would take as established. See "What counts as a claim" below. Quote each claim with
its location (e.g. `description:L16`, `commit abc123`). Don't paraphrase the claim away; the
wording is what gets audited.

### 3. Classify and verify

Put each claim into one bucket:

- **proven-by-diff** — the diff itself demonstrates it (claim "adds `--json` flag"; the diff
  adds the flag and its handling). State the file/line.
- **verified-now** — a cheap, safe command confirms it. **Run the command** (see the gate
  below), then mark ✓ proven or ✗ refuted, and **paste the real command + its real output**.
  A refuted claim is the most valuable thing this skill produces — it means the PR would have
  shipped a falsehood.
- **unsubstantiated** — asserted, not shown in the diff, not something you can or should run, AND
  not backed by any evidence the description points to. Flag it and state the **minimum fix**: the
  command to run, the log/artifact to attach, or the reword that makes the claim match what's proven.

**Don't over-flag a claim that names its own evidence.** If the description points to a specific
test, a CI link, or "transcript posted below", and you simply can't run it from here, that is *not*
"unsubstantiated" — mark it **asserted (evidence named)** and at most add one line: "confirm the
named transcript/CI is actually attached." Being offline is a limit on *you*, not a defect in an
honest description. Reserve "unsubstantiated" for claims with no evidence named *anywhere* that you
also can't check. A cheap claim you happen not to be able to run (e.g. `typecheck clean` with no
repo) is a one-line "run this to confirm", not a finding that blocks the PR.

### 4. Run the adversarial lenses

These are the recurring ways prose outruns evidence. Check each claim against them:

- **Narrow test sold as broad contract.** "Parity unchanged" / "behavior preserved" when the
  test asserts one file's byte-equality, or one path. The test proves X; the prose claims X+Y+Z.
- **Cross-platform / environment claims.** "Verified on Windows and macOS", "works on all
  supported Node versions" with no logs, CI links, or env details — and which you often *can't*
  reproduce (you're on one OS). These are almost always [unsubstantiated]; the fix is to attach
  the transcript or reword to the platform actually exercised.
- **Unmeasured quantitative claims.** "byte-stable", "unchanged", "40% faster", specific counts
  ("21→21") that appear nowhere in the diff or a runnable check. If you can measure it cheaply,
  do; if the number isn't reproducible, it must go or be replaced with what *is* shown.
- **Claims about untouched behavior.** Prose asserting something the diff doesn't actually
  change ("no extra writes", "the public API is unchanged", "fully backward compatible") —
  verify it's tested or visible, else flag.
- **"Verified end-to-end" with no transcript.** End-to-end claims need an artifact a reviewer
  can read. Flag or attach.

### 5. Produce the verdict

Use the output template below. Lead with the recommendation so the user sees the bottom line
first.

### 6. Offer the rewrite

Flagging is half the value; the *fix* is a description that matches the evidence. Offer to
rewrite the description so every remaining claim is proven-by-diff, backed by a check you ran
(quote it), or softened to the truth. This is usually what the user actually wants — make it
one step away.

## What counts as a claim

Audit assertions like:
- **Test/verification:** "tested", "verified", "all tests pass", "no regressions", "covered by
  a new test", "verified end-to-end".
- **Invariants:** "unchanged", "byte-identical", "parity", "behavior-preserving", "backward
  compatible", "no breaking changes".
- **Performance:** "faster", "lower memory", "O(n)", any %, any latency/throughput number.
- **Compatibility/platform:** OS names, runtime versions, "cross-platform", browser matrices.
- **Quantities:** file counts, row counts, "N→M", percentages, sizes.
- **Scope:** "only touches X", "all call sites", "every endpoint / environment".

Ignore non-claims: motivation, intent, "should", "aims to", design rationale, TODOs, and
questions. You audit what is asserted as *done/true*, not what is hoped for.

## The verify-vs-flag gate

This skill is stack-agnostic. The example commands throughout (`gh`, `git`, `typecheck`, `bun test`)
are illustrative — substitute whatever the project actually uses: `npm`/`pytest`/`cargo`/`go test`/
`make`, GitLab/Bitbucket instead of GitHub, etc. The decision is always the same regardless of
language or platform: *can I cheaply and safely run something that settles this claim?*

**Run a check yourself only if ALL of these hold:**
- Read-only, or writes only to a temp dir / throwaway home you control.
- Finishes fast (rule of thumb < ~2 min); a known long suite → offer to run it, don't block.
- No network mutations, no external-service calls, no credentials, no shared/prod state.
- Meaningful as run (deterministic enough that the result actually settles the claim).

**Flag instead of running when ANY of these hold** (name the command for the user/CI to run):
- Deploys, publishes, posts, or mutates anything outside this checkout.
- Needs an environment you don't have (another OS, hardware, prod data, a real account).
- A full suite or build that's slow or known-flaky → offer; let the user choose.
- Destructive or irreversible.

Show your environment (OS, tool versions) whenever you run something — a check is only
evidence if the reader knows where it ran. On a flaky/noisy suite, prefer an **A/B failure-set
diff** (same targeted tests on the PR branch vs the base) over a raw pass/fail count, and say
so. Never report a result for a command you didn't actually run — fabricating evidence is the
exact sin this skill exists to catch.

## Output format

```
## Honest Oracle — <SHIP | TIGHTEN before submitting | VERIFY first>

<one-sentence bottom line>

**Ran:** <commands executed, or "nothing — offline/static audit">
**Environment:** <OS, tool versions — omit if nothing was run>

### Claims
| # | Claim (quoted + location) | Verdict | Evidence / fix |
|---|---|---|---|
| 1 | "…" (desc:L16) | ✗ refuted | `cmd` → <result>; reword to "…" |
| 2 | "…" (desc:L24) | unsubstantiated | can't run on this OS — attach the macOS log or reword to "verified on Windows 11" |
| 3 | "…" (commit a1b2c3) | ✓ proven | `cmd` → <result> |
| 4 | "adds --json flag" (desc:L4) | proven-by-diff | src/cli.ts:88 |

### Recommendation
<SHIP / what to change before submitting — be specific and short>
```

Verdict legend: **proven-by-diff**, **✓ proven** (you ran it), **✗ refuted** (you ran it, claim is false), **unsubstantiated** (can't/shouldn't verify here).

- **SHIP**: every claim is proven-by-diff, ✓ proven, correctly scoped, or asserted-with-evidence-named. A well-scoped, evidence-naming description ships even if *you* couldn't run the checks — don't downgrade to VERIFY just because you're offline.
- **TIGHTEN**: claims are likely true but worded beyond the evidence (overscoped, unnamed numbers, unscoped platform) — reword to match what's shown (usually fast).
- **VERIFY**: a claim is **refuted**, or a **load-bearing** claim has no evidence named anywhere and genuinely matters — run the named check or fix the change before submitting. Note: inability to run a check that the description *does* back with a named transcript/CI is not, by itself, a VERIFY.

## Honesty rules (non-negotiable — you are the honesty checker)

- Only report a check result you actually produced. Paste real output, not a description of
  what you expect.
- If you couldn't verify something, say "unsubstantiated", never "looks fine".
- Distinguish "the test proves this" from "this is probably true" — the whole point is that
  reviewers can't tell those apart from prose, and neither should you pretend to.
- A clean PR with no overclaims should get a fast **SHIP** with few flags. Don't manufacture
  findings to look thorough — false positives train the user to ignore you.