Paper Claim Audit

---
name: paper-claim-audit
description: "Check paper numbers, comparisons, and scope claims against raw result files with a fresh reviewer. Use for paper-to-evidence fidelity before submission."
argument-hint: "[paper-directory]"
allowed-tools: Bash(*), Read, Write, Edit, Grep, Glob
---

<!-- EVO-INQUIRY:BEGIN -->
For research work, read [EvoInquiry](../shared-references/evo-inquiry.md),
or reuse it if already loaded and unchanged in this session. Apply relevant
principles within the user's scope and this skill's existing outputs, review
standards, and recovery contracts. Carry them into nested research tasks and
reviewer briefs. Continue authorized work through its necessary steps; honor
budgets, blocking checkpoints, and stopping conditions.

For this stage, read [project-governance](../shared-references/project-governance.md). Audit against fixed raw execution/result/report refs and preserve the actual verdict digest. Governance integrity and scientific fidelity are separate checks.
Check review availability early. If fresh review is unavailable or prohibited,
finish the requested preliminary findings and the required blocked audit
artifacts; preserve the fresh-review gate. The bounded fallback below permits
executor checks only when clearly labeled, never a fabricated reviewer verdict.
<!-- EVO-INQUIRY:END -->


# Paper Claim Audit: Zero-Context Evidence Verification

> **Codex assurance:** write `review_independence: same-family` and
> `acceptance_status: provisional` into base audit JSON. A fresh Codex PASS may
> advance the pipeline but cannot produce submission-ready yes. Missing/failed
> review emits BLOCKED; overlay/deterministic acceptance uses accepted.

Verify that every claim in the paper matches raw evidence for: **$ARGUMENTS**

## Why This Exists

The executor writes experiments AND writes the paper. It "knows" what the results should be. This creates confirmation bias:
- Rounding 84.7% up to 85.3%
- Reporting best seed instead of average
- Citing metrics from a different experiment config
- Claiming "improves by 15%" when the delta is actually 12.8%

A **fresh reviewer with zero prior context** catches these because it has no expectations — it just compares paper text vs raw files.

## How This Differs From Other Audit Skills

| Skill | Question it answers |
|-------|-------------------|
| `/experiment-audit` | Is the experiment code honest? (fake GT, normalization fraud) |
| `/result-to-claim` | Does the data scientifically support this claim? |
| **`/paper-claim-audit`** | **Does the paper report the data truthfully and precisely?** |

## Core Principle

**Zero-context, fresh reviewer.** The auditor receives ONLY:
- Paper .tex files (the claims)
- Raw result files (the evidence)

It does NOT receive:
- ❌ EXPERIMENT_LOG.md
- ❌ EXPERIMENT_TRACKER.md
- ❌ AUTO_REVIEW.md
- ❌ NARRATIVE_REPORT.md
- ❌ Any executor summary or interpretation
- ❌ Any prior audit results
- ❌ Any conversation history

This is **stricter than reviewer-independence** — it's zero-context evidence audit.

## Evidence calibration

Keep `claim_supported: yes | partial | no` and existing review gates. Explain
whether an unsupported claim has contrary evidence or simply lacks sufficient
evidence; `no` is not automatically a refutation. File existence, number matching,
review acceptance and scientific support are distinct judgments. Preserve gains,
regressions, unknowns and the original primary metric in the reviewer brief.

## When the required reviewer cannot run

Keep the fresh-review path below unchanged when a reviewer is available. If the
review is prohibited or unavailable, record the review gate as BLOCKED and
write the required canonical audit JSON with reason `reviewer_unavailable`;
never fill reviewer identity or verdict fields with an imagined call.

When the user requests numerical consistency findings, the executor may still
compare the supplied paper with raw results to deliver preliminary notes.
Label their author and verification level explicitly. Preserve mismatches,
matching numbers, confounds and unknowns. These notes do not satisfy fresh
review, do not grant acceptance and must not enter a later zero-context
reviewer's brief. Save the requested notes, the canonical blocked receipt and
necessary navigation, verify their paths, then finish the response. Do not
simulate review rounds or repeat the audit merely because the reviewer is absent.

## Workflow

### Step 1: Collect Files (Executor — Codex)

Locate paper and result files WITHOUT reading or interpreting them.

**Paper files** (claims) — paths shown relative to the shell's working
directory so you can find them with `ls`; when writing them into
`audited_input_hashes`, use paths relative to the paper dir (no `paper/`
prefix) per the "Submission Artifact Emission" section below:
```
paper/main.tex                # → hash key: main.tex
paper/sections/*.tex          # → hash key: sections/*.tex
paper/tables/*.tex (if separate)   # → hash key: tables/*.tex
```

**Result files** (evidence):
```
results/*.json, results/*.jsonl, results/*.csv, results/*.tsv
outputs/*.json, outputs/*.csv
wandb-summary.json (if exists)
**/metrics.json, **/eval_results.json
**/config.yaml, **/args.json (experiment configs)
```

**Exclude** (no summaries, no interpretations):
```
EXPERIMENT_LOG.md, EXPERIMENT_TRACKER.md, AUTO_REVIEW*.md
NARRATIVE_REPORT.md, PAPER_PLAN.md, findings.md
Any .md file that is an executor-written summary
```

### Step 2: Fresh Reviewer Audit (GPT-6-Astra — NEW thread, no reply)

**CRITICAL: Use a fresh reviewer agent every run.** Never reuse an old reviewer context for this audit.

```text
spawn_agent:
  model: gpt-6-astra
  reasoning_effort: ultra
  message: |
    You are a paper-to-evidence auditor. You have ZERO prior context about
    this research. You will receive only paper source files and raw result
    files. Your job is to verify that every number in the paper exactly
    matches the raw evidence.

    Paper files to read:
    [list .tex file paths]

    Result files to read:
    [list .json/.csv/.yaml file paths]

    ## Audit Protocol

    ### A. Extract Every Quantitative Claim
    For each number, percentage, comparison, or scope statement in the paper:
    - Location (section, table, caption, or inline text)
    - Exact claim text
    - The number or comparison being made

    ### B. Trace Each Claim to Evidence
    For each extracted claim, find the supporting raw data:
    - Which result file contains this number?
    - What is the EXACT value in that file?
    - Match status: exact_match / rounding_ok / mismatch

    ### C. Check These Specific Failure Modes

    1. **Number inflation**: Paper says 85.3%, raw file says 84.7%
       Rule: only standard rounding to displayed precision is allowed

    2. **Best-seed cherry-pick**: Paper says "achieves 90.2%" but
       that's the best of 5 seeds; mean is 87.1%
       Rule: check if paper specifies "average" / "best" / "median"

    3. **Config mismatch**: Paper compares Method A vs Baseline B,
       but they used different hyperparameters / datasets / splits
       Rule: verify config files show same settings for compared methods

    4. **Aggregation mismatch**: Paper says "average over 5 seeds"
       but result files show only 3 runs
       Rule: count actual runs vs claimed count

    5. **Delta error**: Paper says "improves by 15%" but
       actual delta is (85.3 - 73.1) / 73.1 = 16.7%
       Rule: verify arithmetic of all relative improvements

    6. **Caption-table mismatch**: Figure caption describes
       something different from what the figure/table actually shows
       Rule: cross-check every caption against its content

    7. **Scope overclaim**: Paper says "consistently outperforms"
       but only tested on 2 datasets
       Rule: check if language matches actual evaluation scope

    ## Output Format (per claim)
    For each claim, report:
    - claim_id: sequential number
    - location: section/table/figure
    - paper_text: exact quote from paper
    - paper_value: the number claimed
    - evidence_file: which raw file
    - evidence_value: the actual number
    - status: exact_match | rounding_ok | ambiguous_mapping |
              missing_evidence | config_mismatch | aggregation_mismatch |
              number_mismatch | scope_overclaim | unsupported_claim
    - details: explanation if not exact_match

    Overall verdict: PASS | WARN | FAIL
```

### Step 3: Write Report (Executor — Codex)

Parse the reviewer's response and write `PAPER_CLAIM_AUDIT.md`:

```markdown
# Paper Claim Audit Report

**Date**: [today]
**Auditor**: GPT-6-Astra ultra (fresh zero-context thread)
**Paper**: [paper title from tex]

## Overall Verdict: [PASS | WARN | FAIL]

## Claims Verified: [N total]
- exact_match: [count]
- rounding_ok: [count]
- ambiguous_mapping: [count]
- missing_evidence: [count]
- mismatch: [count]

## Issues Found

### [FAIL/WARN] Claim #N: [description]
- **Location**: Section X / Table Y / Figure Z
- **Paper says**: "..."
- **Evidence shows**: ...
- **Status**: [status]
- **Fix**: [specific correction needed]

## All Claims (detailed)

| # | Location | Paper Value | Evidence Value | Status |
|---|----------|-------------|---------------|--------|
| 1 | Table 2 | 85.3% | 85.28% | rounding_ok |
| 2 | Abstract | "15% improvement" | 12.8% | number_mismatch |
| ... |
```

Also write `PAPER_CLAIM_AUDIT.json` for machine consumption.

### Step 4: Print Summary

```
📋 Paper Claim Audit Complete

  Claims verified: 24
  exact_match:     18
  rounding_ok:      3
  ambiguous:         1
  ⚠️ mismatch:      2

  Overall: ⚠️ WARN

  See PAPER_CLAIM_AUDIT.md for details.
```

## When to Run

1. **After `/paper-write`** — first check before improvement loop
2. **After `/auto-paper-improvement-loop`** — recheck if improvement loop changed numbers
3. **Before submission** — final verification

## Integration with Other Skills

### Read by `/auto-paper-improvement-loop` (if exists)

```
if PAPER_CLAIM_AUDIT.json exists:
    read mismatched claims
    fix them as priority items in the improvement round
```

### Advisory, Never Blocking

Same pattern as `/experiment-audit`:
- `PASS` → continue normally
- `WARN` → print warning, continue, flag draft as "check numbers before submission"
- `FAIL` → print alert, continue, but do NOT mark as submission-ready

## Render HTML view (auto, when `RENDER_HTML = true`, default)

After writing `paper/PAPER_CLAIM_AUDIT.md` and `paper/PAPER_CLAIM_AUDIT.json`, invoke `/render-html` on the audit report:

```
/render-html "paper/PAPER_CLAIM_AUDIT.md" --json "paper/PAPER_CLAIM_AUDIT.json"
```

Uses **full review gate** (audit-class artifact; base Codex review is fresh same-family provisional). Output: `paper/PAPER_CLAIM_AUDIT.html` with embedded source SHA256 + `.review.json` sidecar.

**Non-blocking**: if `/render-html` fails (helper missing, secondary Codex agent unavailable, file write error), log the failure and treat the audit as complete — the JSON + MD verdict files are canonical; the HTML view is a human-reader convenience.

Skip if `RENDER_HTML = false` is set in `AGENTS.md` / `CLAUDE.md` or passed as `— render html: false`.

## Key Rules

- **Fresh thread EVERY run.** Never use a continuation reply. Never carry context.
- **Zero executor interpretation.** Only file paths. No summaries.
- **Only raw results.** No EXPERIMENT_LOG, no AUTO_REVIEW, no human summaries.
- **Rounding rule.** Only standard rounding to displayed precision. 84.7% → 84.7% or 85% is OK. 84.7% → 85.3% is NOT OK.
- **Review class.** Base Codex reviewer is same-family provisional; only an overlay may record cross-family accepted.

## Review Tracing

After each reviewer agent call, save the trace following `shared-references/review-tracing.md` (Policy C — forensic; never silently skip). Use `save_trace.sh` (resolved per the chain in `shared-references/integration-contract.md` §2) or write files directly to `.ei/traces/<skill>/<date>_run<NN>/`. Respect the `--- trace:` parameter (default: `full`).

## Submission Artifact Emission

This skill **always** writes `paper/PAPER_CLAIM_AUDIT.json`, regardless of
caller or detector outcome. A detector-negative run (paper has no numeric
claims) emits verdict `NOT_APPLICABLE`; a paper-with-numeric-claims-but-no-
raw-results run emits `BLOCKED`. Silent skip is forbidden — `paper-writing`
Phase 6 and `verify_paper_audits.sh` both rely on this artifact
existing at a predictable path.

The artifact conforms to the schema in `shared-references/assurance-contract.md`:

```json
{
  "audit_skill":      "paper-claim-audit",
  "verdict":          "PASS | WARN | FAIL | NOT_APPLICABLE | BLOCKED | ERROR",
  "reason_code":      "all_numbers_match | rounding_drift | missing_raw_results | ...",
  "summary":          "One-line human-readable verdict summary.",
  "audited_input_hashes": {
    "main.tex":                              "sha256:...",
    "sections/5.evidence.tex":               "sha256:...",
    "/abs/path/to/results/run_2026_04_19.json": "sha256:..."
  },
  "trace_path":       ".ei/traces/paper-claim-audit/<date>_run<NN>/",
  "thread_id":        "<codex mcp thread id>",
  "executor_model":   "codex-gpt-6-astra",
  "executor_family":  "openai",
  "reviewer_model":   "gpt-6-astra",
  "reviewer_family":  "openai",
  "review_independence": "same-family",
  "acceptance_status": "provisional",
  "reviewer_reasoning": "ultra",
  "generated_at":     "<UTC ISO-8601>",
  "details": {
    "total_claims":   <int>,
    "mismatches":     [ ... per-claim issue records ... ],
    "result_files":   [ ... raw files consulted ... ]
  }
}
```

### `audited_input_hashes` scope

Hash the **declared input set** passed into this audit invocation — i.e. the
exact `.tex` files and raw result / config files this run read — not a
repo-wide union and not the reviewer's self-reported subset. If a caller
passed only `main.tex` + a single result file, hash those two files and no
others. The external verifier rehashes these entries; any mismatch flags
`STALE`.

**Path convention** (must match what `verify_paper_audits.sh`
expects): keys are **paths relative to the paper directory** (the arg
passed to the verifier) for in-paper files — so `main.tex`, not
`paper/main.tex` — and **absolute paths** for out-of-paper files such as
external `results/` dirs. The verifier resolves relative entries via
`os.path.join(paper_dir, key)`; prefixing with `paper/` produces
`paper/paper/main.tex` and false-fails as STALE.

### Verdict decision table

| Input state                                           | Verdict          | `reason_code` example |
|-------------------------------------------------------|------------------|-----------------------|
| No numeric claims detected in paper                   | `NOT_APPLICABLE` | `no_numeric_claims`   |
| Numeric claims detected, no raw result files found    | `BLOCKED`        | `no_raw_evidence`     |
| All claims reconcile to raw data                      | `PASS`           | `all_numbers_match`   |
| Minor rounding drift only, no material mismatch       | `WARN`           | `rounding_drift`      |
| Any material mismatch (wrong number, config mismatch) | `FAIL`           | `claim_mismatch`      |
| Reviewer invocation failed (network / malformed)      | `ERROR`          | `reviewer_error`      |

### Thread independence

Every invocation uses a fresh reviewer agent. Never continue a prior audit via
`send_input`. Do not accept prior audit outputs (PROOF_AUDIT, CITATION_AUDIT,
EXPERIMENT_LOG, AUTO_REVIEW summaries) as input to this audit — the fresh
thread preserves reviewer independence per
`shared-references/reviewer-independence.md`.

### Human-readable sibling

`paper/PAPER_CLAIM_AUDIT.md` is written alongside the JSON for readers.
The JSON is authoritative for `verify_paper_audits.sh`; the Markdown
is for humans. The parent skill (`paper-writing` Phase 6) plus the verifier
decide whether the verdict blocks finalization — this skill itself never
blocks; it only emits.