Report Gold Standard Rubric

---
name: report-gold-standard-rubric
description: "The Phase-5 iterate-to-gold-standard loop contract for report-regeneration — score a review-ready draft against 4 dimensions (Accurate HARD FLOOR / Dynamic / Inclusive / Polished), each anchored to a deterministic harness leg with an honest judged residue. Owns the loop mechanics: stop = PASS or plateau(2) or cap(6), monotonic ratchet (revert on any regression), per-node edit budget, N=3 median for judged dims, and the do-no-harm zero-diff fixture. Also owns feedback-instrumentation: classify the client's real peer-review edits substantive-vs-mechanical and feed that back to tune the bars. Use after report-fidelity-harness emits a receipt and before a draft is handed to the human peer reviewer."
---

# Skill: report-gold-standard-rubric

You are running the **Phase 5 — iterate-to-gold-standard** loop (FORGE plan
§6a, binding TB-4a / RT2-F6) for a `report-regeneration` review-ready draft.
This skill turns a `report-fidelity-harness` receipt into a per-dimension
verdict, decides whether the draft needs another revision pass, and — when
it does — enforces the guardrails that keep the loop honest: a hard floor
that is never traded, a ratchet that reverts regressions, a budget that
freezes a stuck node instead of grinding forever, and a cap that always
terminates.

**This skill never claims "the report is correct."** It reproduces the
plugin's honest guarantee (`CLAUDE.md` §1): auto-QA proves *the checked
surfaces*; a PASS here means every checked dimension cleared its bar, not
that a human peer reviewer has nothing left to do. The manual residue this
skill surfaces (`escalate_to_human`, judged-stub dimensions, plateau
escalation) is exactly the mechanism that keeps that claim honest.

## The four dimensions

Each dimension has a **deterministic anchor** (a harness leg or a scanner —
proven, machine-checked) and, for three of the four, a **judged residue**
(behavioral, N=3-median-scored by a model judge). The anchor is a
**necessary condition**: a judged score can never rescue a red anchor.

| Dimension | Bar | Deterministic anchor | Judged residue |
|---|---|---|---|
| **Accurate — HARD FLOOR** (never iterated, never traded) | V1-V6 + period-coherence ALL green | the WHOLE `report-fidelity-harness` receipt | **none.** Fails CLOSED to `"unverified"` — **never** `"pass"` — when V1 is in binding-correctness mode (RT1-F12: no live Power BI data route, so V1 degraded from value-accuracy to binding-correctness). |
| **Dynamic** | zero static fossils; every manifest slot demonstrably rebound; variance narrative direction matches data direction (says "up" iff Δ>0) | V1 leg pass + a sign-consistency check | narrative fluency |
| **Inclusive** | per-format a11y gate green (axe-core HTML / veraPDF Office-PDF); reading order sane; alt-text present *and specific* | axe-core / veraPDF | alt-text *quality*, plain-language — LLM-judged, advisory |
| **Polished** | layout lint clean; render referee pass (Playwright HTML / LibreOffice Office — local for both formats); number/date formats consistent; no orphans (`NaN`, `{{`, empty nodes) | V5 leg (both formats) + a format-consistency scanner | typographic taste, visual hierarchy |

**Why Accurate is categorically different, in code as well as in prose:**
every other dimension collapses to a `[0,1]` judged score once its anchor is
green; Accurate never does — `rubric.score_accurate()` returns exactly one
of `"pass" | "fail" | "unverified"`, and `"unverified"` is reachable **only**
from the anchor, never from a judge. There is no code path in `rubric.py`
that lets a judged score turn Accurate green. See `rubric.py`'s own
docstring on `score_accurate()` for the exact precedence between a genuine
other-leg failure (reports `"fail"` — a proven leak outranks an honest "we
couldn't verify") and the V1-degrade case (reports `"unverified"`).

## Reading a `report-fidelity-harness` receipt

`rubric.py` consumes the receipt shape defined by
[`../../knowledge/fidelity-receipt.schema.json`](../../knowledge/fidelity-receipt.schema.json)
(see also [`../../knowledge/core-architecture-spec.md`](../../knowledge/core-architecture-spec.md)
§5). It reads `receipt["legs"]` (each `{leg, verdict, label, ...}`) and
`receipt["overall_gate"]`. It does **not** call the harness itself — that is
`report-fidelity-harness`'s job; this skill is downstream of it.

Three signals the rubric needs for Dynamic/Inclusive/Polished are **not**
part of `fidelity-receipt.schema.json` (that schema only carries the six
V-legs + period-coherence): the sign-consistency check, the two a11y gates,
and the format-consistency scanner. `rubric.py`'s scoring functions accept
these as **explicit optional keyword arguments** — a bare receipt with none
of them supplied still scores Accurate correctly, and the other three
dimensions degrade honestly (their anchor reports `False`, never a silent
pass) rather than assuming a signal that was never actually run.

```python
from rubric import score_all, passes_bar

scores = score_all(
    receipt,
    sign_consistency_pass=True,       # Dynamic anchor's second half
    a11y_axe_pass=True,                # Inclusive anchor (HTML)
    a11y_verapdf_pass=True,             # Inclusive anchor (Office PDF)
    format_consistency_pass=True,      # Polished anchor's second half
    dynamic_judged=0.91,               # N=3 median, already computed
    inclusive_judged=0.88,
    polished_judged=0.93,
)
if passes_bar(scores):
    ...  # PASS — every dimension's status == "pass"
```

## Loop mechanics

```
score_all(receipt, ...) → per-dimension {status, anchor_pass, judged_score, judged_stub}
                        → terminate(iterations)
                             ├─ PASS      : every dimension status == "pass" (checked FIRST, at
                             │              any iteration count — a cap/plateau that coincides
                             │              with a clean pass is still reported PASS)
                             ├─ cap       : len(iterations) >= 6
                             ├─ plateau   : the last 2 iterations both show "improved" == False
                             └─ otherwise : keep iterating — apply accept_revision() to the next
                                            candidate revision, respecting the per-node edit budget
```

### Stop condition (binding — TB-4a)

**PASS ∨ plateau(2 consecutive no-improvement) ∨ cap(6), whichever fires
first.** `rubric.terminate(iterations)` is the pure, model-free predicate —
see its docstring for the exact precedence. `STATUS_PASS = "PASS"`,
`STATUS_PLATEAU = "plateau"`, `STATUS_CAP = "cap"` are the fixed vocabulary;
plateau and cap both set `escalate_to_human = True` and route the residual to
`architect` — non-convergence is never silently swallowed, and genuine
human-only substantive residue is flagged, never faked as a pass.

### Monotonic ratchet (binding — RT2-F6)

`rubric.accept_revision(prev_scores, new_scores, target_dim)` — accept a
revision **only if** its target dimension improves **and no other dimension
drops**; otherwise revert. Accurate is additionally **never** allowed to
regress from `"pass"`, regardless of which dimension the revision targeted —
a revision that lifts Polished but silently corrupts a verified value node is
rejected outright, not traded.

### Per-node edit budget (binding)

`rubric.node_edit_budget_exceeded(edit_counts, node_id, budget=2)` — any
node revised twice without reaching the bar is **frozen** and flagged to the
human reviewer. That residue is theirs by the draft+QA model (`CLAUDE.md`
§1) — the loop does not grind on a single stuck node past its budget.

### N=3 median for judged dimensions (binding — RT2-F6, noisy-judge robustness)

Every judged dimension (Dynamic/Inclusive/Polished, once their anchor is
green) is scored by **3 independent judge calls**, reduced with
`rubric.judged_median(scores)`; the judge is fingerprinted in the receipt
(per `fidelity-receipt.schema.json`'s `judge_fingerprint` field) for
reproducibility. `rubric.judged_sample_insufficient(scores)` is the teeth:
a caller that tries to trust a median computed from fewer than 3 samples
must check this first — the N=3 rule is not just a docstring, it is a
checkable predicate.

### Do-no-harm zero-diff fixture (binding — RT2-F6, required acceptance test)

**An input already at the bar must exit at iteration 1 with ZERO content
diffs.** This is the loop's do-no-harm invariant: a report that is already
gold-standard is never edited "for the sake of iterating." Concretely — call
`terminate([iteration_1])` where `iteration_1["scores"]` already satisfies
`passes_bar()` and `iteration_1["content_diff_count"] == 0`; the loop must
stop with `STATUS_PASS` at `iterations == 1`, and the caller must never have
applied a revision to reach that state. `tests/test_rubric.py`'s
`TestZeroDiffFixture` is exactly this test, run against `rubric.py` directly.

### Advisory dimensions are not iterated to convergence

The judged residues (narrative fluency; alt-text quality/plain-language;
typographic taste/visual hierarchy) are **advisory** — the loop has no
obligation to grind them to a perfect score once the dimension's own bar is
cleared. Only a red anchor or a below-bar judged median keeps a dimension
`"fail"`; once `"pass"`, further polish on that dimension is out of scope for
this loop (a human reviewer's taste call, not a re-iteration trigger).

## Feedback instrumentation (G0-d — the calibration signal)

Peer review is a **downstream human step**; the plugin emits a review-ready
draft only (`CLAUDE.md` §1, §4 Scope). There is no in-plugin secured human
grader — instead, **the client's real peer-review feedback, over successive
real reports, is the calibration signal that tunes this rubric's bars.**

The contract (FORGE plan §5-P0 G0-d):

1. **Capture** the reviewer's demanded edits per shipped report — every
   change the human peer reviewer requested before the report went out.
2. **Classify each edit substantive vs. mechanical/QA:**
   - **Mechanical/QA** — a wrong value, a stale label, a broken layout, a
     missed alt-text, an inconsistent number format: exactly the class of
     defect one of the four dimensions' deterministic anchors is *supposed*
     to catch. A mechanical edit surfacing in real peer review is a
     **miss** — the anchor should have caught it, or the bar is set too low.
   - **Substantive** — a judgment call about framing, emphasis, narrative
     interpretation, or a domain fact no automated check could verify: this
     is genuine human-only residue, exactly what `manual_residue` on the
     fidelity receipt is honestly enumerating (`CLAUDE.md` §1: "does NOT
     guarantee a human-free-correct report").
3. **Feed the classification back** into the Binding Manifest + this
   rubric: a recurring *mechanical* miss on a given dimension is evidence
   that dimension's bar, anchor coverage, or judged-residue weighting needs
   tightening; a *substantive*-only pattern is evidence the harness + rubric
   are doing their job and the residual is correctly human-owned.
4. **The measured question, over successive real reports:** does auto-QA
   leave the reviewer giving **only substantive** feedback? That trend line
   — not any single report — is the calibration signal. It replaces the
   pre-secured human-grader gate the original G0-d design called for
   (struck per rev. 2 scope change — RT2-F1/critic-P0).

**This is an ongoing calibration instrument, not a hard pre-build blocker.**
It does not gate any release; it tunes the rubric's bars (`DEFAULT_BAR` in
`rubric.py`, the anchor thresholds, which judged dimensions get weight) over
time as real feedback accumulates. Implementation note: the classifier
itself (substantive vs. mechanical) is a downstream, report-regeneration-
specific instrument that consumes real reviewer edits — it is out of scope
for `rubric.py` (which is a pure, receipt-scoring + loop-termination module
with no I/O beyond its own CLI); wire it as a separate script that reads the
shipped-report + reviewer-diff pair and writes its classification alongside
the run's fidelity receipt under `.ravenclaude/runs/`, per the plugin's Run
Artifacts convention.

## Files

| file | role |
|---|---|
| `rubric.py` | pure, stdlib-only, Python 3.9-compatible: `score_accurate` / `score_dynamic` / `score_inclusive` / `score_polished` / `score_all`, `passes_bar`, `judged_median` / `judged_sample_insufficient`, `dimension_score`, `accept_revision` (monotonic ratchet), `node_edit_budget_exceeded`, and `terminate` (the PASS/plateau/cap stop predicate) — plus a CLI (`--receipt` to score, `--iterations` to run `terminate()`). |
| `tests/test_rubric.py` | `unittest`, stdlib-only. Covers the three required acceptance tests (zero-diff fixture exits at iteration 1 unchanged; a rigged never-improving scorer triggers plateau at 2; the Accurate floor fails closed to `"unverified"` on a V1-binding-correctness-mode receipt) plus the monotonic ratchet, the per-node edit budget, the cap stop, and the N=3 median helper. |

## Reuse ledger (no duplication)

- **`refine-to-rubric` / `converge.py`** (`ravenclaude-core`) is the
  domain-neutral Convergence Engine this skill's loop mechanics are modeled
  on (plateau/cap/keep-best, model-free `terminate()`, N-agreement judging,
  never-claim-perfect vocabulary) — **not imported or wrapped**, because the
  two rubrics are shaped differently: `converge.py` grades one weighted
  score across freely-tradeable dimensions; this skill grades four
  independently-gated dimensions, one of which (Accurate) is a hard floor
  that can **never** be traded for gains elsewhere. Anyone extending this
  skill should read `converge.py`'s docstring first — the anti-failure-mode
  reasoning (objective-first, cross-model judge, model-free stop authority)
  applies verbatim here.
- **`agent-quality-rubric`** (`ravenclaude-core`) is the sibling pattern for
  an anchored, multi-dimension scorecard with remediation guidance — read
  for the "quote, don't vibe-grade" review discipline when a human is
  grading the judged residues by hand.
- **`report-fidelity-harness`** (this plugin, Phase 2) owns V1-V6 +
  period-coherence themselves; this skill only *reads* the receipt they
  emit. Do not re-implement any harness leg here.
- **`report-qa-gate`** (this plugin) is the downstream consumer that
  assembles the per-format tiered report + manual-residue checklist a human
  reviewer sees; it calls this skill's `score_all` to get the four
  dimension verdicts and this skill's `terminate` to decide whether another
  revision pass is warranted before handing the draft off.

## Acceptance tests (binding — RT2-F6)

- A seeded-mediocre regeneration reaches the bar within the cap (6
  iterations) — drive `terminate()` with a monotonically-improving
  iteration history and assert `STATUS_PASS` before `iterations` hits 6.
- A rigged never-improving scorer triggers the plateau escape (must-fail
  half) — `tests/test_rubric.py::TestPlateauEscape`.
- The zero-diff fixture exits at iteration 1 untouched —
  `tests/test_rubric.py::TestZeroDiffFixture`.
- The Accurate floor fails closed to `"unverified"` when V1 is in
  binding-correctness mode — `tests/test_rubric.py::TestAccurateHardFloor`.

Run: `python3 plugins/report-regeneration/skills/report-gold-standard-rubric/tests/test_rubric.py`