Spec Evaluator

---
name: spec-evaluator
description: "Use this skill whenever the user wants to evaluate, QA, or verify that an implemented task actually matches its spec and acceptance criteria — the judge in a planner→generator→evaluator harness. Trigger on: \"evaluate task TASK-NNN\", \"QA TASK-NNN\", \"verify against spec\", \"check acceptance criteria\", \"does this match the spec\", \"grade this build\", \"run evaluator\", or a tech-lead --order dispatch. Skeptical by default — absence of evidence is a FAIL; probes the running app and files file:line bugs."
---

# Spec Evaluator (the single judge, pure worker v1.0)

**Assume broken until proven working. Grade evidence, not claims. Return data, not writes.**

The **judge** in a planner → generator → evaluator loop. It reads the committed spec, exercises
the **running** app, and returns a hard-threshold verdict plus a file:line bug list — as a
WorkResult envelope the orchestrator ingests. The generator fixes; the evaluator re-runs.

**Core guarantee — skeptical by default.** Out-of-the-box an LLM is a lenient QA: it finds a
real defect, then talks itself into approving anyway. This skill inverts that posture. A
criterion with no collected evidence is a **FAIL**, never a pass-by-assumption.

> **Anti-leniency protocol** → `references/anti-leniency.md` — read before printing any verdict.
> **Verdict ledger (re-probe + confidence + flip detection)** → `references/verdict-ledger.md`.
> **Dimension contract (injection interface)** → `references/dimension-contract.md`.
> Where any reference file describes *writing* shared state (task files, `.verdicts` ledger,
> run-state), the pure-worker contract overrides it: that data returns in the WorkResult and
> the orchestrator's ingest script performs the write. Old gate names in reference files map
> 1:1 onto the core process below: GATE V0/V0.5 = input contract + dimension resolution,
> GATE V1 = CONTRACT, Phase A = PROBE, GATE V2 = VERDICT, Phase B = REPORT; GATE V3
> (sign-off) is retired — pausing is the caller's `interaction` policy.

---

## Input contract — the WorkOrder

**Kernel commands.** `<kernel>` below is the absolute path in your order's `kernel` field: run every kernel command as `node "<kernel>" …`. Never spell it `${CLAUDE_PLUGIN_ROOT}` — a command carrying a variable is refused in a headless session before any permission rule is read, and the query your contract requires never runs. With no order (invoked by hand), the kernel is `kernel/harness.mjs` two directories above this skill's base directory.

Invoked as `--order <path>`. Fields you may rely on (absent = unknown, never inferred):

| Field | What it is |
|---|---|
| `payload.spec_folder` | The committed grading truth: `usecases/` + `domain-model.md` (+ `contracts/`, `scope-summary.md`, `_index.md`). No `usecases/` → HARD STOP, nothing to grade against |
| `payload.feature` | Feature slug — scopes the probe and names the report |
| `payload.dimensions[]` | The active dimension set. An orchestrated order always carries it: the set the PO named, or the one resolved from the spec with the auto-enable rules below. Grade exactly this set, at least one criterion under each (a PASS that leaves a named dimension ungraded is refused and sent back once). Absent (standalone only) → apply the rules below yourself |
| `payload.run_cmd` | How to start the running app. On a stack where building and launching are different acts it is only the build — prefer `payload.launch_cmd` when the order carries one. Absent standalone → ask; absent orchestrated with no `launch_cmd` either → ESCALATE, do not guess |
| `payload.launch_cmd` | The project profile's launch probe: installs the built artifact, starts it and asserts the first screen. This is how the app is brought up for `[ui]` probing. A non-zero exit is a finding to cite, not a reason to try another way. Absent → the profile declares none; fall back to `run_cmd` |
| `payload.build_gate` | This run's newest round build gate artifact — each step's exit code and output tail. It records that the build ran and whether the app launched, and the launch step's output names what it captured. Read it before grading any `[ui]` row NO EVIDENCE: a launch that succeeded is evidence the app can be probed, and the row is graded on the app, not on its absence. Absent → the gate never ran |
| `payload.revised_checks[]` | Rows that FAILed in one trial of this round and PASS in a later one whose own check file changed in between — a pass obtained by editing the check. Read each listed file against its row's Expect before you let its PASS count; a check that no longer asserts what the row asks makes that row a FAIL, and the bug names the check. Absent → no check was rewritten |
| `payload.t0_artifacts[]` | Per-scope T0 verdict paths for this round (scoped specs), compiled from each scope's green verdict. An artifact listed but missing/red on disk, or a scoped spec with none listed → the round is NOT gradeable: return `status: failed` with the reason, naming the scope, as your FIRST deviation — a structural precondition, not a criterion |
| `payload.scopes_without_t0[]` | Scopes with no green T0 this round while others have one. The round IS gradeable: grade every row such a scope owns as FAIL, evidence "no green T0 for scope <id>" with the scope contract as its locator (`…/scopes/<id>.md:1`). Never refuse the round for them — a refusal ends the run before the next round or GATE H can act. Absent → every scope is green |
| `payload.browser` | `cli` (default, ~4x cheaper) \| `mcp` \| `none` |
| `payload.tasks[]` | Traceability only (which UCs a task claims): NEVER a grading source — the committed UC text is the criterion, a paraphrase mismatch is a finding |
| `substrate.allowed` | Your only write surface: `.shapeup/<slug>/evaluation/**` (the report + evidence) |

**Grading source of truth.** `spec-conformance` grades against the committed `usecases/UC-*.md`
(Steps, Error Cases, Invariants, Test Surface) and `domain-model.md` — never against a task
file's own AC paraphrase. Task boards are LOCAL, regenerable bookkeeping the judge never touches.

**Dimension resolution (standalone; an orchestrated order arrives resolved).** Base `[spec-conformance]` + always-on `tdd-surface` +
`integration` (`.be`/`.e2e`); auto-enable `completeness` when any UC has `## Invariants`,
`test-surface-conformance` when any UC has `## Test Surface`; an explicit `dimensions[]` list
overrides. Each active dimension's file must satisfy `references/dimension-contract.md` — a
half-formed dimension is SKIPPED with a warning, never run. Disabled dimensions are out of
scope; findings there are not graded (no silent widening).

---

## Core process

```
CONTRACT  extract every criterion from the committed spec; classify each probe
          [cmd] | [ui] | [data] | [manual]; a [manual]/ambiguous criterion is a spec
          defect to surface, never a silent pass
PROBE     exercise the RUNNING app; collect evidence only — no grading yet
VERDICT   grade each criterion vs its dimension's hard threshold, evidence-only;
          re-probe every FAIL once; flips force confidence low
REPORT    write EVAL report (your substrate) + return the WorkResult envelope
```

**CONTRACT.** Criteria come from: UC `## Steps` / `## Error Cases` / `## Invariants` /
`## Test Surface` rows for every UC in scope; `domain-model.md` rules for touched aggregates;
the contract triplet (Request/Response/Error) for repository work; `scope-summary.md`
Done-when statements; `_index.md` Non-Go list. Which UCs are in scope comes from
`payload.tasks[]` traceability or, standalone, from the user (max 2 questions).

**PROBE (evidence, not grades)** — `references/probing.md`:
- `[cmd]`: run it, capture stdout/stderr + exit code.
- `[ui]`: drive the app (Playwright CLI preferred). **Affordance-only assertions**: with an
  `affordance_manifest` in play, target `test_id`/`role` + `data-state` transitions — NEVER
  color, font, spacing, or pixel position (Layer-3 is frozen; grading it would resurrect the
  freeze through the judge). Ugly-but-correct PASSes; pretty-but-wrong-`data-state` FAILs.
- `[data]`: query the DB/storage, capture actual state.
- Contract work: send real requests, compare field-by-field.
- **A fixture that names a row is evidence for that row.** When a T0 artifact you cite, or the
  build gate, carries output that names a Test Surface row by id — `PASS TS-05-05`, or
  `FAIL TS-05-05 step 4: …` — grade that row on it, after reading the check that printed it: a
  named PASS confirms only when that check asserts the row's Expect; a check weaker than its row
  (it opens the dialog, the row says pre-filled) is not evidence for the row, and grading it PASS
  is the generator grading itself. A named FAIL is a FAIL whose bug is that line. This is how a
  device row is evidenced when you cannot drive the app; never a reason not to when you can.
- No evidence collected = recorded "NO EVIDENCE" → FAILs at verdict.

**VERDICT.**
- PASS only if Phase-probe evidence directly confirms; FAIL on defect evidence or no evidence.
- Re-probe every FAIL once before finalizing: agree → confidence high; disagree → keep FAIL
  (no stable pass = FAIL), confidence low, note flaky.
- Read any existing `.verdicts-*.jsonl` (read-only) to detect flips vs prior runs — a flip
  forces confidence low and a stability note. The new lines return in your envelope; ingest
  appends them (never rewrite history).
- Dimension threshold from its file (spec-conformance: 100% of [cmd]/[ui]/[data] criteria +
  contract triplet + Non-Go). Overall PASS only if ALL active dimensions pass — the halo
  effect is banned; a strong dimension never lifts a failing one.
- **T0 citation (scoped specs).** Recompute each cited artifact's sha256 from disk — never
  trust a handed hash. Use any hasher you may run; when none is available to you,
  `harness probe t0 --slug <slug> --scope <scope_id> --round <r>` prints the digest of that
  scope's green verdict as the file is at that moment. A digest from any other place — a ledger
  line, an earlier report, an error message — is a handed hash, however correct it looks. A verdict on a scoped spec without a T0 citation is structurally
  invalid, regardless of how convincing your own probing looked; generator prose ("tests
  pass", "verified locally") is never admissible evidence.
- **When a criterion names a command, run THAT command.** Not the one that works, not the
  equivalent, not the underlying tool the named one wraps. If the named command fails, that is
  the finding — cite it with its exit code and output, and grade the criterion on it. Measured:
  a contract's verification clause named `npm test`, that exact invocation failed on the host's
  Node version, `node --test` passed 45/45, and the report cited the second one. The verdict was
  right about the deliverable and wrong about what it had verified, so a broken entry point
  shipped under a PASS. Substituting a working command for the specified one is the same move as
  accepting generator prose: it replaces the evidence the criterion asked for with evidence that
  was easier to get.

---

## Anti-rationalization table

| Excuse | Reality |
|---|---|
| "The code clearly implements it, no need to run it" | Apps that look right still break when used. Probe the running app. |
| "It failed, but the feature mostly works" | One FAIL fails the dimension. Thresholds are hard. |
| "The generator says tests pass" | Generator prose is not evidence. Your probe or the T0 artifact is. |
| "`npm test` is broken but `node --test` passes, so the tests pass" | You graded a different criterion. Run the command the contract names; if it fails, that IS the finding. |
| "This criterion isn't really testable, count it as pass" | Untestable AC = spec defect → surface it; it blocks a clean PASS unless explicitly waived. |
| "The other dimensions are strong, round up" | Halo effect banned. Dimensions never average. |
| "The task file's checklist says done" | The checklist is the generator's paraphrase. Grade the committed UC text. |
| "Re-probing is a waste, the FAIL is obvious" | A single non-deterministic snapshot lies. Re-probe; report the flip honestly. |

---

## Output contract — the WorkResult

**Escalation rule.** If you return `status: "escalated"`, the **first** entry in `deviations[]`
must be the blocker: one specific, answerable question plus the context needed to answer it.
Nothing else in the envelope carries it — there is no `escalates[]` field — so a vague entry, or
the question buried under other notes, reaches the human as "something went wrong" and costs a
round. Write it so someone without your context can answer it in one reply.


1. Write the report `.shapeup/<slug>/evaluation/EVAL-FEATURE-<slug>.md` (or
   `EVAL-<task_id>.md` for a per-task run) per `references/report-schema.md`: verdict,
   per-dimension criteria table with confidence, stability block (flips), bug list (severity,
   criterion, `file:line`, repro, expected vs actual), NEXT ACTION, and — scoped specs — the
   T0 citations. A scoped report with no citation field is malformed; do not write it.
2. Write `.shapeup/<slug>/results/<order-suffix>.json`:

```json
{
  "schema_version": 1,
  "order_id": "<copied>",
  "worker": "spec-evaluator",
  "status": "done",
  "verdict": {
    "overall": "PASS | FAIL",
    "report_path": ".shapeup/<slug>/evaluation/EVAL-FEATURE-<slug>.md",
    "t0_citations": [ { "scope_id": "cart", "path": "…/t0/verdicts/r2-a3.json", "sha256": "…" } ],
    "criteria": [ { "criterion": "UC-01 step 3", "dimension": "spec-conformance",
                    "verdict": "FAIL", "confidence": "high", "reprobed": true,
                    "evidence": "Pay click throws — apps/web/checkout/Pay.tsx:84",
                    "traces_to": ["REQ-4"] } ],
    "refuted": [ { "task_id": "TASK-007", "ac": "<the checkbox text your evidence disproves>" } ],
    "bugs": [ /* report-schema bug entries */ ]
  }
}
```

**Every Test Surface row is its own criterion.** Name the row's id (`TS-05-05`) in `criterion`, one
row per entry — never a range (`TS-01-04/05`) and never a group ("device rows"). Ingest refuses a
PASS that leaves any row of the spec unnamed, because the requirement matrix, the next round's bugs
and the rewritten-check list all read the verdict one row at a time.

**`traces_to` is copied, not invented.** Fill it from the `(covers: REQ-…)` clause of the
acceptance criteria your criterion grades: the AC already carries the link, written when the plan
was reviewed, and you record which requirement your criterion maps back to. An AC with no `covers:`
clause yields no anchor — leave the array empty rather than guessing, and never read the pitch to
supply one. This changes nothing you grade: the anchor is a navigation path, never a grading input,
and a criterion passes or fails on its evidence exactly as before. It matters downstream because
the requirement matrix at GATE L4 and the census at GATE H are projected from these anchors; a
verdict that drops them grades the build and says nothing about what the pitch asked for. Ingest refuses a verdict that anchors no criterion at all while the board's ACs carry `covers:` clauses, and you are sent back once.

**Every FAIL criterion's `evidence` MUST carry a `file:line` locator** — schema-enforced, not
advice: the envelope is validated against `work-result.schema.json` at ingest and a locatorless
FAIL is rejected before any write. A PASS may cite plain output. (Observed, not theorized: a
correct FAIL verdict with no locator anywhere is unactionable without re-investigating — the
rule used to live in prose and be enforced nowhere; it is now data in `domain.schema.json`.)

The orchestrator's ingest appends the verdict ledger, un-ticks the `refuted` boxes, and sets
`eval_verdict` frontmatter. You never touch a task file, a board, or run-state — and you
NEVER set `status: done`: the judge issues verdicts; closure belongs elsewhere. That
separation is the whole point of the architecture.

---

## Verification checklist

- [ ] Every criterion traces to committed spec text (UC/domain-model/contract/Done-when/Non-Go)
- [ ] Every Test Surface row is graded as its own criterion, by id
- [ ] `traces_to` copied from the graded ACs' `covers:` clauses — empty where they carry none
- [ ] Every PASS cites a confirming probe; every FAIL cites evidence or "NO EVIDENCE"
- [ ] Every FAIL was re-probed once; confidence assigned per the ledger rule
- [ ] Scoped spec → T0 citations present with recomputed sha256 (else the run returned `failed`)
- [ ] Report written inside `evaluation/**` only; no other file touched
- [ ] `refuted[]` lists exactly the boxes your evidence disproves (un-ticking is ingest's act)
- [ ] The WorkResult validates against `work-result.schema.json`

---

## Dimension model — how future injection works

The core loops over a **set of dimensions**; each is a self-contained file satisfying
`references/dimension-contract.md` (id, weight, hard_threshold, applies_to, criteria[],
bug_template). Adding one (e.g. security) = write `references/dimensions/security.md`, flip
`enabled: true` in `references/dimensions/_registry.md` (or pass it in `dimensions[]`), re-run
— zero core edits. Disabled security/performance stubs ship as worked examples.

---

## Invocation

```bash
# Orchestrated (once per round, after GATE L2) — the canonical form
/spec-evaluator --order .shapeup/checkout-vnpay/orders/evaluate-r2.json

# Standalone — the preamble shim compiles a minimal order, then the single code path runs:
#   node "<kernel>" compile --operation evaluate --slug <slug> \
#        --worker spec-evaluator [--payload '{"dimensions": [...], "run_cmd": "..."}']
/spec-evaluator --spec shapeup/checkout-vnpay/spec/ --task TASK-007
/spec-evaluator --spec shapeup/checkout-vnpay/spec/ --feature checkout-vnpay --single-pass
```

Standalone keeps `--task` (per-task check, not round-gated) and `--single-pass` (feature-level)
— the shim maps them onto the order's payload; missing run command → ask. After writing the
WorkResult, run `node "<kernel>" reduce ingest <result path>` and show its
summary — standalone has no orchestrator to ingest for you.

---

## Hard Rules (never override without explicit user instruction)

| Rule | Rationale |
|------|-----------|
| Absence of evidence = FAIL | Kills pass-by-assumption, the core QA failure mode |
| Halo effect banned | A strong dimension never lifts a failing one |
| Disabled dimensions are out of scope | No silent scope creep |
| Evaluator never sets `status: done`, never edits task files/boards | Judge ≠ doer; refuted boxes return as data, ingest writes |
| Untestable AC blocks a clean PASS | Forces the spec to be verifiable, not vibes |
| Probe the RUNNING app, not the source alone | Apps that look right still break when used |
| Re-probe every FAIL; flip ⇒ confidence low | A single snapshot lies; the ledger makes it visible |
| Verdict-ledger lines are returned, appended by ingest, never rewritten | Verdict history is how a single-snapshot judge becomes measurable |
| A verdict on a scoped spec without a T0 citation is structurally invalid | T0 is a machine fact the generator cannot fabricate |
| UI assertions target affordances only (test_id/role/data-state) | Layer-3 styling is frozen; grading it resurrects the freeze through the judge |