Spec Qualification

---
name: spec-qualification
description: Qualifies an implemented change, Builder Execution Report, and git diff against its governing behavioral specification (contracts, rules), anti-phantom call paths, and Architecture Decision Records (ADRs) to issue a binary Verified or Unverified verdict without executing builds directly.
---

# Spec Qualification

## Overview

`spec-qualification` audits an implemented change to confirm that it strictly fulfills its governing behavioral specifications and preserves architectural invariants.

It is a **contractual acceptance gate**, distinct from general code review (`code-review-and-quality`):
- `code-review-and-quality` evaluates **code craftsmanship & health** (readability, simplicity, security, maintainability).
- `spec-qualification` evaluates **contractual conformance** (rule traceability, assertion authenticity, ADR invariants, anti-phantom call paths).

### The Independent Inspector Rule
The evaluator operates as a **non-executing inspector**:
- **Cognitive & Context Isolation**: The evaluator must be independent of the author/implementer and operate with read-only inspection tools.
- **Evidence-Based Inspection**: The evaluator audits the code diff and execution receipts (test logs, pass/fail counts, coverage evidence). It does not compile or run test runners directly.
- **Dual Audit Pairing**: Run concurrently with `code-review-and-quality` so that contract qualification and code craftsmanship are evaluated together.

### Command-Backed Reviewer Evidence, Coverage Checklists & Convergence
To prevent hallucinated quotes, wrong line numbers, unbacked coverage claims, or non-converging review loops:
1. **Command-Backed Citations**: Every reviewer must copy every quote and every line number directly from the output of a command (`grep -n`, `sed -n`) or line-numbered file view executed in the same run, naming the exact file and line number.
2. **Post-Compaction Re-Read**: After a context compaction, or when resuming from a summary, a reviewer must run its inspection commands again before reporting any quote, line number, status, or structural reference (such as a rule ID `R1..Rn` or a conformance table row). It must never write its report without tool calls after a compaction.
3. **Checklist-Backed Coverage Evidence & Review Lanes**: Every count, breakdown, or coverage claim (e.g., "100% Example Mapping rule coverage" or "all N rows verified") must come from command output of the same run, and the report must list the files and commands inspected. The orchestrator must pass the list of specification rules (`R1, R2, ...`) or conformance rows as an explicit checklist, and the reviewer must answer each checklist item with the command it ran and the result. When two or more reviewers run, the orchestrator may assign each reviewer a focused review lane whose checklists together cover the whole scope.
4. **Class-Once Findings & Invariant-Complete Remediation**: Each reviewer finding about a defect must state the invariant that the defect breaks and list every sibling site in the reviewed artifacts where the same invariant is broken or must hold. When routing to PB-5 remediation, the brief must carry the invariant and every sibling site so the Builder fixes all of them in one round.
5. **Orchestrator Verification, Bounded Retries & Convergence**: Before issuing a canonical `VERIFIED` or `UNVERIFIED` verdict, the orchestrator must verify every cited quote, line number, status, structural reference, and coverage count against its source file (and tool trace where available). Any missing report or unreviewed scope must be re-run **at most once** before asking the operator, and the gate must never pass with unreviewed scope. A gate runs at most `review.max_rounds` fix rounds (default 3) on one artifact set; if the last allowed round still requires a change, or if a finding repeats a previously fixed causal theme, the orchestrator stops, records `converged: false` (when at the round limit), and escalates to the operator with the five-element format (original requirement, requested change and whether it adds to the contract, smallest compliant change, consequences of accepting/declining, and recommendation).

---

## Required Ingest Artifacts

Before qualifying a change, obtain and review:
1. **Normative Behavioral Specification**: System operations, postconditions, and Example Mapping rules ($R_1, R_2, \dots$).
2. **Governing Architecture Decision Record (ADR)**: Boundary invariants, prohibited shortcuts, and layer rules.
3. **Execution Report**: Test runner output, pass/fail counts, and commit hashes.
4. **Code Diff**: All modified production and test files.

---

## The Four Qualification Criteria

### 1. Rule Traceability (100% Coverage)
- Every specification rule ($R_1, R_2, \dots$) must map to an explicit, passing automated test.
- Negative edge cases and boundary conditions must be explicitly verified with expected error states.
- Zero unverified or skipped specification rules allowed.

### 2. Assertion Authenticity
- Inspect test source code: tests must prove observable postconditions, state transitions, or domain events.
- Reject vacuous assertions (e.g. `expect(true).toBe(true)`), tautologies, or assertions that test only mock mechanics rather than domain outcomes.

### 3. ADR Invariant Fidelity
- Inspect production code diff against governing ADRs.
- Verify that declared architectural boundaries (e.g. Hexagonal ports & adapters, domain isolation, aggregate consistency boundaries) are preserved.
- Reject unauthorized shortcuts, bypassing of defined adapters, or unapproved dependencies.

### 4. Anti-Phantom Call Paths (Live Integration)
- Trace the static and dynamic call graph of all newly introduced production functions, methods, and configurations.
- Verify that every new capability has an active caller in production runtime paths.
- Any function or method called solely by unit tests or test mocks is dead code / a phantom feature and must be rejected (`UNVERIFIED`).

---

## Workflow

```text
Execution Report + Diff + Rule Checklist (R1..Rn)
           │
           ▼
[Step 1: Rule Traceability] ──────► Map every spec rule (R1, R2...) to tests with command + result
           │
           ▼
[Step 2: Assertion Authenticity] ─► Verify tests prove real domain postconditions (file:line citations)
           │
           ▼
[Step 3: ADR Invariant Audit] ────► Verify declared architectural constraints (file:line citations)
           │
           ▼
[Step 4: Anti-Phantom Audit] ─────► Verify live production callers for all new code
           │
           ▼
[Step 5: Citation & Coverage Check]► Orchestrator verifies all quotes, lines, and checklist items
           │
           ▼
[Step 6: Verdict & Report] ───────► Issue VERIFIED or UNVERIFIED with report
```

---

## Verdicts & Return Routing

Verdicts are strictly binary: **`VERIFIED`** or **`UNVERIFIED`**.

### When `VERIFIED`
All specification rules have authentic passing test evidence, ADR invariants are preserved, and anti-phantom call path checks pass.
* **Action**:
  1. Update specification frontmatter: set `status: implemented`, and record verification metadata `verified: { by: "human:<id>", at: "<ISO-8601>" }` or `{ by: "process:qualification", at: "<ISO-8601>" }`.
  2. Author the durable qualification report in `docs/verification/qualification-<module-id>.md`.
  3. Re-index stage directories using OKF utility (`bun scripts/okf.ts index docs`).
  4. Unblock downstream integration or trigger pull request publication (`create-pr`).

### When `UNVERIFIED`
Route findings deterministically to the responsible phase:

| Failure Category | Root Cause | Return Destination | Action Required |
| :--- | :--- | :--- | :--- |
| **Implementation Defect** | Test failed, vacuous assertion, missing edge case | **Implementation** | Fix implementation or add genuine test assertions. |
| **Phantom Code** | New production function has no caller outside unit tests | **Implementation** | Connect capability to live execution path or remove dead code. |
| **Architectural Breach** | Violated ADR pattern, unauthorized dependency | **Implementation** | Refactor code to conform to the governing ADR. |
| **Specification Gap** | Contradictory rules, missing error contracts | **Requirements / Contracts** | Clarify or amend the specification rules. |
| **Architectural Defect** | ADR invariant proves infeasible or contradictory | **Architecture / ADR** | Author an amending ADR. |

---

## Qualification Report Template

Save all qualification reports to `docs/verification/qualification-<module-id>.md` conforming to [OKF v0.2](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md):

```markdown
---
type: "Qualification Report"
title: "Qualification Report: [Slice / Module Name]"
description: "Specification qualification audit for [Slice/Module Name] against [Spec ID] and [ADR-XXX]."
status: "accepted" # accepted if VERIFIED, draft if UNVERIFIED
date: "YYYY-MM-DD"
requirements: "[path/to/docs/requirements/<module-id>.md]"
contract: "[path/to/docs/contracts/<module-id>.md]"
governing_adr: "[path/to/docs/decisions/adr-XXX-*.md]"
implementation_revision: "[commit SHA]"
verdict: "[VERIFIED | UNVERIFIED]"
generated: { by: "agent/thinker", at: "YYYY-MM-DDTHH:MM:SSZ" }
verified: { by: "human:sirius", at: "YYYY-MM-DDTHH:MM:SSZ" }
---

# Qualification Report: [Slice Name]

## 1. Executive Summary
- **Governing Spec**: `[spec path]`
- **Governing ADR**: `[adr path]`
- **Implementation Revision**: `[commit SHA]`
- **Final Verdict**: **[VERIFIED | UNVERIFIED]**

## 2. Behavioral Compliance Matrix
| Rule # | Rule Description | Test Location | Test Status | Audit Assessment |
| :--- | :--- | :--- | :--- | :--- |
| **R1** | [Happy path description] | `tests/...::test_...` | PASS | Valid assertion of postcondition |
| **R2** | [Edge case description] | `tests/...::test_...` | PASS | Valid rejection handling |
| **R3** | [Boundary description] | `tests/...::test_...` | PASS | Valid boundary check |

## 3. Test Integrity & Assertion Authenticity
- [ ] Every rule maps to a genuine passing automated test.
- [ ] Assertions verify observable state changes/events, not mock mechanics.
- [ ] Zero vacuous, tautological, or skipped tests.

## 4. Architectural Invariant Audit (ADR Compliance)
- [ ] Governing ADR invariants and boundaries are strictly maintained.
- [ ] Domain logic remains isolated from transport/persistence.
- [ ] Zero unauthorized dependencies or bypassed adapters.

## 5. Anti-Phantom Call-Path Audit
- [ ] Every new production function/method has an active caller in live runtime code.
- [ ] Zero functions called solely by unit tests or mocks.

## 6. Findings & Return Path (if UNVERIFIED)
- **Failure Category**: `[Implementation Defect | Phantom Code | Architectural Breach | Specification Gap | Architectural Defect]`
- **Return Destination**: `[Implementation | Requirements / Contracts | Architecture / ADR]`
- **Action Items**:
  1. [Specific issue that must be addressed before re-qualification]

## 7. Conclusion
[Clear summary explaining why the change is verified or what steps are required next.]
```