Validation Pipeline

---
name: validation-pipeline
domain: meta
triggers:
  - "loaded by every *-lead agent via skills frontmatter"
  - "selecting validators per tier (Workflow validate phase primary; validator_lg.py standalone / HITL)"
  - "interpreting validator output verdicts + canonical envelope"
  - "deciding when to pause on critical findings (--interrupt-on-critical)"
description: |
  [PROTOCOL] Cross-cutting validation chains. Defines which
  validators to run, in what order, with which triggers, across marketing /
  dev / design domains. Preloaded by lead agents (marketing-lead, dev-lead,
  design-lead). Pure reference — no triggers, not invoked by secretary.
---

# validation-pipeline

Shared validator-orchestration playbook for the three agency domains. Defines WHICH validator runs WHEN and HOW it is recorded in `engagement/validation-log.md`. In the agency cascade the **`engagement-workflow` validate phase** runs the applicable validators (by agentType), adversarially verifies each finding, and writes the `validation-outputs/*.json` proof files; this skill is the reference that phase follows. (A lead run standalone, or a focused mid-engagement check, may invoke validators via the Task tool or `validator_lg.py` — see §"Execution patterns".) This skill is read, not called.

## Core principles

1. **Validators run before handoff, not after.** On M/L tier, adversary (`adversary_lg.py --consilium {M|L}`) provides the second-opinion check at acceptance time, but it does NOT replace lead's pre-handoff validator pass. Lead's first pass must already be complete and logged.
2. **Validation log is append-only.** One entry per validator per run, with verdict + findings count + resolution if findings addressed.
3. **Findings trigger re-dispatch, not hand-wave.** If validator flags ≥1 issue above "minor", dispatch the specialist who produced the artefact for fix, then re-run the validator on the fix ONLY.
4. **Scope to what changed.** Don't re-validate artefacts untouched by the engagement. Cost-efficiency matters — adversary on M/L is the second-opinion safety net, not a substitute for missing pre-handoff validator runs.

## Log format

Two artefacts work together: a per-validator JSON output (proof of run) and a markdown log indexing them.

### `engagement/validation-outputs/{validator}-iter-{N}-{timestamp}.json` (mandatory)

Every validator run produces an output file in this directory. The file is the validator's actual output (verdict, findings, metrics). Without this file, the run is presumed not to have happened — director rejects with `validator output missing — likely not run`.

Naming: `{validator-name}-iter-{N}-{YYYYMMDD-HHMMSS}.json` (e.g. `code-reviewer-iter-1-20260506-143211.json`).

Contents are the validator agent's JSON return value, captured verbatim.

### `engagement/validation-log.md` (index, append-only)

```markdown
# Validation log — {engagement-name}

## Iteration 1

### {validator-name}
- Ran on: {artefact path(s)}
- Triggered by: {reason — mandatory rule / risk flag / user criteria}
- Verdict: {clean | N findings}
- Findings summary: {short bullet list if any}
- Resolution: {specialist re-dispatched + fix applied + re-run clean | deferred with justification}
- **Output file**: `engagement/validation-outputs/{validator}-iter-1-{timestamp}.json` (REQUIRED — director cross-checks file exists)

### {next validator}
...

## Iteration 2
(appended after director reject + rework)
...
```

### Lead duty: capture validator output to file

Every Task-tool dispatch to a validator agent must end with the lead writing the agent's JSON return value to `engagement/validation-outputs/{name}-iter-{N}-{timestamp}.json`. This is non-optional. The lead's prompt to the validator should explicitly ask for JSON return value (matching the agent's documented schema).

If a validator returns text instead of JSON (older agent / human-readable mode) — wrap it: `{"format": "text", "verdict": "...", "raw": "..."}` and save. The wrapper is acceptable; missing file is not.

**Per re-run, not just per validator.** A validator RE-RUN after rework at iteration N writes its OWN `{validator}-iter-N-{ts}.json` — a prior `iter-1` output does NOT satisfy an `iter-2` re-run. Every `(re-run)` entry in `validation-log.md` must cite that `output:` path. `handoff-precheck.py`'s per-validator freshness check WARNs on a `(re-run)` entry whose `{validator}-iter-N` output is missing (claimed-without-proof, seen in the field: a logged `test-reviewer (re-run)` with no `test-reviewer-iter-2` slipped past the old any-iter-N check).

### Concurrency rules (preventing race corruption)

`validation-log.md` is written by the **lead only**. Specialists do NOT append to it directly — the lead aggregates after each Task return. This avoids POSIX `O_APPEND` non-atomicity for messages > 4 KB.

`validation-outputs/{name}-iter-N-{timestamp}.json` is per-validator-per-iteration with timestamp suffix → collision-free even under parallel dispatch. Each lead-side write is a single `Path.write_text(...)`, atomic.

`executor-reports/{name}.md` is per-specialist (one file per specialist). When the lead re-dispatches the same specialist for iter N+1, the specialist appends `## Iteration N+1` section — never overwrites prior. Reads happen between dispatches, so concurrent writes from the same specialist do not occur.

If the lead must dispatch ≥2 specialists in parallel that both touch the same artefact (rare, usually a project-side file like `tech-spec.md` lock), explicitly serialise — protocol does not currently model file-level locks. For pipeline parallelism, ensure each specialist's outputs land in disjoint paths.

### Director duty: cross-check files exist

For every validator listed in `validation-log.md`:
- File exists at the cited path? If no → REJECT `validator output missing — likely not run`.
- File parses as JSON? If no → REJECT `validator output corrupted`.
- File's `verdict` field matches what the log claims? If diverges → REJECT `validator-log discrepancy: log says {X}, output file says {Y}`.

Mechanical check is part of `handoff-precheck.py` (sub-check `validator-outputs`).

### Lead self-recovery from broken validator output (Tier 14)

Retry-on-parse-failure is absorbed into `validator_lg.py`'s retry edge: when a validator's output fails to parse as JSON or is missing required fields (`verdict`, plus `methodology` for the numerical validators `accessibility-validator | performance-validator | security-auditor | ux-review | anti-pattern-detector`), the graph automatically re-dispatches that validator. Up to 2 retries per validator per iteration before escalation. Use `--resume` to skip already-clean validators on re-run.

```bash
python ~/.claude/scripts/validator_lg.py engagement/ --auto --resume
```

Anti-pattern: editing the JSON file by hand to "fix" it. Re-dispatch is the only valid recovery — hand-edits hide the original validator's actual finding.

## Domain matrix — mandatory validators

### All domains, before the waves run

| Condition | Validator | Owner |
|---|---|---|
| A `tasks/` set was produced (any domain) | `anti-pattern-detector` with `mode=tasks` | mandatory when `tasks/` exists |

This one runs BEFORE dispatch, not after it, and it is the only entry in this file that does. A decomposition defect (umbrella task, missing source anchor or common objective, dropped non-goals, same-shape work split across one specialist seat each) costs one edit here and a whole wave once the specialists are already running. Rubric: `~/.claude/skills/task-decomposition/references/decomposition-integrity.md`. The `mode=diff` and `mode=executor-reports` runs of the same validator stay dev-only and stay after the waves; see the Dev table below.

### Marketing

| Condition | Validator | Owner |
|---|---|---|
| ANY data claim (numbers, rankings, CTR, share) | `reality-checker` | always |
| ANY campaign recommendation / assumption | `skeptic` | always |
| Copy deliverable (landing, ad, email) | per-task critique pass in the deliver loop; log that it happened | always |

### Dev

| Condition | Validator | Owner |
|---|---|---|
| Auth / data access / external API / secrets / dependencies touched | `security-auditor` | mandatory |
| Any code-producing wave | `code-reviewer` | mandatory |
| Any code-producing wave's diff | `anti-pattern-detector` | mandatory |
| Tech-spec or tasks produced | `reality-checker` + `skeptic` | mandatory |
| User-spec + tech-spec + tasks produced | `completeness-validator` | mandatory |
| Tech-spec claims existing patterns/code | `skeptic` (additional sweep) | mandatory |
| Tasks reference files/functions | `reality-checker` on hallucinations | mandatory |
| Migration file present | `migration-validator` | mandatory |
| Deploy boundary crossed | `pre-deploy-qa` / `post-deploy-qa` | mandatory |
| Deliverable is an HTTP surface (route/controller/API endpoint), including `ux_heavy: false` | HTTP-contract exercised endpoint test that drives the assembled request path (router/guards/validation/controller/serializer) via supertest/TestClient/equivalent; browser E2E not required | mandatory |
| `ux_heavy: true` | `ux-review` on screens + traces + handoff §6 | mandatory |
| Tests written | `test-reviewer` | recommended |
| Infrastructure change | `infrastructure-reviewer` | mandatory if infra work |
| CI/CD change | `deploy-reviewer` | mandatory if CI/CD touched |
| ≥2 specialists touch shared contract (API/type/schema) | Cross-validation note in handoff §4 | mandatory |

### Design

| Condition | Validator | Owner |
|---|---|---|
| Any UI deliverable | per-task critique pass in the deliver loop (ui-ux-methodology lens); log that it happened | always |
| Full UI flow (landing, multi-screen) | per-task design-review pass in the deliver loop (ui-ux-methodology lens); log that it happened | always |
| Any interactive surface (form, nav, CTA, modal) | `accessibility-validator` | mandatory |
| Brand claims / competitor claims | `reality-checker` | mandatory |
| Multi-screen consistency | Token propagation audit by `design-lead` manually | mandatory |
| `ux_heavy: true` (almost always for design) | `ux-review` on screens + traces + handoff §6 | mandatory |
| ≥2 design tasks share tokens | Cross-validation note in handoff §4 | mandatory |

### Cross-domain (≥2 domains contributing artefacts)

| Condition | Validator | Owner |
|---|---|---|
| L-tier launch involving dev + design + marketing | `product-context-validator` | mandatory |
| M-tier engagement with ≥2 domains (e.g. design UI + marketing copy on same landing) | `product-context-validator` | recommended |
| Cross-domain handoff from primary → secondary engagement | `product-context-validator` on secondary handoff | mandatory at L |

`product-context-validator` checks coherence ACROSS domains — API↔UI shape, copy↔shipped behavior, brand voice consistency, persona alignment, metaphor collisions. Not a substitute for single-domain validators; it catches misalignment they miss.

### Embedding-researcher prerequisites (NEW — Layer 5 of agency)

Before design/brand/marketing specialists begin work on an existing-project engagement, the orchestrating lead SHOULD dispatch the matching researcher to produce `{engagement}/design-research.md` / `brand-research.md`. These are READ by specialists during their work, NOT validators — they answer "what already exists in this project" before new work decides to ignore or align with it.

| Engagement | Researcher | When |
|---|---|---|
| Design engagement on existing project with prior design system (M+ tier) | `design-system-researcher` | Before any UI/UX specialist dispatch |
| Brand / marketing engagement with existing brand history (M+ tier) | `brand-context-researcher` | Before strategist / copywriter / banner dispatch |
| Dev feature on existing codebase | `code-researcher` (existing) | Before tech-spec authoring |

Researchers are SKIPPED for: S-tier single-asset work; new-project bootstrap (nothing to research); engagements explicitly scoped as rebrand / redesign-from-scratch.

## Execution patterns

### Primary path — the `engagement-workflow` validate phase

In the agency cascade, validation runs INSIDE the `engagement-workflow` Workflow. Its **validate phase** runs every validator in `plan.validators` (each as its agentType, in parallel), then **adversarially verifies** each non-info finding (an independent skeptic per finding; refuted findings dropped), and writes the per-validator `validation-outputs/{validator}-iter-{N}-{ts}.json` proof files with the **canonical envelope** — byte-identical to what `validator_lg.py` writes, so `handoff-precheck.py` and the manager consume them unchanged. This is the path for ALL tiers when the engagement runs through the Workflow; the lead only LISTS the applicable validators in the plan (per the domain matrix above).

### Standalone path — `validator_lg.py` (out-of-Workflow + HITL-in-validation)

`validator_lg.py` is retained as the standalone tool for two cases: (a) a validation run OUTSIDE the Workflow (a lead/manager doing a focused mid-engagement check), and (b) **HITL-in-validation** — `--interrupt-on-critical` pauses on any critical finding for a human directive (the Workflow validate phase has no native mid-run interrupt). Same output contract + retry-on-parse-failure, crash-resume (`--resume`), and observability hooks.

| Tier | In-cascade (default) | Standalone (out-of-Workflow / HITL wanted) |
|---|---|---|
| S | Workflow validate phase (or a lead's 1–2 manual Task dispatches) | `validator_lg.py --validators <subset>` |
| M | **Workflow validate phase** | `validator_lg.py --auto` (+ `--interrupt-on-critical` for HITL) |
| L | **Workflow validate phase** | `validator_lg.py --auto --interrupt-on-critical` for HITL-in-validation |

The output contract is IDENTICAL across paths — the same `validation-outputs/{validator}-iter-{N}-{ts}.json` files with the canonical block — so downstream (`handoff-precheck.py`, manager acceptance) consume them the same way regardless of which path produced them.

1. **Standalone manual parallel dispatch.** A lead run outside the Workflow invokes each validator via the Task tool in a single message — Claude Code runs them in parallel. Use on S-tier or for legitimately custom partial coverage (document the reason in `validation-log.md`).

2. **`validator_lg.py` LangGraph fan-out (standalone tool) — out-of-Workflow + HITL.** Single command runs all applicable validators in parallel via `Send` fan-out, writes per-validator JSON to `validation-outputs/`, appends to `validation-log.md`. Use when:
   - Engagement needs the FULL applicable validator set per `validation-pipeline` rules (use `--auto` to plan from criteria.md domain + predicates).
   - Lead wants crash-resume (`--resume` skips already-completed validators, re-runs missing + failed).
   - Lead wants retry-on-parse-failure (built into the graph as a retry edge).
   - Validator coverage benefits from observability (LangSmith hook auto-instrumented).
   
   ```bash
   # Auto-plan from criteria.md + run all applicable validators in parallel:
   python ~/.claude/scripts/validator_lg.py engagement/ --auto

   # Explicit subset:
   python ~/.claude/scripts/validator_lg.py engagement/ \
       --validators code-reviewer,security-auditor,reality-checker

   # Crash-resume after partial failure:
   python ~/.claude/scripts/validator_lg.py engagement/ --auto --resume

   # M/L tier — pause for human directive on any critical finding:
   python ~/.claude/scripts/validator_lg.py engagement/ --auto --interrupt-on-critical
   # Graph pauses at critical_check; prints thread_id + resume hint to stderr.
   # Manager / human inspects validation-outputs/ and resumes:
   python ~/.claude/scripts/validator_lg.py engagement/ \
       --resume-interrupt <thread_id> --decision PROCEED|REJECT|DIRECTED [--note ...]
   # Resume writes engagement/human-directive.md via human-directive.py
   # (same format adversary_lg.py uses), so manager phase can pick up cleanly.
   ```

   `--interrupt-on-critical` is silently ignored on tier=S (no manager phase
   exists for S, so pausing has no consumer). It also requires the SQLite
   checkpointer; `--no-checkpoint` together with the flag is auto-promoted
   with a stderr warning. Defaults are unchanged: without the flag, the graph
   runs to completion exactly as before.

   Output contract is IDENTICAL to manual dispatch — `handoff-precheck.py` and director acceptance consume the same `validation-outputs/{validator}-iter-{N}-{ts}.json` files unchanged.

   **Canonical envelope:** every `validation-outputs/*.json` file written by `validator_lg.py` now carries a `canonical` block alongside the raw validator output. The canonical schema is the cross-validator stable shape that downstream consumers (manager, Langfuse, analytics) read:

   ```json
   {
     "...raw fields from the validator (status, findings, summary, ...)": "...",
     "canonical": {
       "schema_version": "1.0",
       "validator": "code-reviewer",
       "validator_type": "judgement",
       "verdict": "approved | approved_with_caveats | changes_required | suspicious | skipped | error",
       "summary": "...",
       "methodology": null,
       "findings": [
         {
           "id": "code-reviewer#0",
           "severity": "critical | high | medium | low | info",
           "category": "...",
           "issue": "normalized message",
           "fix": "...",
           "evidence": null,
           "location": null
         }
       ],
       "metrics": null
     }
   }
   ```

   Raw fields remain unchanged — manual-dispatch validators (older agents that predate canonicalization) still write the raw shape. Consumers MAY read `canonical` when present and fall back to raw fields otherwise. Severity normalization absorbs the 15+ free-form severity strings into 5; verdict normalization collapses LENIENT_STATUSES into 6 canonical values. Numerical validators (accessibility / performance / security-auditor / ux-review / anti-pattern-detector) carry `validator_type: "numerical"` + a `metrics` block extracted from top-level numeric extras.

   The canonical envelope is also surfaced in the event ledger (`validator_completed` events) as `canonical_verdict` field + `output_schema_version: "1.0"` so cross-engagement aggregation has a stable key.

### Sequential (when validator input depends on prior validator)

Only when output of validator A is explicit input to validator B:
- `completeness-validator` AFTER `userspec-quality-validator` (needs validated user-spec)
- `post-deploy-qa` AFTER `pre-deploy-qa` (needs deferred criteria list)

### Re-dispatch loop

```
validator → finding → re-dispatch specialist → fix → re-run validator on fix only → record in log
```

Cap at 2 re-dispatches per finding per iteration. If 3rd re-dispatch needed, escalate: "finding cannot be resolved; defer with scope note OR mark as blocker and surface to director".

## When to skip (explicit permission)

You may skip a mandatory validator ONLY if:
- Criteria.md "Explicitly out of scope" covers it.
- User scope-sync waived it in writing.

Never skip silently. Log the waiver in validation-log.md:

```markdown
### security-auditing (SKIPPED)
- Reason: criteria.md §out-of-scope — "auth is unchanged, skip security audit"
- Waived by: user scope-sync 2026-04-20
```

Director will verify the waiver exists.

## Anti-patterns

- **Don't run validators after handoff.** Adversary at M/L acceptance provides cross-perspective second opinion, but does NOT substitute for lead's pre-handoff validator run — your first pass is your credibility.
- **Don't hide findings.** Every finding → resolution or deferral with justification.
- **Don't re-run the entire validator on partial fix.** Re-run on the fixed artefact only.
- **Don't skip `reality-checker` / `skeptic` because "the specialist is trustworthy".** Trust applies to humans; these validators exist because LLM specialists hallucinate.
- **Don't parallelize sequential dependencies.** Order matters when validator B needs A's output.
- **Don't skip `ux-review` on `ux_heavy: true` engagements** — that's the validator that keeps Wave-2-style visual regressions from making it past handoff.
- **Don't skip `anti-pattern-detector` on code waves** — catches skipped tests / dead code / hidden-tab fakes that other validators miss.
- **Don't accept Cross-validation as "all match" without the actual list.** The list is the proof; "trust me, they match" = phantom claim.