Validation And Qa

---
name: validation-and-qa
description: Use when code changes need a gate report, operator asks "is this ready?", or a single observed failure needs reproducibility check; runs lint/typecheck/tests, smoke, probes, reproducibility-before-fix. Do NOT approve deploy (that's release-deploy-reporting).
source_inspiration: conceptual-only
license_status: clean-room-original
fusebase_flow_version: 3.1
risk_level: medium
invocation: automatic
expected_outputs:
  - gate report (chat or docs/verification/<slug>-gate.md)
  - smoke report (docs/tmp/handoff/<date>-<slug>-smoke/)
  - reproduction notes (when applicable)
related_workflows:
  - verification-gate.md
  - smoke-verification.md
hook_dependencies:
  - stop
---

# Validation and QA

## Purpose

Validate changed behavior with deterministic checks before a deploy is approved. Five sub-modes: (a) verification gate after implementation; (b) smoke prompt verification post-deploy; (c) reproducibility-before-fix when a single observed failure is reported; (d) test-data hygiene cleanup; (e) **Lightweight-lane live-proof** (FR-21) — the compressed gate for a Lightweight ticket.

## When to invoke

- Active phase is `Verify` (per FLOW_RULES state announcement)
- AI Developer has reported a gate from `greenlight-implement` and the operator pasted it back
- Operator asks "is this ready?" / "did it pass?" / "validate this change"
- Operator describes a single observed failure: "the system did X, that's wrong" → run reproducibility-before-fix sub-mode (FR-10)
- Post-deploy: smoke-prompt verification when `verification-gate.md` specified numbered smoke prompts

## Do not invoke when

- Operator wants deploy approval — invoke `release-deploy-reporting` instead
- No code change exists yet (the gate runs against actual implementation, not a plan)
- Spec is still in DRAFT with unresolved clarifies — gate cannot pass an incomplete spec

## Required inputs

| Input | Where it lives | If missing |
|---|---|---|
| Verification-gate contract | `docs/specs/<slug>/verification-gate.md` | Stop; gate without contract is meaningless. Scoped local acceptance also requires a complete changed-risk/AC-to-test ledger. |
| AI Developer gate report | chat paste from AI Developer session | Stop; ask operator to paste report |
| Lint / typecheck / test commands | project-specific section of `AGENTS.md` or `package.json`/`pyproject.toml` | Stop; ask operator to provide commands |
| Worker-undisturbed list | `policies/protected-paths.yml` | Stop; gate cannot complete without protected-path verification |
| UI / E2E test context, if applicable | `verification-gate.md`, `templates/smoke-test-playwright.md`, route/component files, API/backend paths | Require route, primary action, stable locator, auth plan, test data, expected outcome, diagnostic surfaces |
| CLI edition map, for Fusebase Apps work | `docs/fusebase-cli-edition.md` | Continue with Flow-only validation, but mark app-domain probe coverage unknown |

## Procedure

### Sub-mode A — Verification gate

1. Read `verification-gate.md` to learn the contract (smoke prompts, probe list); required gate-report fields are canonical in `policies/gate-contracts.yml: gate_report` (producer template: `templates/gate-report.md`).
2. Verify completeness of the pasted gate report against that field schema.
3. If any field missing, redirect implementer: "Gate report missing <field>. Per FR-05, complete reports only. Re-run."
4. Run cross-artifact consistency check (every AC exercised in tasks; every locked decision cited where applied; no TODO/FIXME/WIP markers; spec status still DRAFT).
5. **Apply the 3-question test for empirical coverage** to each AC:
   1. *Did this AC actually run on a real input?* (or only in unit tests with mocks?)
   2. *Was the observed output compared to the expected output?* (or did the test only assert "no exception"?)
   3. *Could a reader reproduce the run from the gate report alone?* (commands + inputs + outputs visible?)

   If any answer is "no" for any AC, the gate is incomplete. Redirect implementer to add the missing evidence, even if all the unit tests pass. Mock-only tests + green CI do NOT prove the AC works against real inputs.

   **Self-recording systems (FR-23):** if the system under test has durable evidence surfaces (journals, run records, logs, snapshots), report fields carry POINTERS to them — transcribe only what no system records; redirect a report that re-transcribes system-recorded events.
6. For UI / E2E evidence, verify the test plan is specific enough to reproduce:
   - Route/navigation path to the feature.
   - Stable selectors or accessible locators for controls and meaningful outputs.
   - Primary action under test and expected user-visible result.
   - Auth/session plan, including whether synthetic test data or live-user verification is required.
   - Test data setup using unique values; no exact-count assumptions against shared state unless the test created the records.
   - Browser-visible evidence plus backend/log/API diagnostic evidence when the feature spans frontend and backend.
7. If passes, output approval to operator with explicit phrase "Gate verified. Phase advances to Deploy."

**Release evidence is CI, never a local run.** No `hooks/tests/run-tests.sh` invocation on a developer host is release evidence — the CI `verify` job on the tagged SHA is (`PUBLISHING.md` § Release evidence authority). A local run is unpinned-host feedback; it records neither SHA nor platform. Do not accept a local result as a release claim.

**Local run tiers.** `bash hooks/tests/run-tests.sh` = the FAST LOCAL DEFAULT (cheap phases only, ≤10 min budget). `FF_FULL=1 bash hooks/tests/run-tests.sh` = the full unscoped set (CI takes this path automatically). `FF_ONLY="tag1,tag2"` = a scoped subset for implement-loop speed. `FF_LIST=1` prints the tag list for that invocation without running.

**Exact-state validator evidence.** After exact staging, run `bash hooks/local/run-validators.sh` when the repository supplies it. Reusable validator evidence is currently unavailable because `FUSEBASE_FLOW_VALIDATOR_CONTEXT`, including `schema: 1`, `complete: true`, and nonempty declaration arrays, is supplied by the caller and does not independently prove complete validator-visible inputs, environment, dependencies, or toolchains. The runner still executes lint and typecheck, but it does not create a reusable receipt; pre-commit reruns them. Existing authentication, command/config binding, live-state checks, and Windows refusal remain fail-closed foundations for a future independently established protocol. Secret, protected-path, module-size, and release controls remain live.

**Complete-suite vs scoped local acceptance.** Only the full unscoped run writes `state/audit/hook-test-results.md` and prints the strict `[run-tests] N/N PASS`. Fast and scoped summaries retain their suffixes, remain rejected by `ffhc_run_tests_pass_ok` / `ffhc_count_pass_lines`, and write only `hook-test-results-fast.md` / `hook-test-results-scoped.md`. Never cite a subset as full-suite or release evidence.

**Risk-scoped local acceptance (B6).** A Full-lane gate may accept focused local evidence only through the per-ticket verification gate's changed-risk/AC-to-test ledger. Each row records changed behavior/risk and AC, source/config/input dependencies, toolchain and platform, exact command/selection, expected positive and negative outcomes, actual exit/result, durable evidence, status (`NEW`, `REUSED`, `DEFERRED`, `UNVERIFIED`), and invalidation rationale. Changed dependencies invalidate matching rows; unknown dependency completeness requires the affected group to rerun. HEAD or filename equality alone never validates prior evidence. Missing evidence stays open. Broad mutation, compatibility, and platform rows may be `DEFERRED` to maintainer/release CI; never mark them PASS without execution. Normal lint/typecheck, staged secret scan, protected-path, module-size, and pre-commit controls remain live.

**Execution visibility and budget.** Before a selected local run, declare its expected total budget from the ledger. Every selected phase must expose START/END, canonical tag, elapsed time, child rc, and configured timeout budget. Missing, zero-row, crashed, or timed-out phases are failures. After an interrupted phase, record a full owned-descendant process scan and clear every survivor; parent-process absence alone is insufficient evidence. A heartbeat proves life, not forward progress; unexplained overruns trigger diagnosis, never automatic timeout inflation or prefix replay.

**CI scope truth.** The existing release verification runs the full unscoped suite on Linux and Windows and release publication depends on that verify job. Configuration inspection proves wiring only: it does not prove a run occurred or that the suite completes within the configured 60-minute wall. Do not claim ordinary push/PR/nightly automation where no trigger exists.

### Sub-mode E — Lightweight-lane live-proof (FR-21)

For a Lightweight-lane ticket there is no `verification-gate.md` and no long-form gate report — but the **live proof is never skipped** (it is the safety floor). Compress, don't drop:

1. Run the change on a **real input** (not a mock) and **compare observed to expected** — the first two of the 3-question test, applied to the one acceptance criterion in the change-note.
2. Make it **reproducible from the change-note alone** (command + input + observed output in 1–3 lines) — the third question.
3. Confirm the FR-07 protected-path re-check is clean (`git diff` against `policies/protected-paths.yml`).
4. Report in **1–3 lines**: `<what changed> · observed <X>, expected <Y> ✓ · FR-07 clean`. That is the LL gate.

If the live proof cannot be produced, continue bounded diagnosis and assessment. Declare an objective trigger and promote only when evidence supports it; otherwise stop at `BLOCKED-AT-lane-assessment`. Never turn missing proof into an inferred safe result.

### Sub-mode B — Smoke prompt verification (post-deploy)

1. Invoke `flow-skills/smoke-testing/SKILL.md` — the canonical smoke contract (outcome-first sufficiency, UI/browser plan, shared-state discipline, dual-surface evidence) — and run S1..Sn from `verification-gate.md` per `workflows/smoke-verification.md` mechanics.
2. Persist evidence to `docs/tmp/handoff/<date>-<slug>-smoke/`; compute the pass ratio and verify it against the threshold in the gate contract.
3. If below threshold, or if end-to-end smoke is not feasible, do NOT mark spec DONE. Report failure or `PENDING-OPERATOR-SMOKE` with concrete `Sn observed Y, expected Z` / missing prerequisite.

### Sub-mode C — Reproducibility before fix

1. Operator reports single observed failure ("the system did X").
2. Classify the system under test FIRST — the verdict rules differ:
   - **DETERMINISTIC app code:** ordinary application/backend/UI logic — same inputs are supposed
     to give same outputs. Async timing, job ordering, and shared state make the *reproduction
     rate* vary, not the bug's reality.
   - **LLM / agent behavior:** the failure is in model output or agent-path choice, where
     run-to-run variance is inherent.
3. Reproduce 3 times under the same conditions. Verdicts:

   | System type | 3/3 | 1/3 or 2/3 | 0/3 |
   |---|---|---|---|
   | Deterministic app code | Systemic → `requirements-specification` fix spec | **REAL bug** — intermittent ≠ nonexistent. Route to flaky/race investigation: capture `dev-debug-logs` (local) or `remote-logs` (deployed) evidence from a failing run; hunt ordering/timing/state-leak causes. NEVER no-op close. Sole escape — Explained-external (mirrors `smoke-testing` § Flaky-result "Explained"): a NAMED external cause (third-party outage, deploy-propagation delay, cold start) with the ground-truth diagnostic ATTACHED to the ticket evidence; then 3/3 clean under the same conditions after the cause clears before any PASS/close. Bare "couldn't reproduce twice" with no attached diagnostic = REAL bug. | Not reproduced — record exact attempted conditions; vary ONE plausible condition (data state, timing, concurrency) for one more round before recommending close |
   | LLM / agent behavior | Systemic → fix spec (prompt/tooling/guard) | Model variance — document and recommend no-op close (the classic FR-10 close) | Close as no-op-needed with attempts documented |

   A partial reproduction of deterministic code is the most-laundered bug class: "couldn't
   reproduce twice" reads as noise but is a race/state bug waiting for a client.
4. Document attempts in chat with concrete commands, inputs, observed outputs.

### Sub-mode D — Test-data hygiene cleanup

Run BEFORE marking the spec DONE. Smoke runs and reproducibility attempts often write throwaway artifacts (probe payloads, screenshots, temp database rows, mock response captures). These must be cleaned up so they don't pollute the next session or leak into commits.

1. **Inventory test artifacts created during this ticket:**
   - Files written under `docs/tmp/handoff/<date>-<slug>-smoke/` (KEEP — evidence)
   - Files written under `tmp/`, `/tmp/`, project root with random names (DELETE)
   - Database rows tagged with test slugs / fixture IDs (DELETE if scoped to this ticket; keep if shared fixture)
   - Mock response captures left in feature directories (DELETE)
   - Screenshots / response bodies that captured operator session content during live-user verification (REDACT or DELETE per `workflows/live-user-verification.md` Step 6)
   - External-service test objects, notifications, webhooks, or payment-like side effects (CLEAN UP or document as intentionally retained with approval)
2. **Run cleanup commands** with explicit before/after counts. Example:
   ```
   # Before: list what will be removed
   find tmp/ -type f -name '<slug>-*' | wc -l    # → 7
   # Cleanup
   find tmp/ -type f -name '<slug>-*' -delete
   # After: confirm zero remain
   find tmp/ -type f -name '<slug>-*' | wc -l    # → 0
   ```
3. **Verify no test data slipped into git:** `git status` after cleanup; if anything is staged that looks like test artifact, unstage and delete (or move to `docs/tmp/handoff/<date>-<slug>-smoke/` if it is genuine evidence).
4. **Document in the gate report:** one-line "Test-data hygiene: <N> ephemeral artifacts cleaned; 0 remaining; git status clean."

Skip this sub-mode only if no test data was written during the ticket (rare — even pure typecheck tickets often leave `node_modules/` / `__pycache__/` churn that should be ignored, not deleted).

## Worked example

An ordinary Lightweight correction has a one-command probe. Run it on the diagnosed input, compare observed and expected output, and record reproduction steps in the change-note. If the probe cannot reach a real input, continue bounded diagnosis; declare an objective Full trigger only with evidence, otherwise stop at `BLOCKED-AT-lane-assessment`. A Full-lane ticket instead records the same evidence against its P-check in the gate report.

## Output artifacts

| Artifact | Path / location | Mode |
|---|---|---|
| Gate verdict | chat output to operator | Mode A |
| Smoke evidence | `docs/tmp/handoff/<date>-<slug>-smoke/` | Mode B (full) |
| Reproduction notes | chat output + optionally `docs/specs/<slug>/spec.md` audit-log | Mode A + Mode B |

## Verification cost

For any verification that spans wall-clock time (deploys, jobs, agent runs): record-then-read by default — see `flow-skills/smoke-testing` § Verification cost discipline (polling is the bounded exception, not the default).

## Failure cases

| Failure mode | Detection | Response |
|---|---|---|
| Lint/typecheck failed in implementer gate report | report shows "lint: errors" or "typecheck: errors" | Redirect implementer to fix; do not advance phase |
| Worker-undisturbed file diffed unexpectedly | `git diff` against `protected-paths.yml` shows non-empty | Stop; redirect to check FR-07 + filed exception artifact |
| UI / E2E plan is vague | no route, locator, primary action, test data, auth plan, or expected outcome | Mark gate incomplete; require a reproducible browser test plan |
| Shared-state assumption | asserts exact counts or empty state without creating/isolating data | Revise test to create unique data or isolate state before claiming PASS |
| Scoped evidence lacks a complete dependency or AC row | changed/unknown dependency, expected outcome, actual result, or evidence pointer is absent | Keep the row open; rerun the affected group or record it as `DEFERRED` / `UNVERIFIED` |
| Selected phase is missing, silent, or overruns its declared budget | no START/END pair, zero result rows, crash/timeout rc, or unexplained overrun | Fail the selected run; diagnose the named phase before retrying |
| Interrupted phase leaves owned descendants | full descendant scan finds a surviving child or only the parent PID was checked | Terminate the verified owned tree and record the post-cleanup scan before another run |
| Smoke S<n> failed | screenshot or log shows divergence from expected | Document `Sn observed Y, expected Z`; do NOT mark spec DONE; surface to operator with rollback/fix-forward options |
| Browser PASS but backend diagnostic shows error | console/UI looks right while server log/request dump/job row shows failure | Smoke FAIL; attach both evidence surfaces |
| Partial reproduction (1/3 or 2/3) — deterministic app code | attempts reproduce intermittently and the system is ordinary app logic | REAL bug: route to flaky/race investigation with `dev-debug-logs` / `remote-logs` evidence from a failing run; never no-op close — sole escape is Sub-mode C Explained-external (named external cause + diagnostic ATTACHED, 3/3 clean re-run after it clears) |
| Partial reproduction (1/3 or 2/3) — LLM/agent behavior | failure is model output / agent-path variance | Document model variance; recommend no-op close per FR-10 |

## Escalation path

- Smoke threshold not met → operator decides rollback or fix-forward via follow-up task; surface options, don't decide. `git revert` is a valid rollback ONLY for a `code-only` deploy; a migration / secret / sidecar / cross-app-contract deploy needs the surface-appropriate plan (`flow-skills/release-deploy-reporting/SKILL.md` § Rollback-surface classification)
- Test infrastructure broken → file infra ticket; gate cannot proceed; do not weaken gate to "skip tests"
- Reproduction needs operator session credentials → propose via `workflows/architect-escalation.md` (live-user verification with explicit consent)
- External-service side effects needed for verification → require explicit approval or use a sandbox/test-mode path before execution

## Anti-patterns

- Do NOT approve deploy from this skill (that's `release-deploy-reporting`)
- Do NOT weaken gate criteria mid-flight ("just this once skip lint")
- Do NOT print or persist session keys / cookies if a live-user verification is in play; mask in output
- Do NOT auto-fix lint or typecheck errors without operator approval — surface them
- Do NOT skip reproducibility-before-fix when a single failure is reported (FR-10)
- Do NOT close a partially-reproduced failure in deterministic app code as "model variance" — that close is reserved for LLM/agent-behavior failures (Sub-mode C verdict table); an intermittent deterministic failure is a race/state bug unless Explained-external with the named cause's diagnostic ATTACHED (Sub-mode C escape — evidence-gated, never a bare assertion)
- Do NOT use browser automation for API-only, unit-only, load, or performance testing when a lighter deterministic probe is sufficient
- Do NOT claim UI/E2E coverage from a test plan with placeholders, uncreated data, brittle selectors, or unspecified auth state
- Do NOT allow external-service smoke to send real notifications, charges, or customer-visible messages without an approval path

## Clean-room note

Original Fusebase Flow content. Designed after reviewing public AI coding workflow patterns; no third-party code, prompts, skill files, or hook scripts are copied. See `docs/source-map.md`.