VERIFY-ONLY QA pipeline for a ticket a dev claims done (the spec is the oracle). Reads the ticket + spec + schema to derive expected behavior → designs 2–5 test cases (exact repro, boundary, whole-screen sanity, read-only DB verify for writes) → self-provisions missing data through the REAL UI flow (write-gated) → runs them HEADED in the browser → collects evidence (named screenshots, annotated images with in-image captions) → cross-checks every ticket claim against an evidence file → machine...
Installs into .claude/skills of the current project.
Are you the author of Qa?
Add the live security badge to your README. It updates with every re-scan.
[](https://www.skillsdirectory.com/skills/connorpham-vteam-harness-5240d65c)
---
name: qa
description: "VERIFY-ONLY QA pipeline for a ticket a dev claims done (the spec is the oracle). Reads the ticket + spec + schema to derive expected behavior → designs 2–5 test cases (exact repro, boundary, whole-screen sanity, read-only DB verify for writes) → self-provisions missing data through the REAL UI flow (write-gated) → runs them HEADED in the browser → collects evidence (named screenshots, annotated images with in-image captions) → cross-checks every ticket claim against an evidence file → machine gate (evd_check.py) → challenger sign-off → plain-language REPORT.md anyone can read → tracker comment posted once both machine gates are green, ticket transition per the full-auto policy. NO code change, NO fix — verification only."
argument-hint: "<TICKET: VT-nnn | tracker URL> [assignee=<name>] [base=<app URL — overrides config app.url for this run>]"
---
> **Model routing for this tool** (from `model-routing.data.yaml`, snapshot 2026-08-17):
> `frontier` → **fable** · `workhorse` → **opus** · `standard` → **sonnet** · `utility` → **haiku**
> Roles → tiers: ba-challenger: standard · ba-draft: standard · dev-r1: workhorse · dev-r2: standard (high-stakes: workhorse) · dev-r3: workhorse · explore: utility · qa-challenger: standard · sa-background: workhorse · sa-challenger: workhorse · sa-writer: workhorse
> Resolve at runtime: `python3 .vteam/scripts/model_route.py <role> --tool claude-code [--high-stakes]` — high-stakes diffs (review.high_stakes_*) bump dev-r2 to the workhorse tier.
> Spawning a subagent: pass the resolved name as the Agent tool's `model` parameter.
> **Environment:** `app:` is not configured in vteam.config.yaml — env bring-up is
> manual this session, and `app_check.sh` prints `APP: SKIP`. Set `app.start` +
> `app.url` (+ optional `app.health`, `app.open_files`, `app.headed` — schema:
> DESIGN.md §2) to enable the watchable session: headed Chrome via
> `.vteam/scripts/browser.mjs`, health probe via `.vteam/scripts/app_check.sh`,
> editor opening via `.vteam/scripts/open_files.sh`.
# /qa — verify a delivered ticket against the spec (QA lane, no code)
**What this workflow is NOT:** it never edits product code, never fixes anything,
never expands the dev's scope. It answers ONE question with evidence: **does the
app now behave the way the spec says it should, for the requirement this ticket
describes — without breaking the rest of the screen?**
**Immutable principles** (violation = the verification does not count):
1. **The spec is the oracle.** "Done" means the app matches the spec (+ the schema
for data rules), NOT "matches the ticket prose" and NOT "no error appears".
Every expected value cites its spec section / schema line. Ticket contradicts
spec → spec wins; report the contradiction. Spec silent/self-contradictory →
that TC is BLOCKED and **goes to the BA lane for arbitration** (the BA opens a
3-condition gap question; per `raci.md` the BA is R for "bug or intended", the
owner only A when the BA can't answer from the spec either); never invent an
expected value.
2. **Real test, no shortcuts, no lying.** Every verdict comes from a browser run
actually performed THIS session against the running app — never from reading
code, never from an HTTP status, never from a previous run. A step that could
not run is BLOCKED with a reason + unblock path — never inferred.
3. **Reading code is allowed only to know WHERE to look** (which route, which
role gate), never to derive the expected value.
4. **Data through the REAL flow only.** Missing data → create it via the app UI
(login → screen → action). Direct SQL/ORM writes are forbidden as a test path;
the DB is read-only, used to VERIFY writes/rollbacks. Every write on a shared
env follows the write gate: **user present in the session → ask before each
write**; scheduled session with the user absent (at `autonomy.level: full`) →
proceed without waiting, with the full record: the write block (screen +
action + values + cleanup path) into the session minutes BEFORE writing, rows
marked `ZZTEST`, cleaned through the app's reverse flow after V4. No reverse
flow to clean with → that TC is BLOCKED, nothing is written.
5. **Evidence a stranger understands.** Per TC: `manifest.md` in plain language +
per-step named screenshots + a box drawn on the exact region with an in-image
caption (`annotate.py box --label`). `evd_check.py` must be green before
reporting.
6. **Cross-check before reporting.** Re-read the ticket line by line; map every
claim → the evidence file that proves it. An unmapped claim = verification
incomplete.
7. **A fresh challenger signs off.** Before the verdict is final, spawn one fresh
agent (empty context) briefed with the ticket + verify-sheet + evidence paths,
whose job is to FALSIFY (wrong account/role? data-shaped diff? boundary
missed? stale evidence? jargon in REPORT.md?). Record both cards in
`evd/<TICKET>/debate.md`.
8. **Visible, human-readable process.** One plain-language `▶` line at every
phase transition. REPORT.md and the final summary pass the bar: **a
non-programmer reads 2 minutes and understands everything.**
Tempted to infer a verdict instead of running the step? Your excuse is catalogued in `red-flags.md`, next to the gate that catches it.
Tracker recipes: the configured tracker provider. Evidence tooling:
`.vteam/scripts/annotate.py` (box + side-by-side diff) and
`.vteam/scripts/evd_check.py` (machine gate) — use them, don't hand-roll.
External payment/integration environments are SANDBOX — never a real transaction.
---
## V0 — RESOLVE + ANNOUNCE THE TICKET
1. Fetch issue + ALL comments + attachments. Env vars missing → STOP, ask.
2. **Assignee filter.** Default from config (`assignee=` overrides). Different
assignee → STOP and ask before verifying someone else's ticket.
3. **Status must be verifiable.** In Review / Ready for QA / Done-claimed = the
target. To Do / In Progress = not delivered yet → STOP: `BLOCKED (not
delivered)`. Closed → ask whether to re-verify.
4. **ANNOUNCE before any work** (one block); auto-picked ticket → wait for the go:
```
▶ Verifying <TICKET> — <summary>
assignee: <name> | status: <status> | screen/feature: <what>
target: <app URL> (branch: <protected|PR branch — say which>)
```
5. **Pin WHAT code is being verified.** Dev's PR merged → verify the protected
branch. Not merged → check out the PR branch (read-only) and say so. Unclear →
ask.
## V1 — READ: ticket + spec + schema (understand as QA, not as dev)
**Who is verifying:** read `docs/team/competencies/qa/qa-identity.md` first —
the senior tester this lane runs as (the oracle rule, severity by consequence,
the third outcome) — then `docs/team/roles/qa.md`. Immediately after, read
`docs/team/competencies/qa/qa-requirement-smells.md` and run its three-line
test on every acceptance criterion **before** deriving a single expected value:
a criterion that fails it is `BLOCKED (not testable as written)`, and that is
the verification's first finding, batched to the requirement owner now — not the
developer, and not after a day of testing.
**KB preflight (index-only):** `docs/qa/knowledge-base.md` §0 + INDEX BEFORE
designing anything; open ONLY tag-matching lessons and answer them in the
verify-sheet (cite by TITLE). Scan `docs/qa/known-issues.md` headings — any
divergence found later is deduped against it before being called a bug.
**Read the change ledger BEFORE the shard:** `docs/specs/changes.md` — filter
CH-nn rows touching this ticket's shard/screen. An expected value that ignores a
decided CH is a wrong expected value.
Write `evd/<TICKET>/verifysheet.md` (REPORT.md points to it in the
appendix; it commits with the V7.4 dossier so the link outlives the session):
- Requirement + AC as the ticket states them; the dev's claimed delivery (PR
link, comment) verbatim.
- **EXPECTED (from the spec, cited)** — quote the governing sections. Data rules
from the schema (cite model:field).
- **UI EXPECTED (from the design oracle)** — ① design link on the ticket:
download frames to `evd/<TICKET>/design/`, exact values from node
data; ② else look the frame up by screen code in the design index. A UI TC
compares the real screen against it (`annotate.py diff` → `design_vs_app.png`),
judged at block/color/text level. The dev submitted a committed `fidelity.md`
→ READ it: re-run `node .vteam/profiles/generic/scripts/ui_fidelity.mjs <TICKET>` to confirm the
numbers still hold on the build being verified, and audit the INTENTIONAL
deviation lines — does the declared reason stand (real a11y/spec, or an
excuse)? QA still designs its own boundaries — never take TCs from the dev's
spec. Design oracle contradicts the spec (verbatim messages/validation) → spec
wins; record the design deviation.
- **CLAIM (from ticket + dev comment)** — what they say now works.
- Accounts/roles needed per criterion, data needed.
- Ambiguity → ask the user BEFORE spending effort.
## V2 — DESIGN THE VERIFICATION (before touching a browser)
**Load the competencies this verification needs** — from
`docs/team/competencies/qa/INDEX.md` (read the index, not the tree): **every
row whose `Applies` reads `always`, each opened at the step its `Loads at` column
names** — read them off the index, do not work from a list memorised here: this
sentence used to enumerate them and had drifted both ways, naming two rows that are no
longer `always` and missing three that are. `qa-report-writing` loads at V5 for that
reason: it is needed when the report is written, not when the pack is designed. The
token rows load when the ticket matches — `qa-security-probes` on any auth/roles/money/PII/upload
ticket (`label:`/`term:` match). Record the loaded list in the verify-sheet
under **Competencies:**. Their reference tables (full hostile-input, heuristic,
security and checklist catalogues) live under `competencies/qa/reference/` —
open the one row-set the field in front of you needs; never paste a whole table
into the plan. `qa-test-design` governs how the 2–5 TCs below are chosen and
budgeted by consequence:
Plan in the verify-sheet — usually 2–5 TCs. **Every TC carries a `TITLE:` (a
sentence about behaviour — "A quantity of 0 is refused", never "test qty") and a
`KIND:` from the six the gate knows** — `acceptance` · `boundary` ·
`whole-screen` · `write-readback` · `exploratory` · `security`; the pack MUST
hold a `boundary` and a `whole-screen` case, and the folder is named
`TC_<n>_<what_it_proves>` so `ls` reads as a test plan:
- ① **Exact acceptance path** (`KIND: acceptance`) — the criterion as the ticket/spec states it.
- ② **Boundary pair** — the adjacent input that must behave the OTHER way
(empty/duplicate identifier, insufficient balance, cancelled record…). A
delivery that overshoots is a bug.
- ③ **Whole-screen sanity** — the rest of the screen/flow still behaves per
spec, not only the new spot.
- ④ **Write → read-only DB verify** — if the feature writes, verify the row via
a read-only query AFTER the UI action; and the rollback path if the spec
defines one.
- **Per TC, the JOURNEY — you are a person using the product, not a script
hitting routes.** `evd_check` demands these by name, because asking politely
did not get them written:
- **AS** — which account and which role is signed in. A verdict with no actor
is untraceable, and half of all UI bugs are role-shaped.
- **PRECONDITION** — what must already be true: the record exists, in which
state, created by whom. Resolve it now (read-only); never trust an id from
the ticket to still exist.
- **ENTRY** — where the user *starts* and what they *click* to arrive: sign in
→ which list → which row → which button. **Typing the screen's address
proves the address, not the product**: if the menu item is missing, the
permission is wrong, or the row is unreachable, a deep link hides all three.
Keep the deep link if you like — as a SECOND path, never the only one.
- **STEPS** — numbered, each one screenshotted, in the order a person does them.
- **EXPECTED** — spec-cited, and the region to box.
- **AFTER** — what changed once the action landed: the message, the list row,
the value **still there after a reload** (a save that dies on refresh is not
a save), the neighbouring screen that reads the same data.
- **BACK** — how the user leaves: Back, Cancel, browser back. What state do
they return to — is the filter preserved, is a half-typed draft lost, does
Cancel actually cancel? This is where "it works" usually stops working.
- **Data check (read-only):** does the needed data exist? Never trust values in
the ticket to still exist — resolve a currently-valid key; an empty list from
stale data = "data moved", not a bug.
- **The boundary value (②) is chosen with `qa-hostile-inputs`** — one value the
spec refuses + one a real user produces this week; a boundary chosen for
convenience is a happy-path case with a different number. **Every TC's STEPS
carries at least one real-user move** (`qa-user-mindset` — Enter not Save,
double-click, Back after, refresh, paste) and names its `PERSONA:`. When the
spec is silent, `qa-heuristics` supplies a consistency oracle so the finding
has a named owner, not a shrug. Write each record to the
`qa-case-writing` fifteen-second standard — TITLE a behaviour sentence,
EXPECTED an observable fact with a citation, none of the banned filler words.
## V2b — ENV BRING-UP
- DB up → migrations clean → dev server up: run the configured `app.start` in a
BACKGROUND terminal (the Environment block above carries the resolved values;
gates never start servers — they probe).
- **Health proof, machine-quotable:** `bash .vteam/scripts/app_check.sh --wait 60`
(a `base=` argument on this run → pass it as `--url <base>`). PASTE the
`APP: UP …` line into the verify-sheet — that line IS the bring-up proof.
`APP: DOWN` → BLOCKED with the printed unblock path. `APP: SKIP` → the repo
has no `app:` config: ask the owner for the URL (and propose the `app:`
section — schema DESIGN.md §2) before any TC runs.
- Resolve test accounts per role (seed data or existing rows, read-only). No
usable account → V3 (create via the app's real sign-up flow, write-gated).
- Browser: HEADED — a run the user could watch, and the mechanism is named:
a REAL Chrome window via `.vteam/scripts/browser.mjs`. Preflight it NOW:
`node .vteam/scripts/browser.mjs check` — missing Playwright is BLOCKED with
the printed install command, never a silent headless run. (`app.headed:
never` — unattended shifts — drops only the visibility, never a screenshot.)
## V3 — PROVISION DATA VIA THE REAL UI FLOW (only if V2 found gaps)
- Trace the producing flow (sign-up → login → the screen that creates the
record). Drive it in the browser like a real user.
- **Write gate (principle 4): user present → ask first, one block per write**
(screen + action + exact values + cleanup action); **user absent (scheduled
session)** → write the block into the session minutes BEFORE acting, then
proceed. Both paths: capture the run under
`evd/<TICKET>/data_prep/`, mark rows `ZZTEST`, verify by read-only
SELECT, clean up via the app's reverse action after V4. No reverse flow → that
TC = BLOCKED, nothing written.
- Producing flow impossible (feature not built, external dependency) → that TC =
`BLOCKED (missing data)` with what was tried; never fake it.
## V3b — TICKETS WITH NO SCREEN (the non-UI verify branch)
Activates when the ticket **produces no screen**: a migration, a model, a shared
library function, a background process. Without this branch such tickets are
unverifiable by a browser pipeline — and they are the foundation everything else
stands on, where errors are most expensive.
Unchanged principle: **the spec is still the oracle, and QA still RUNS things
itself — never reads the dev's report as proof.** Only the viewport changes: from
screenshots to database state and command output.
Every TC on this branch needs all four, recorded in
`evd/<TICKET>/TC_n/`:
1. **Migrations run on a CLEAN database, not only the dev's.** Create a scratch
DB → deploy migrations → status clean. A migration that only works on the dev
machine (old data masking the flaw) is a broken migration.
2. **Spec-mandated data invariants, checked by read-only SELECT.** Cite the exact
rule id. Record the statements and real results in `db_verify.md`.
3. **The dev's tests must go RED when the behavior is flipped.** Pick one core
constraint of the ticket, temporarily break it, run the tests, confirm they
RED, restore, and `git diff` confirms clean. A green test proves nothing; a
test that reds in the right place does.
4. **A boundary pair QA invents itself, not taken from the dev's tests.** The dev
tested their understanding; QA probes exactly where that understanding could
be wrong.
`evd_check.py` accepts a TC without images when its `manifest.md` declares
`TYPE: NON-UI` — but then a verification file is MANDATORY: `db_verify.md`
(read-only data checks) for anything with a database, or `cmd_verify.md`
(real command + real output pairs — test runs, direct invocations) for pure
libraries and CLIs that have no database to check. No images and no
db/cmd verification is not verification. Never stuff command transcripts into
db_verify.md to satisfy the gate — the file names must stay honest.
## V4 — RUN THE VERIFICATION, HEADED (the actual test)
Announce each run in one line (TC / account / what it proves). Walk the journey
you wrote in V2 — from ENTRY, clicking what a user clicks. For each TC:
screenshot every meaningful step as `01_<what>.png`, `02_<what>.png`… — name the
files after what they SHOW (`01_orders_list.png`, not `01.png`), because the
filenames are the first thing the reader sees — then box the region that carries
the verdict. **Required on every executed UI TC, not just failures**: an
unannotated full-page shot makes the reader guess which pixels mattered, and the
caption is what a stranger reads instead of asking you.
**The headed mechanism — one journey script per TC, kept as evidence:** write
`evd/<TICKET>/TC_<n>/journey.mjs` importing `launch`/`shot` from
`.vteam/scripts/browser.mjs`, walking EXACTLY the V2 journey (ENTRY as clicks,
STEPS in order, AFTER including the reload, BACK), calling
`shot(page, dir, n, "<what_it_shows>")` at every meaningful step — the helper
enforces the `NN_<what>.png` naming by construction. `node TC_<n>/journey.mjs`
opens a real Chrome window the owner can watch. The script STAYS in the TC
folder: it is evidence, and the next QA session re-runs the same journey
against a new build instead of re-improvising it.
```bash
python3 .vteam/scripts/annotate.py box \
--img evd/<TICKET>/TC_<n>/03_result.png --rect X,Y,W,H [--rect ...] \
--label "TC_<n>: <what this proves / what diverges>" \
--out evd/<TICKET>/TC_<n>/03_result_boxed.png
```
The box **hugs exactly X,Y,W,H** (a 4-px frame, no padding) so it proves that
region and no other; measure the rect from the screenshot the box is drawn on.
The caption is burned in on a bar **below** the image, word-wrapped in
`project.language` — below, never above, so no pixel coordinate moves. Several
`--rect` flags box several regions under one caption.
Evidence layout (one folder per TC):
```
evd/<TICKET>/
├── manifest.md # plain language: the requirement + the COVERAGE: block
│ # (security: TC_n | n/a — reason · accessibility: …) +
│ # the evd_index block (what was tested / which file is
│ # from which case) — regenerated, never hand-edited
├── <TICKET>_testcases.xlsx # V7: the same record as a workbook (generated by
│ # xlsx_export.py; ignored by git — regenerate, never edit)
├── debate.md # V6 cards (verifier + challenger)
├── data_prep/ # V3 runs, if any
└── TC_<n>_<what_it_proves>/ # the folder name IS the first line of the test plan
├── journey.mjs # the headed run itself (browser.mjs launch/shot) —
│ # re-runnable evidence, committed with the dossier
├── manifest.md # the journey, in plain language — gate-checked:
│ # TITLE: Changing a quantity and pressing Save recalculates the total
│ # KIND: acceptance|boundary|whole-screen|write-readback|exploratory|security
│ # RESULT: PASS|FAIL|BLOCKED
│ # AS: staff@demo (role STAFF)
│ # PRECONDITION: order #4102 exists, state PENDING
│ # ENTRY: signed in → Orders list → row #4102 → Edit
│ # STEPS: 1 change qty 2→3 · 2 press Save
│ # EXPECTED (spec §3.2): total recalculates to 450,000
│ # ACTUAL: total shows 450,000, success message shown
│ # AFTER: 'Saved' toast · list shows 3 · survives reload
│ # BACK: Back → Orders list, filter preserved
├── 01_*.png … # per-step shots, named for what they show
│ # (+ *_boxed.png REQUIRED on the verdict step, with
│ # a caption — a red box with no words explains nothing)
└── db_verify.md # write TCs only: the read-only SELECT + rows after the action
```
Gate: `python3 .vteam/scripts/evd_check.py --evd evd/<TICKET>
--expect-tcs <N from V2>` green — pass the PLANNED TC count, so "planned 5, ran
1" can go red — before V5.
## V5 — CROSS-CHECK AGAINST THE TICKET
Table in the verify-sheet — one row per claim in the ticket AND the dev's
delivery comment: `claim → TC → evidence file → matches spec?`. Rules:
- A claim with no evidence row = INCOMPLETE — back to V2/V4; do not report.
- Criterion met but a NEW divergence appeared nearby → verdict `NEW-BUG` for that
finding; describe it with evidence. A finding to report, never to fix here.
- Ticket misdescribed the behavior but the app still diverges from spec →
verdict on the REAL divergence, note the correction.
## V5b — DRAFT REPORT.md (plain language, so V6 can grade it)
Beside `COMMIT:` and `VERIFIED-AT:`, the report header carries two more anchors
the gate demands: `ENVIRONMENT: <name — url>` (local · dev · stg · prod — a bug
found on staging is not evidence about production) and `ORACLE: <what "correct"
was compared against>` (the spec shard, or `NONE` when nothing written decided).
Write `evd/<TICKET>/REPORT.md` — in `project.language`, black-box
voice, ZERO jargon in the body (no file:line, route names, "oracle" — appendix
only). Bar: **a non-programmer reads 2 minutes and understands everything.**
Fixed template:
```markdown
# Verification report <TICKET> — <verdict, large: PASS / FAIL / PARTIAL / NEW-BUG / BLOCKED / UNCLEAR>
COMMIT: <sha verified — HEAD of the code the TCs ran on; the verdict binds to the code, evd_check requires it>
VERIFIED-AT: <ISO 8601 timestamp of the run — on squash/rebase repos the pinned SHA dies with the branch, and this is the fallback anchor stale_verdict_check compares post-verdict commits against>
## 1. What does this ticket ask for? (told as a user)
On screen <name>, when <the user does what>, the system must <the right outcome —
per the spec>. The dev reports it done (PR <link>).
## 2. How did I check?
| # | What I did (account, screen, action, data) | Expected (per spec) | Actual | Match? |
(plus: where test data came from / how it was UI-created, if applicable)
## 3. Evidence — what to look at, where?
One line each: file path → what that file SHOWS (covering every TC_<n>).
## 4. Conclusion
1–3 sentences: requirement met or not · new findings or none · recommendation
(move to Done / return to dev).
FAIL / NEW-BUG verdicts → one extra line per finding:
**Severity:** Blocker / Critical / Major / Minor (definitions: roles/qa.md) ·
**Origin:** DEV stage (code diverges from AC) or BA/spec stage (AC/spec wrong or
missing — work returns to the BA lane).
## 5. If BLOCKED / UNCLEAR — why, and what's needed?
**Could not verify <what>** because <reason> · **Tried:** <…> · **To unblock:** <who/what>.
## Appendix (for technical readers)
verify-sheet · spec §/schema citations · debate.md · remaining evidence files.
```
## V6 — CHALLENGER SIGN-OFF
1. Write YOUR verdict card into `debate.md` FIRST (verdict + strongest evidence +
`MY WEAK SPOT:` the one thing most likely to be wrong).
2. Spawn ONE fresh agent (never forked; model `standard` per model-routing) with:
ticket verbatim + verify-sheet + evidence paths + REPORT.md + **the
`## Reviewer lens` block of every competency the verify-sheet lists under
Competencies:** + this instruction: "FALSIFY this verification: wrong
role/account? data-shaped difference? boundary untested or chosen for
convenience? a 'unique/normalised' rule never tested across representations?
an illegal transition tried only through the UI button? evidence stale or not
showing what the caption says? a FAIL whose severity or origin is wrong? any
REPORT.md sentence a non-technical reader can't follow? Work the pasted lens
items one by one. Return your own card." A challenger card that ignores a
pasted lens item without saying why is returned.
3. Challenger found a hole → run the decisive experiment (a re-run, a SELECT),
never argue in prose. Append the resolution + `Remaining dissent:` line.
4. Consensus without a run that actually executed = UNCLEAR, never PASS.
**Optional human report (BDD).** Alongside REPORT.md you MAY emit
`evd/<TICKET>/REPORT.bdd.md` — the verdict as Given/When/Then
scenarios a non-programmer scans in a minute (`docs/team/bdd-report.md`;
`bdd_report_check` guards it: every scenario complete, no code-speak, not
rambling). Code details stay in cmd_verify.md, never in a scenario.
## V7 — REPORT + CLOSE THE LOOP
1. Finalize REPORT.md (challenger's readability fixes applied). Re-run
`evd_check.py` — must be green. Then make the folder introduce itself and
produce the spreadsheet, because most of the people this verdict is for will
never open a markdown file:
```bash
python3 .vteam/scripts/evd_index.py --evd evd/<TICKET> # index block → manifest.md
python3 .vteam/scripts/xlsx_export.py --evd evd/<TICKET> --strict # <TICKET>_testcases.xlsx
```
The workbook is the same record laid out to ISO/IEC/IEEE 29119-3 — a Summary
that answers *how many defects and how bad is the worst one* on the first
screen, the cases as a test-case specification, the defects graded by
severity, a requirement traceability matrix, and an index of every evidence
file (headers in `project.language`). It **never invents a value**: a field the
pack lacks prints NOT DECLARED, and `--strict` exits 1 naming each one — fill
the pack and re-run. **Fix the pack, never the spreadsheet**: it is generated,
and a hand-edited workbook is a workbook nobody can check against the evidence.
2. **Summary to the user** = REPORT.md content (verdict + V5 table + evidence
paths + new findings + remaining dissent).
3. **Tracker comment — self-posted (full autonomy):** 3 parts (result + attached
evidence + notes/recommendation), posted right after both machine gates are
green (`evd_check.py` + challenger V6). Print the comment verbatim into the
session minutes.
3a. **Attach QA images to the tracker — mandatory, with read-back** (images live
outside git, so the tracker is the only place they survive the session):
`python3 .vteam/scripts/evd_check.py --evd evd/<TICKET> --attach <TICKET>`
— uploads every TC image (attach the `<TICKET>_testcases.xlsx` alongside —
it is the artefact a manager will actually open), re-reads the attachment
list to confirm each filename, and writes a `## TRACKER ATTACHMENTS` section (name · md5 · url)
into the root manifest. That pointer is what lets a clean checkout reproduce
the verdict when images are gone from disk.
3b. **Ticket-closing policy (FULL-AUTO at `autonomy.level: full`):** verdict
**PASS** + PR merged + CI green → transition the ticket to **Done**. Verdict
FAIL / NEW-BUG → return the ticket to **To Do + `reopen` label** (distinguishes
returned work from new work — /dev's DoR gate reads this label and points at
REPORT.md first) with a comment linking REPORT.md + severity.
**Blocker/Critical** findings → recorded in the minutes as the NEXT dev item,
before any new ticket (SLA per roles/qa.md). BLOCKED/UNCLEAR → keep In Review +
record in the minutes. Every closed ticket is listed in the minutes for the
owner's acceptance review.
4. **Commit the verification dossier (a verdict must be reproducible):**
REPORT.md, manifest.md (with TRACKER ATTACHMENTS), verifysheet.md, debate.md,
TC_*/manifest.md, db_verify.md commit via the light path — a docs branch + PR,
self-merged when CI is green (images stay out of git). QA never touches
product code, but its own text dossier MUST reach git — a verdict living on
one machine is a fabricated report.
5. **Cleanup**: reverse V3 `ZZTEST` data via the app; record any residue in the
manifest.
6. **A finding OUTSIDE the ticket's scope that deserves its own ticket → QA files
the Bug ticket itself** (full autonomy; the exemption list still waits for the
owner). Bug template — missing sections make the DoR gate red:
```
[Environment] <app URL> — branch, test account
[Steps to reproduce] 1. … 2. … 3.
[Expected] per spec §x.y (docs/specs/<feature>.md)
[Actual] what happened
[Severity] Blocker / Critical / Major / Minor (roles/qa.md)
[EVD] evd/<TICKET>/TC_n/… (boxed image, db_verify if a write is involved)
```
Labels: `qa-found` + area; estimate left for BA refinement (noted in the
ticket). Dedup against known-issues BEFORE filing (principle from V1).
7. **Close the learning loop** — append a lesson (+ INDEX line) to
`docs/qa/knowledge-base.md`, even for FAIL/BLOCKED runs (or state "no new
lesson"). Prefer GRADUATION (§0): gate-shaped → `evd_check.py`; cross-screen →
known-issues; unconditional → this workflow — then delete from KB.
## Definition of Done
- [ ] V0: assignee matches; status verifiable; announced (incl. WHICH code:
protected branch or PR branch)
- [ ] Expected derived from spec/schema with citations (never from ticket prose or code)
- [ ] Every TC ran HEADED this session against the running app — bring-up proven
by app_check's `APP: UP` line quoted in the verify-sheet, journeys driven
through `browser.mjs` journey scripts kept in their TC folders; blocked
TCs carry reason + unblock path
- [ ] Boundary TC + whole-screen sanity TC ran (not only the happy path)
- [ ] V1/V2: verify-sheet carries **Competencies:** — `qa-identity` +
`qa-requirement-smells` + every INDEX row matching this ticket; the loaded
Reviewer lens blocks went into the V6 challenger brief
- [ ] V1: requirement-smells three-line test run on each criterion; untestable
ones raised as questions to the owner before testing (not after)
- [ ] Writes verified via read-only DB checks; all test data UI-created, per the
write gate (ask when present / minutes-first when absent), ZZTEST-marked,
cleaned up
- [ ] V5 table: every ticket/dev claim mapped to an evidence file
- [ ] Every UI TC walked from its ENTRY (a click path, not a typed address); AFTER
checked incl. survives-a-reload; BACK/Cancel checked
- [ ] `evd_check.py --expect-tcs <N>` green; REPORT.md follows the template
(with the COMMIT:, VERIFIED-AT:, ENVIRONMENT: and ORACLE: lines), jargon-free body
- [ ] Every TC has TITLE: (a behaviour sentence) + KIND:; the pack holds a
`boundary` and a `whole-screen` case; folders are `TC_<n>_<what_it_proves>`;
EXPECTED/ACTUAL are observable values, never a judgement word
- [ ] Root manifest carries the COVERAGE: block (security + accessibility each a
case or a reasoned waiver) and a CURRENT evd_index block (`evd_index.py --check`)
- [ ] `xlsx_export.py --evd … --strict` green — `<TICKET>_testcases.xlsx` produced
from the pack, attached to the ticket, never committed or hand-edited
- [ ] Every *_boxed.png hugs the exact region it proves, caption burned in below
- [ ] QA images attached with read-back + TRACKER ATTACHMENTS section in the manifest
- [ ] V1: change ledger filtered for this ticket's shard/screen
- [ ] Failing verdict: every finding carries Severity + Origin (DEV / BA-spec) —
evd_check blocks otherwise; ticket returned To Do + `reopen`
- [ ] Out-of-scope findings: deduped, Bug ticket filed per template (labels
`qa-found` + area) or explicitly judged not ticket-worthy
- [ ] The text dossier COMMITTED via the light path — the verdict reproducible from git
- [ ] debate.md has both cards + resolution; the challenger was a real fresh agent
- [ ] NO product code changed; findings reported, not fixed
- [ ] Tracker comment posted after both machine gates; transition per V7.3b
- [ ] KB preflight done at V1; lesson appended at V7
- [ ] Every phase narrated with a ▶ line; all user-facing output passes the
plain-language bar