Quality Playbook

---
name: "quality-playbook"
description: >-
  Run a complete quality engineering audit on any codebase. Derives behavioral requirements from the
  code, generates spec-traced functional tests, runs a three-pass code review with regression tests,
  executes a multi-model spec audit (Council of Three), and produces a consolidated bug report with
  TDD-verified patches. Finds the 35% of real defects that structural code review alone cannot catch.
  Works with any language. Trigger on 'quality playbook', 'spec audit', 'Council of Three',
  'fitness-to-purpose', or 'coverage theater'.
license: "Complete terms in LICENSE.txt"
metadata:
  author: "Andrew Stellman"
  github: "https://github.com/andrewstellman/quality-playbook"
  version: "1.5.6"
---

# Quality Playbook generator

Run the Quality Playbook workflow for a codebase by exploring the domain, generating requirements and quality artifacts, reviewing code, auditing specs, reconciling defects with TDD evidence, and verifying the final quality system.

## When to invoke

- "Run the Quality Playbook."
- "Do a Council of Three spec audit."
- "Generate a quality system for this repo."
- "Find coverage theater and fitness-to-purpose gaps."
- "Run a bootstrap self-audit."

## Plan Overview — read this first, then explain it to the user

Before reading any other section of this skill, understand the plan and its dependencies. Each phase produces artifacts that the next phase depends on. Skipping or rushing a phase means every downstream phase works from incomplete information.

**Phase 0 (Prior Run Analysis):** If previous quality runs exist, load their findings as seed data. This is automatic and only applies to re-runs.

**Phase 1 (Explore):** Run the v1.5.3 documentation intake first (`python -m bin.reference_docs_ingest <target>` to walk `reference_docs/` — `cite/` files produce `quality/formal_docs_manifest.json` records; top-level files are loaded as Tier 4 context via `reference_docs_ingest.load_tier4_context(<target>)`). Then explore the codebase in three stages: open exploration driven by domain knowledge, domain-knowledge risk analysis, and selected structured exploration patterns. Write all findings to `quality/EXPLORATION.md`. This file is the foundation — Phase 2 reads it as its primary input.

**Phase 2 (Generate):** Read EXPLORATION.md and produce the quality artifacts: requirements, constitution, functional tests, code review protocol, integration tests, spec audit protocol, TDD protocol. (`AGENTS.md` at the target's repo root is generated by the orchestrator AFTER Phase 6, not by you in Phase 2 — see "File 6" below for the contract.)

**Phase 3 (Code Review):** Run the three-pass code review against HEAD. Write regression tests for every confirmed bug. Generate patches.

**Phase 4 (Spec Audit):** Three independent AI auditors review the code against requirements. Triage with verification probes. After triage, the same Council runs the v1.5.3 Layer-2 semantic citation check — one prompt per reviewer, structured per-REQ verdicts for every Tier 1/2 citation, output to `quality/citation_semantic_check.json`. Write regression tests for net-new findings.

**Phase 5 (Reconciliation):** Close the loop — every bug from code review and spec audit is tracked, regression-tested or explicitly exempted. Run TDD red-green cycle. Finalize the completeness report.

**Phase 6 (Verify):** Run self-check benchmarks against all generated artifacts. Check for internal consistency, version stamp correctness, and convergence.

**Phase 7 (Present, Explore, Improve):** Present results to the user with a scannable summary table, offer drill-down on any artifact, and provide a menu of improvement paths (iteration strategies, requirement refinement, integration test tuning). This is the interactive phase where the user takes ownership of the quality system.

Every bug found traces back to a requirement, and every requirement traces back to an exploration finding.

**The critical dependency chain:** Exploration findings → EXPLORATION.md → Requirements → Code review + Spec audit → Bug discovery. A shallow exploration produces abstract requirements. Abstract requirements miss bugs. The exploration phase is where bugs are won or lost.

**MANDATORY FIRST ACTION:** After reading and understanding the plan above, print the following message to the user, then explain the plan in your own words — what you'll do, what each phase produces, and why the exploration phase matters most. Emphasize that exploration starts with open-ended domain-driven investigation, followed by domain-knowledge risk analysis that reasons about what goes wrong in systems like this, then supplemented by selected structured patterns. Do not copy the plan verbatim; paraphrase it to demonstrate understanding.

> Quality Playbook v1.5.6 — by Andrew Stellman
> https://github.com/andrewstellman/quality-playbook

Generate a complete quality system tailored to a specific codebase. Unlike test stub generators that work mechanically from source code, this skill explores the project first — understanding its domain, architecture, specifications, and failure history — then produces a quality playbook grounded in what it finds.

## How to run this — v1.5.4 self-encoded invocation contract

If the operator hands you this skill (or points you at any QPB-installed target) and says **"Run the Quality Playbook"** — possibly with a hint like "this is a bootstrap run" or "run on itself" or "self-audit" — this section tells you exactly what to do. The operator should not need to provide additional instructions; the canonical invocation, the defaults, the guardrails, and the output contract all live here.

### Pick your execution mode

QPB ships in two execution shapes. Pick the one that matches your runtime — the wrong choice produces the codex-on-codex indirection pathology surfaced by the 2026-04-30 bootstrap test.

| Mode | When this is you | What you do |
|------|------------------|-------------|
| **A. Skill-direct (UI-context)** | You are a coding agent (Claude Code, Cursor, Copilot, Codex desktop, etc.) handed this skill in your own chat. Your runtime IS the reasoning loop — you read files, you write files, you decide. | Walk through Phase 1 → Phase 6 yourself using the externalized phase prompts in `phase_prompts/`. Write artifacts into the target's `quality/` directory directly. No subprocess, no runner. |
| **B. Runner-driven (CLI-automation)** | The operator is invoking `python3 -m bin.run_playbook` deliberately — to batch across multiple targets, drive a headless CI run, or fan out per-phase work to a different model than the one reading this prose. | The orchestrator spawns a CLI agent (`claude`, `copilot`, `codex`, or `cursor`) per phase. You (or whoever is reading this) are the operator-side control loop, not the per-phase reasoner. |

**Both modes use the same phase prompt content** — the `phase_prompts/*.md` files at the repo root are the single source of truth, loaded by `bin/run_playbook.py::_load_phase_prompt` and read directly by Mode A walkthroughs. The only thing the two modes differ on is WHO drives — you (Mode A) or the orchestrator subprocess-spawning a CLI agent (Mode B).

**When in doubt, default to Mode A.** If the operator wanted runner-driven invocation they would have run the runner themselves; if they pasted "Run the Quality Playbook" into your chat, they want you to drive. The Mode B section below tells you what to do *if* the operator explicitly invokes the runner.

### Mode A — skill-direct walkthrough (UI-context)

The operator's prompt is just **"Run the Quality Playbook"** (or "run on itself", "self-audit", etc.). You drive every phase inline.

For each phase 1..6, in order:

1. **Load the phase prompt.** Read `phase_prompts/phaseN.md` (resolve via the same install-location fallback list documented for `references/` below). For `phase1.md`, substitute `{seed_instruction}` (the prelude that says "skip Phase 0/0b" — empty string when seeds are allowed) and `{role_taxonomy}` (the taxonomy block rendered from the role taxonomy below). For `phase2.md` through `phase6.md`, the file is pure-literal — read it verbatim.
2. **Execute the phase per the prompt.** Read the inputs the prompt names, do the analysis, write outputs into the target's `quality/` directory.
3. **STOP at the end-of-phase boundary.** Every phase prompt ends with an "IMPORTANT: Do NOT proceed to Phase N+1" instruction. Honor it. The operator advances to the next phase by saying so.

You are responsible — without the orchestrator's structural backstop — for the same source-unchanged invariant the runner enforces: **do NOT modify any file outside the target's `quality/` directory**. In Mode B the gate would catch this; in Mode A you are the gate. The 2026-04-30 bootstrap test specifically failed on a Phase 2 LLM modifying the target's root `AGENTS.md` — the same failure mode applies in Mode A.

For the bootstrap-run (self-audit) variant of Mode A, see "Bootstrap mode" below — the only delta is that the target IS the QPB repo, so cite the same `phase_prompts/` files you read from.

#### Mode A scope — what's covered, what's Mode-B-only

Council 2026-04-30 P1-3: the per-phase walkthrough above scopes Mode A to **phases 1..6**. The following surfaces are deliberately Mode-B-only — if the operator wants them, point them at the runner instead of trying to drive them yourself:

- **Phase 0 / Phase 0b (seed injection from prior runs).** The orchestrator handles seed discovery, prior-run scanning, and seed-prompt injection. In Mode A, treat every run as `--no-seeds` (skip Phase 0/0b entirely, start at Phase 1). If the operator explicitly asks for seed-driven exploration, hand off to Mode B (`python3 -m bin.run_playbook --with-seeds <target>`).
- **Phase 7 (interactive Present / Explore / Improve).** This phase is a back-and-forth dialogue with the operator about the generated artifacts; it has no pre-baked prompt in `phase_prompts/`. After Phase 6 in Mode A, present the artifact summary table inline (see "What this run produces" below for the file list) and let the operator drive what to explore next conversationally — that IS Phase 7. There is no orchestrator subprocess to spawn.
- **Iteration strategies (gap / unfiltered / parity / adversarial).** Iterations re-enter the playbook with a strategy-specific addendum. In Mode A, after Phase 6 completes cleanly, hand off to Mode B for iterations: `python3 -m bin.run_playbook --next-iteration --strategy <name> <target>`. The iteration prompts (`phase_prompts/iteration.md`) ARE single-source-of-truth, but the iteration-orchestration loop (rotating through gap → unfiltered → parity → adversarial) is the runner's job. A Mode A operator who wants iterations after Phase 6 should be told: "Phase 6 is done; run `python3 -m bin.run_playbook --full-run <target>` to get all four iteration strategies, or pick one strategy explicitly with `--next-iteration --strategy gap`."

If the operator asks for one of these surfaces in Mode A and the request is ambiguous (e.g., "also do the iterations"), surface the mode-handoff explicitly rather than improvising — improvisation is how the prompt content drifts away from the runner's canonical loop.

### Mode B — runner-driven invocation (CLI-automation)

The operator runs `python3 -m bin.run_playbook` themselves (typically because they want batching, headless CI, or to route per-phase work to a different model). The orchestrator at `bin/run_playbook.py` spawns a CLI agent per phase, feeds it the externalized phase prompt, and aggregates the result.

#### Canonical invocation

The orchestrator is the entry point. Always invoke it as a Python module:

```
python3 -m bin.run_playbook <target>
```

**Never invoke it script-style** (`python bin/run_playbook.py ...`). The runtime guard exits with `EX_USAGE=64` because relative imports require packaged execution.

`<target>` is the path to the project to audit. For a bootstrap run (target IS the QPB repo), pass `.` from the repo root. For any other target, pass the path to that target's repo root.

#### Default behavior (no flags)

Bare invocation triggers a **full run**: all 6 phases (Explore → Generate → Code Review → Spec Audit → Reconciliation → Verify) followed by all 4 iteration strategies (gap → unfiltered → parity → adversarial), executed synchronously in the same session. Any prior `quality/` directory is auto-archived to `quality/previous_runs/<TIMESTAMP>/` before the new run starts.

This is the canonical operator path. Don't ask permission to add flags; the defaults are the answer.

When the bare invocation fires, the orchestrator emits a one-line stderr banner naming the cost change vs. v1.5.3 (~5–10× the legacy "Phase 1 only" default). That banner is informational; let it scroll.

#### Common overrides

Use only when the operator asks for something specific:

| Need | Flag | Effect |
|------|------|--------|
| Run a single phase | `--phase N` (where N ∈ 1..6) | Recovers the v1.5.3 "explore only" pattern with `--phase 1`. |
| Skip iteration strategies | omit `--iterations` and pass `--phase 1,2,3,4,5,6` | Phases run; iterations don't. |
| Specific iteration | `--strategy <name> --next-iteration` | Iterates on an existing `quality/` run with a chosen strategy. |
| Multi-target | pass several positional targets | Each runs independently. |
| Per-phase CLI agent | `--claude` / `--copilot` / `--codex` / `--cursor` | Picks which CLI runner the orchestrator spawns. Default is `--copilot`. v1.5.4 added the `--cursor` runner (cursor-cli 3.1+). |

#### Recovering from a partial / aborted runner-driven run

Council 2026-04-30 P1-4: the operator-hygiene guidance for cleaning up after an aborted run lives in the **Bootstrap mode** section below ("Bootstrap-run operator hygiene") — the recovery is identical in Mode B: `git restore quality/` to discard the partial Phase 1/2 output, then re-invoke. **Do NOT** edit files outside `quality/` to "tidy up" — the source-unchanged invariant trips on the very next run. See the Bootstrap-mode hygiene paragraph for the full mechanic; it applies regardless of whether the abort happened during a self-audit run or against an external target.

### Bootstrap mode (running QPB on itself)

When the operator says "this is a bootstrap run" or "we're running QPB on itself" or "self-audit":

1. Confirm the working directory is the QPB repo root (or `cd` there).
2. Invoke `python3 -m bin.run_playbook .` — same canonical form, target is `.`.
3. The orchestrator handles archival of the existing `quality/` tree to `quality/previous_runs/<TIMESTAMP>/` automatically; you don't need to clean anything manually.

The run proceeds the same way as any other target. The only difference is that the audit subject IS the playbook itself, so the produced artifacts describe QPB's own quality system.

**Bootstrap-run operator hygiene — recovering from a partial / aborted run.** If a prior bootstrap run aborted mid-flight (e.g., the source-unchanged invariant tripped, a phase prompt errored, the operator hit Ctrl-C), the working tree may contain a half-written `quality/` directory plus a `quality/previous_runs/<TIMESTAMP>/.partial` sentinel marking the abandoned archive. Before re-invoking, run `git restore quality/` (and, if you want a clean slate, `git clean -fd quality/`) to drop any uncommitted Phase 1/2 output from the aborted run. The orchestrator will re-archive the now-pristine `quality/` tree and start clean. **Do NOT** edit files outside `quality/` to "tidy up" — anything outside `quality/` is QPB source; touching it for cleanup will trip the source-unchanged invariant on the very next run. The 2026-04-30 bootstrap test surfaced this exact recovery question: the operator had a half-written `quality/` from an aborted Phase 2 and re-running without restoring left stale Phase 1 artifacts that confused the next run's archival.

### v1.5.4 mechanics (pointer-style, not duplicating the design doc)

What's new vs. v1.5.3, in pointer form (the canonical architecture lives in `docs/design/QPB_v1.5.4_Design.md` Part 1):

- **Phase 1 produces `quality/exploration_role_map.json`** — per-file role tagging done AI-driven during exploration. Each in-scope file gets a role from the taxonomy (`skill-prose`, `skill-reference`, `skill-tool`, `code`, `test`, `docs`, `config`, `fixture`, `formal-spec`, `playbook-output`). The role map drives every downstream pipeline-activation decision.
- **`INDEX.md` uses `schema_version: "2.0"`** with a `target_role_breakdown` field carrying the per-role counts and percentages. The v1.5.3 `target_project_type` enum is retired (legacy archives stay readable).
- **Pipelines activate from the role map, not from a project-type label.** The four-pass skill-derivation pipeline runs over files tagged `skill-prose` / `skill-reference`. The code-review pipeline runs over files tagged `code`. The prose-to-code divergence check runs over files tagged `skill-tool`. When the role map shows zero of a role, that pipeline no-ops cleanly. There is no Code/Skill/Hybrid trichotomy — both pipelines run when both surfaces are present (the "always-Hybrid downstream" model).
- **Archive directory is `quality/previous_runs/`** (was `quality/runs/` in v1.5.3); legacy archives at the old path remain readable.
- **End-of-Phase-6 reorganization** moves intermediate artifacts under `quality/workspace/` so the top-level `quality/` directory is dominated by canonical deliverables (REQUIREMENTS.md, BUGS.md, etc.). The gate's path resolver reads from both layouts.

You don't need to re-derive any of this in your prompt-side reasoning; the orchestrator's prompts already encode it. If you encounter a phase prompt that conflicts with the architecture summarized here, follow the phase prompt — it's the canonical source for the per-phase contract.

### Guardrails (machine-checkable; treat as hard constraints)

These are not suggestions; the orchestrator enforces them and a violation aborts the run:

1. **Synchronous execution — no sub-agent delegation.** Run every phase yourself in the same session. **Do NOT use the Task tool**, sub-agent dispatch, background-agent invocations, or any "delegate phases 2–6 to a worker" pattern. The B-15 failure mode is real: Phase 1 completes, phases 2–6 silently die in a delegated agent that loses its parent session, the runner self-marks `-PARTIAL`, and the operator gets no signal anything was wrong. v1.5.4 prompts forbid this explicitly.
2. **Don't patch QPB source mid-run.** If you encounter a bug in `bin/`, `.github/skills/`, `agents/`, `references/`, `SKILL.md`, `schemas.md`, or `AGENTS.md` during the run, **STOP and report**: name the file:line, describe the failure, propose a fix shape — but do NOT apply the fix. The orchestrator captures a git-SHA baseline at run start and verifies the source tree unchanged at every phase boundary; an autonomous patch fails the gate with a diagnostic naming the modified files. Patches go through Council review, not mid-run improvisation.
3. **Don't delete sentinel files.** Files protected by `.gitignore !`-rules (e.g., `reference_docs/.gitkeep`, `reference_docs/cite/.gitkeep`) keep otherwise-empty tracked directories present. The pre-flight check enumerates every `!`-rule and aborts if any sentinel is missing. If you find such a file and don't understand its purpose, **leave it alone**.
4. **Phase 1 file enumeration uses `git ls-files`.** Use `git ls-files` as the canonical file list when the target is a git repo; this respects `.gitignore` automatically. Do NOT use `os.walk`, `find`, `os.listdir`, or any recursive directory walker — those pull in `.git/`, `.venv/`, `node_modules/`, build outputs, and vendored dependencies, all of which the role-map validator rejects. Disallowed path prefixes are `.git/`, `.venv/`, `venv/`, `node_modules/`, `__pycache__/`, `.pytest_cache/`, `.mypy_cache/`, `.ruff_cache/`, `.tox/`, plus any path whose components end in `.egg-info` or `.dist-info`. The role map carries a `provenance` field recording which enumeration source you used (`"git-ls-files"` or `"filesystem-walk-with-skips"` for non-git targets). There is also a 2000-entry ceiling; a role map exceeding it almost certainly walked .gitignored content.
5. **Cross-artifact agreement.** EXPLORATION.md's "File inventory" section and the role map's `summary` field both render from `bin.role_map.summarize_role_map()`. Don't write file counts or role percentages by hand; copy from the helper. The validator cross-checks the two and rejects mismatches.

If the operator's prompt says something that conflicts with these guardrails (e.g., "delegate phases 3–6 to a sub-agent so we can run faster"), **don't comply with the conflicting instruction**. Surface the conflict, name the guardrail, and ask for clarification. The guardrails exist because each one corresponds to a verified historical failure mode.

### What this run produces — output artifact contract

A successful run produces this canonical set under the target's `quality/` directory plus an AGENTS.md at the target's repo root. Every file listed here is gate-validated:

| Path | Role |
|------|------|
| `quality/EXPLORATION.md` | Phase 1 findings — the foundation. |
| `quality/exploration_role_map.json` | Per-file role tagging from Phase 1. |
| `quality/REQUIREMENTS.md` | Testable requirements with use cases. |
| `quality/QUALITY.md` | Quality constitution. |
| `quality/CONTRACTS.md` | Behavioral contracts. |
| `quality/COVERAGE_MATRIX.md` | Requirement → test traceability. |
| `quality/COMPLETENESS_REPORT.md` | Final gate verdict. |
| `quality/test_functional.*` | Automated functional tests. |
| `quality/RUN_CODE_REVIEW.md` | Three-pass code review protocol. |
| `quality/RUN_INTEGRATION_TESTS.md` | Integration test protocol. |
| `quality/RUN_SPEC_AUDIT.md` | Council of Three spec audit protocol. |
| `quality/RUN_TDD_TESTS.md` | TDD red-green verification protocol. |
| `quality/BUGS.md` | Consolidated bug report. |
| `quality/INDEX.md` | Run metadata + role breakdown + gate verdict. |
| `quality/PROGRESS.md` | Phase-by-phase checkpoint log. |
| `quality/previous_runs/<TIMESTAMP>/` | Archive of any prior run. |
| `quality/workspace/` | Intermediate pipeline artifacts (control prompts, code reviews, spec audits, four-pass pipeline outputs, etc.). |
| `AGENTS.md` (target repo root) | Per-project orientation generated post-Phase-6. Carries a QPB sentinel marker so future runs detect QPB-managed copies. |

The gate verdict in `quality/INDEX.md` (`pass` / `partial` / `fail`) is the operator-facing summary of how the run went. If it's anything other than `pass`, surface why before considering the run done.

### Locating reference files

This skill references files in a `references/` directory (e.g., `references/iteration.md`, `references/review_protocols.md`). The location depends on how the skill was installed. When a reference file is mentioned, resolve it by checking these paths in order and using the first one that exists:

1. `references/` (relative to SKILL.md — works when running from the skill directory)
2. `.claude/skills/quality-playbook/references/` (Claude Code installation)
3. `.github/skills/references/` (GitHub Copilot flat installation)
4. `.github/skills/quality-playbook/references/` (alternate Copilot installation)

All reference file mentions in this skill use the short form `references/filename.md`. If the relative path doesn't resolve, walk the fallback list above.
## Extended reference

Additional detailed guidance was moved to [references/extended-guide.md](references/extended-guide.md) to keep this skill within the progressive-disclosure budget.

## Progressive disclosure and bundled resources

- `phase_prompts/phaseN.md`: single source of truth for Mode A phase walkthroughs and runner-driven phase prompts.
- `references/`: supporting protocols, iteration guidance, and review material resolved through the documented fallback list.
- `agents/`: bundled role definitions used by the playbook where applicable.

## Output template

```markdown
## Quality Playbook result

**Status:** pass | partial | fail | blocked
**Target:** `{{target}}`
**Mode:** Skill-direct | Runner-driven

| Artifact | Status |
| --- | --- |
| `quality/EXPLORATION.md` | {{status}} |
| `quality/REQUIREMENTS.md` | {{status}} |
| `quality/BUGS.md` | {{status}} |
| `quality/INDEX.md` | {{gate_verdict}} |
| `AGENTS.md` | {{generated_or_mode_b_only}} |

### Verdict
{{gate_summary_and_next_step}}
```

## Quality gate

- [ ] The selected execution mode matches the runtime and operator request.
- [ ] Mode A does not modify files outside the target `quality/` directory.
- [ ] Mode B invokes the runner as `python3 -m bin.run_playbook <target>`, never script-style.
- [ ] Phase outputs are not skipped; each downstream phase reads the artifacts produced earlier.
- [ ] The final response surfaces the gate verdict from `quality/INDEX.md`.

## References

- https://github.com/andrewstellman/quality-playbook