Sk Create Manual Testing Playbook

---
name: sk-create-manual-testing-playbook
description: Author manual testing playbook packages with deterministic scenarios, evidence collection, and multi-agent execution planning.
allowed-tools: [Read, Write, Edit, Bash, Grep, Glob]
version: 1.0.2.0
---

<!-- generated by sync-skills-hermes.cjs; do not edit -->
> Canonical source: `.skilled/skills/sk-doc/sk-create-manual-testing-playbook/`. This is a generated markdown-only copy so Hermes can load the
> skill without scanning the canonical directory; read `references/`, `assets/` and `scripts/`
> from the canonical path above.

<!-- Keywords: manual testing playbook, testing playbook, deterministic scenario, evidence collection, operator validation, multi-agent execution, release readiness, create:manual-testing-playbook -->

# Manual Testing Playbook Creation

`create-manual-testing-playbook` is the manual-validation package workflow for the `sk-doc` family. It authors `manual-testing-playbook/` packages for skills and systems that need reproducible operator-facing scenarios, evidence capture, release-readiness review, and realistic orchestration or multi-agent execution planning.

Core principle: keep shared rules in the root playbook, keep execution truth in per-feature files, and make every scenario deterministic enough that another operator can reproduce the verdict.

---

## 1. WHEN TO USE

### Activation Triggers

Use this workflow when the request involves:

- `/create:manual-testing-playbook`.
- Creating `manual-testing-playbook/manual-testing-playbook.md`.
- Building one-file-per-feature manual test scenarios.
- Designing deterministic prompts, command sequences, expected signals, evidence, pass/fail criteria, and failure triage.
- Planning manual validation across multiple operators, agents, CLIs, MCP tools, or runtime surfaces.
- Converting an ad hoc release checklist into a reusable evidence-driven playbook package.
- Aligning a manual testing playbook with a feature catalog.

Keyword triggers: `manual testing playbook`, `/create:manual-testing-playbook`, `testing playbook`, `playbook system`, `deterministic scenario`, `evidence collection`, `operator validation`, `multi-agent execution`, `release readiness`.

Strong signals that a playbook is warranted:

- 5+ distinct features need manual validation.
- Release decisions depend on structured evidence.
- Realistic orchestration behavior matters.
- Multiple operators or agents will execute the same scenarios.
- Automated tests cover internals but not operator-visible behavior.

### When NOT to Use

Use another `sk-doc` packet when:

- Test steps fit cleanly in a spec folder checklist.
- The feature is one-off or experimental.
- The system has only a few manually testable behaviors.
- Automated tests already cover the only meaningful acceptance criteria.
- The user asks for a feature catalog rather than executable validation scenarios. Use `create-feature-catalog`.
- The user asks for a README, skill, agent, command, benchmark package, flowchart, or changelog. Use `create-readme`, `create-skill`, `create-agent`, `create-command`, `create-benchmark`, `create-flowchart`, or `create-changelog`.
- The task audits, validates, scores, or optimizes an existing playbook without creating or extending the package. Use `create-quality-control`.

---

## 2. SMART ROUTING

### Decision Rule

Decision rule:

```text
Need reusable manual validation with captured evidence?
YES -> Create a playbook package
NO  -> Keep test steps in spec/checklist docs
```

### Family Boundary

This packet owns manual testing playbook packages only. It consumes shared `sk-doc` standards from `../shared`, but the advisor identity lives at the `sk-doc` hub root. Do not add a packet-local `graph-metadata.json`.

### Smart Router Pseudocode

For this flat-reference packet, the canonical resilient router discovers resources at call
time, guards and loads only what exists, scores the two authoring scopes, and returns a
disambiguation checklist rather than silently loading nothing:

```python
from pathlib import Path

SKILL_ROOT = Path(__file__).resolve().parent
RESOURCE_BASES = (SKILL_ROOT / "references", SKILL_ROOT / "assets")
DEFAULT_RESOURCE = "references/README.md"

# Two authoring scopes; keywords come from this packet's activation triggers.
INTENT_MODEL = {
    "root_playbook": {"weight": 4, "keywords": ["manual testing playbook", "/create:manual-testing-playbook", "testing playbook", "release readiness"]},
    "per_feature_scenario": {"weight": 4, "keywords": ["deterministic scenario", "evidence collection", "operator validation", "multi-agent execution"]},
}
UNKNOWN_FALLBACK_CHECKLIST = [
    "Confirm the target skill or system and the feature set under test",
    "Confirm root playbook index scope vs per-feature scenario file scope",
    "Confirm the evidence and pass/fail validation expectations",
]

def discover_markdown_resources() -> set[str]:
    docs = []
    for base in RESOURCE_BASES:
        if base.exists():
            docs.extend(path for path in base.rglob("*.md") if path.is_file())
    return {doc.relative_to(SKILL_ROOT).as_posix() for doc in docs}

def _guard_in_skill(relative_path: str) -> str:
    resolved = (SKILL_ROOT / relative_path).resolve()
    resolved.relative_to(SKILL_ROOT)
    if resolved.suffix.lower() != ".md":
        raise ValueError(f"Only markdown resources are routable: {relative_path}")
    return resolved.relative_to(SKILL_ROOT).as_posix()

def load_if_available(relative_path, inventory, loaded, seen) -> None:
    guarded = _guard_in_skill(relative_path)
    if guarded in inventory and guarded not in seen:
        load(guarded)
        loaded.append(guarded)
        seen.add(guarded)

def score_intents(request) -> dict:
    text = request.text.lower()
    scores = {intent: 0 for intent in INTENT_MODEL}
    for intent, cfg in INTENT_MODEL.items():
        for kw in cfg["keywords"]:
            if kw in text:
                scores[intent] += cfg["weight"]
    return scores

def route_manual_testing_playbook_request(request):
    inventory = discover_markdown_resources()
    loaded, seen = [], set()
    scores = score_intents(request)

    if max(scores.values() or [0]) < 4:                      # Tier 1: unclear scope
        load_if_available(DEFAULT_RESOURCE, inventory, loaded, seen)
        return {
            "load_level": "UNKNOWN_FALLBACK",
            "needs_disambiguation": True,
            "disambiguation_checklist": UNKNOWN_FALLBACK_CHECKLIST,
            "resources": loaded,
        }

    scope = max(scores, key=scores.get)                       # Tier 2: happy path
    # Flat resource topology: no references/<key>/ subdirectories. The scope selects the
    # authoring target documented above, not a keyed subtree; load the flat refs that exist.
    for path in sorted(inventory):
        load_if_available(path, inventory, loaded, seen)
    return {"scope": scope, "resources": loaded}
```

---

## 3. OUTPUT PACKAGE CONTRACT

### Canonical Package Shape

Author this layout:

```text
manual-testing-playbook/
|-- manual-testing-playbook.md
|-- category-name/
|   |-- feature-name.md
|   `-- another-feature-name.md
`-- another-category/
    `-- feature-name.md
```

Package invariants:

- The root file is always `manual-testing-playbook.md`.
- Per-feature files live in root-level category folders.
- Category directories use descriptive kebab-case names such as `category-name` (no numeric prefix).
- Per-feature files use stable kebab-case slugs such as `feature-name.md`; no numeric file prefix.
- Per-feature snippet order is defined by the root playbook listing order.
- Display order is owned by the root playbook index (`manual-testing-playbook.md`), not the folder name.
- Benchmark tier is owned by the per-feature file's optional `stage:` frontmatter field (`routing` default, or `holdout`/`negative`), not by a filename token.
- Every feature ID maps to exactly one per-feature file.

Two runtimes, where a system ships as both a skill and a hosted assistant:

- Write two scenario sets, never one system tested twice. Category folders carry a
  prefix naming the runtime, and every per-feature file carries a matching
  `- Runtime:` line in `SOURCE METADATA`.
- Each set opens with one identity handover scenario, named as a precondition by
  every other scenario in that set. It passes only when the reply carries a
  verbatim string present in that runtime's own instructions and absent from the
  other's, plus the delivery contract only that runtime can honour. A reply that
  could have come from either runtime is a FAIL.
- A handover string is proved against each runtime's whole load surface, not
  against its top-level instruction file alone. A skill runtime loads that file
  and everything the file sends it to. A hosted runtime loads its own
  instructions and every document attached to it. A string absent from one file
  and present three directories away has not been proved.
- Read the string at run time where it can move. Grading on a literal version
  line fails the next time that version is bumped, which makes a version check
  wearing an identity check's clothes.

Contract boundary:

- The operator-scenario contract is the five-section, execution-evidence contract
  defined here and checked by `scripts/validate-playbook-package.cjs`.
- The routing-gold contract is the compact typed-routing schema checked by
  `sk-create-skill/scripts/validate-playbook-topology.cjs`.
- `playbook-corpus-manifest.json` is an explicit whole-tree override map consumed only
  by the operator validator. A listed routing-gold tree is excluded from operator
  checks; for every other file, a non-empty `expected_workflow_mode` plus at least one
  `expected_leaf_resources` pair in frontmatter classifies that file as routing gold.
  Files without that signature are operator-scenario files. The topology gate does not
  read this manifest and keeps its existing boundary.

Do not create:

- A `snippets/` subtree for canonical per-feature files.
- Separate canonical `review_protocol.md`.
- Separate canonical `subagent_utilization_ledger.md`.
- Duplicated scenario truth across multiple sidecar files.

### Root Playbook Responsibilities

`manual-testing-playbook/manual-testing-playbook.md` is the package directory and review surface. It owns:

- Frontmatter and H1 intro.
- Global overview and coverage note.
- Global preconditions.
- Global evidence requirements.
- Deterministic command notation rules.
- Integrated review protocol and release-readiness rules.
- Integrated orchestration and wave-planning guidance.
- Category sections with short per-feature summaries.
- Automated test cross-reference section.
- Feature catalog cross-reference index.

Root summaries should be concise but useful:

- Provide enough context to understand what each scenario covers.
- Preview operator intent and prompt shape.
- Do not duplicate the full execution matrix from the per-feature file.

Root-to-feature rule: the root document explains package-level policy; per-feature files carry scenario-specific execution truth.

### Per-Feature File Responsibilities

Each per-feature file is the canonical scenario contract for one feature ID.

Required per-feature section order:

1. `## 1. OVERVIEW`
2. `## 2. SCENARIO CONTRACT`
3. `## 3. TEST EXECUTION`
4. `## 4. REFERENCES` or `## 4. SOURCE FILES`
5. `## 5. SOURCE METADATA`

Each per-feature file must include:

- Frontmatter with `title`, `description`, and a 4-part `version`.
- Realistic user request when it clarifies user intent.
- Operator prompt or orchestrator prompt.
- Exact prompt in the scenario table when a table is used.
- Exact command sequence.
- Expected signals.
- Evidence requirements.
- Pass/fail criteria.
- Failure triage.
- Root playbook link.
- Feature catalog link when applicable.

---

## 4. RESULTS STORAGE CONTRACT

A playbook is a corpus. Running it produces evidence, and that evidence has a home: the same
`benchmark/` tree that holds every other measurement of the skill. Without this rule a run leaves
nothing behind, and the next person has no way to know the playbook was ever executed.

### Where A Run Lands

```text
<skill>/
|-- manual-testing-playbook/          # the corpus, an input, never rewritten by a run
`-- benchmark/
    `-- reports/
        |-- README.md                 # the run index, one row per folder
        `-- 2026-07-29--manual-testing-playbook--goal-hook/
            |-- README.md
            |-- skill-benchmark-report.json
            |-- skill-benchmark-report.md
            |-- results.csv
            |-- failed-runs.md
            |-- findings-and-recommendations.md
            `-- source.md
```

Run folders are named `<YYYY-MM-DD>--<subject>--<variant>`, dated by execution. When a run is
**feature-scoped** — a hand-derived validation of one feature or scenario group rather than a full-corpus
harness sweep — name the `<variant>` for the feature (e.g. `goal-hook`) and record the model/executor
inside the report, so the folder stays legible across models. `create-benchmark` owns the grammar in full;
see its naming convention section for the field vocabulary and the one carve-out.

### Manual scenario completion

<!-- MANUAL_PLAYBOOK_RESULT_PERSISTENCE_CONTRACT -->
A manual scenario is incomplete until its `PASS`, `FAIL`, or `SKIP` outcome, its reason and its evidence paths are
recorded in the skill's `benchmark/reports/<dated-run-label>/` folder. The wrapper that once persisted them
was retired with the Lane C harness, so the record is written by hand until a replacement exists.

### Rules

- The corpus is an input. A run never edits `manual-testing-playbook/`, and gold that needs to change
  gets a corpus revision rather than a rewritten scenario.
- Curated summaries and result tables live in the run folder. Raw transcripts and copied artifacts stay
  in the spec packet that produced them, named in `source.md`.
- A run whose result changes gets a new folder. A prior run is never overwritten.
- Every file in the folder derives from the run record. A field the run did not capture reads as not
  recorded; it is never filled in from inference.

---

## 5. HOW IT WORKS - AUTHORING WORKFLOW

Follow this sequence:

1. Confirm the target skill or system, package owner, feature set, and whether a feature catalog already exists.
2. Decide whether a manual testing playbook is appropriate using the decision rule in this skill.
3. Define category directories using a descriptive kebab-case slug such as `category-name`.
4. Define stable feature IDs using a consistent `{PREFIX}-{NNN}` pattern.
5. Create the root `manual-testing-playbook/` directory.
6. Create `manual-testing-playbook/manual-testing-playbook.md` from `assets/manual-testing-playbook-template.md`.
7. Create one per-feature file for each feature ID from `assets/manual-testing-playbook-snippet-template.md`.
8. Write root package policy before writing scenario-specific exceptions.
9. Write each per-feature prompt and execution truth before polishing root summary prose.
10. Fill each scenario contract with the nine display fields: Feature ID, Feature Name, Scenario Objective, Exact Prompt, Exact Command Sequence, Expected Signals, Evidence, Pass/Fail Criteria, Failure Triage. The operator validator's eight unconditional checks cover Feature ID plus the seven execution/link fields; realistic user request, table prompt, and feature-catalog link are conditional checks.
11. Write root category summaries and link every per-feature file.
12. Add automated-test anchors when they exist.
13. Add feature-catalog cross-references when a catalog exists.
14. Explicitly note when a scenario has no dedicated catalog entry.
15. Review destructive scenarios and isolate them in root preconditions and recovery guidance.
16. Validate the root document with shared validators.
17. Manually spot-check per-feature structure, prompt synchronization, feature ID counts, and local links.
18. Report any remaining manual validation scope honestly.

Authoring sequence matters:

- Decide categories and IDs before writing summaries.
- Write root package policy before scenario exceptions.
- Write execution truth before root prose.
- Keep scenario-specific execution truth in per-feature files.

---

## 6. SCENARIO DESIGN RULES

### Determinism

Each scenario must be reproducible by another operator. Include exact prompts, exact command sequences, observable expected signals, captured evidence, and binary pass/fail criteria.

Execution status is limited to:

- `PASS`
- `FAIL`
- `SKIP` with a specific sandbox blocker

Do not classify scenarios outside the `PASS` / `FAIL` / `SKIP` enum. A `SKIP` must name a specific sandbox or runtime blocker.

### Prompt Quality

Prompts should be:

- Realistic, not bare command paraphrases.
- Deterministic enough to produce stable evidence.
- Explicit about what to capture.
- Explicit about the user-facing verdict or outcome.
- Compact enough for a 9-column table cell.

Weak prompt:

```text
Test search
```

Acceptable prompt:

```text
Use advisor_recommend for the flaky routing-threshold issue, capture the returned recommendation set, and return a concise pass/fail verdict with the main reason.
```

### Natural-Human vs RCAF Voice

The canonical `Prompt:` field defaults to natural-human voice. Match how a real user would phrase the request to an AI in conversation.

Use the RCAF wrapper only when the actor is an AI orchestrator:

```text
As a {ROLE}, {ACTION} against {TARGET}. Verify {EXPECTED_OUTCOME}. Return {OUTPUT_FORMAT}.
```

Use natural-human voice when:

- A human asks the AI directly in conversation.
- The scenario is a code review, commit/git workflow, bug fix, code question, or preference question.
- The scenario could plausibly originate from a Slack DM or terminal prompt.

Use RCAF when:

- The actor is an AI orchestrator delegating to another tool, AI, agent, CLI, or runtime.
- Role context determines safety refusal behavior.
- The validation contract treats the orchestrator as the operator.

When in doubt, prefer natural-human voice. The `Real user request:` field is always natural-human and serves as the voice reference baseline.

### Prompt Synchronization Gate

These fields must agree:

- Structured prompt field in `SCENARIO CONTRACT`.
- `Exact Prompt` column in the execution table.
- Any root summary prompt text.
- A conversation chain's Turn 1 input, when the scenario has one, because that is the copy a runner sends.

Do not ship unsynchronized prompt fields. `scripts/validate-playbook-package.cjs` reports each copy that disagrees with the contract as a `PROMPT_UNSYNCED` warning. Quotes, backticks, emphasis and spacing are presentation and are ignored. Any difference in the words is reported.

---

## 7. VALIDATION AND RELEASE GATES

### Automated Checks

Run shared validation on the root playbook before delivery from the repo root (replace `<SKILL_PATH>` with the target skill directory, e.g. `.skilled/skills/system-spec-kit`):

```bash
# New-content naming guard. The staging root must contain only the newly authored
# canonical manual-testing-playbook package, never an ancestor with shipped legacy roots.
python3 .skilled/skills/sk-doc/shared/scripts/check_no_hyphenated_catalog_content.py <new-content-staging-root>

python3 .skilled/skills/sk-doc/shared/scripts/validate_document.py <SKILL_PATH>/manual-testing-playbook/manual-testing-playbook.md --type reference
python3 .skilled/skills/sk-doc/shared/scripts/extract_structure.py <SKILL_PATH>/manual-testing-playbook/manual-testing-playbook.md
```

The staging scope is mandatory until shipped underscore roots are migrated. Do not run this guard against `.skilled/skills` or another ancestor containing legacy `feature_catalog/` or `manual_testing_playbook/` trees.

Also check:

- Broken local links, by grep/script or the repository markdown-link guard.
- Feature ID counts between the root index and per-feature files.
- Automated-test links, when referenced.
- Feature-catalog links, when referenced.

### Automated Checks

Run the operator-contract validator from the repository root:

```bash
node .skilled/skills/sk-doc/sk-create-manual-testing-playbook/scripts/validate-playbook-package.cjs \
  --package .skilled/skills/<skill-id>/manual-testing-playbook
```

**A fleet run scans every root, including packet-owned ones.** `discoverPackages` descends into
`<skill>/<packet>/manual-testing-playbook` as well as `<skill>/manual-testing-playbook`, because a
packet keeps its playbook as a SIBLING of the hub's, not inside it. Enumerating only the first
level once left most of this corpus unscanned: a fleet run reported success while roots it had
never opened carried hundreds of violations. If you change discovery, re-check the scanned count
against `find .skilled/skills -type d -name manual-testing-playbook | wc -l` — a root that is not
scanned cannot fail, so absence looks exactly like success.

**Make a clean package permanent.** When a package reaches zero violations, remove it from
`warnPackages` in `playbook-corpus-manifest.json` in the same change, and add each of its roots to
`playbook-failclosed-allowlist.txt`. Graduating is what converts a one-time cleanup into a standing
guarantee: a warn-listed package reports its violations and ships anyway. The CI workflow
`.github/workflows/playbook-operator-contract.yml` runs the fleet validator and separately asserts
every allowlisted root is still discovered, so a root cannot quietly leave enforcement by being
renamed or moved. Fix a failing package rather than removing its line.

The command validates the operator-scenario contract, not the routing-gold contract. It walks every non-excluded
scenario tree and checks, per feature: five-section ordering; `title`, `description`, and four-part `version`
frontmatter; Feature ID; operator/orchestrator prompt; exact command sequence; expected signals; evidence;
pass/fail criteria; failure triage; root-playbook link; allowed verdicts; filename/category shape; unique IDs;
root-index bijection; local links; evergreen truth; and placeholder exclusion. Conditional checks cover a realistic
user request when user intent is explicitly being clarified, an exact prompt when a scenario table is present,
and a feature-catalog link when catalog applicability is declared.

The validator derives scenario and category counts at run time. A root's hand-typed census is reported as a warning,
including a mismatch, so documentation repair remains separate from enforcement. Existing measured packages are
listed in the validator's staged warning set for the first fleet run; clean packages and new playbooks fail closed.
Promotion removes a package from that warning set only after a clean run.

The validator also checks the root playbook for the result-persistence completion marker and the complete `PASS` / `FAIL` / `SKIP`
vocabulary as advisory warnings. Missing either item never creates a new fail-closed violation for an existing package.
Prompt synchronization is advisory in the same way: a prompt copy that disagrees with the scenario contract is a
`PROMPT_UNSYNCED` warning, never a violation.

Exit codes are direct: `0` means conforming or staged warning, `1` means a fail-closed contract violation, and `2`
means a usage or boundary error. Strict mode is on by default; `--no-strict` is local triage only and must not be
used by CI.

The existing `validate-playbook-topology.cjs` remains the routing-gold consumer and is intentionally unchanged.
It continues reading its current playbook paths without consulting the additive corpus manifest.

---

## 8. RULES

### ✅ ALWAYS

1. Use `manual-testing-playbook.md` as the root file name.
2. Put per-feature files in root-level category folders named with the bare descriptive slug.
3. Keep one canonical per-feature file for each feature ID.
4. Use exact prompts, exact command sequences, observable expected signals, captured evidence, and binary pass/fail criteria.
5. Keep shared review, orchestration, and release-readiness rules in the root playbook.
6. Link scenarios back to the matching feature catalog entry when a catalog exists.
7. Explicitly document exceptions when no matching catalog entry exists.
8. Mark destructive scenarios and include safe recovery expectations.
9. Validate the root playbook and manually review per-feature files before delivery.

### ⛔ NEVER

1. Create a `snippets/` subtree for canonical per-feature files.
2. Create separate canonical `review_protocol.md` or `subagent_utilization_ledger.md` files.
3. Duplicate full execution truth in the root summary.
4. Renumber published feature IDs just to remove gaps.
5. Use packet-history or spec-phase references as scenario identity.
6. Ship unsynchronized prompt fields.
7. Add packet-local `graph-metadata.json`.
8. Add numeric prefixes to category folder names or per-feature filenames; the root playbook index owns display order and the per-feature `stage:` field owns benchmark tier.
9. Never use underscores in category folder or per-feature filename path segments.

### ⚠️ ESCALATE IF

1. The target feature set, category boundary, or package owner is unclear.
2. A scenario would require destructive actions without a safe recovery path.
3. The user expects external credentials, live production access, or privileged tool execution.
4. Feature catalog links are required but no stable catalog exists.
5. Validation fails after structural fixes.
6. The requested package shape conflicts with existing published playbooks.

---

## 9. SUCCESS CRITERIA

A `create-manual-testing-playbook` run is done when:

- The package uses the canonical shape: root `manual-testing-playbook.md` plus kebab-case category folders of per-feature files, with no `snippets/` subtree and no separate `review_protocol.md` or `subagent_utilization_ledger.md`.
- Every feature ID maps to exactly one per-feature file, and no packet-local `graph-metadata.json` was added.
- Every scenario is deterministic: exact prompt, exact command sequence, observable expected signals, captured evidence, and binary `PASS`/`FAIL`/`SKIP` verdicts.
- Prompt fields are synchronized across the scenario contract, the execution table, and any root summary.
- Destructive scenarios are marked and isolated with safe recovery expectations.
- The root playbook passes shared `validate_document.py`, per-feature files are manually spot-checked, and any remaining manual scope is documented honestly.

---

## 10. RESOURCES FOR DEEP DETAIL & REFERENCES

The core executable workflow lives in this `SKILL.md`. Use these only for overflow detail, exhaustive examples, or template text:

- `references/README.md` - reference map routing to the overflow detail below.
- `references/prompt-voice.md` - natural-human vs RCAF decision table and voice guidelines.
- `references/common-pitfalls.md` - recurring package defects and correct fixes.
- `references/examples.md` - shipped reference playbooks and scaffold templates.
- `assets/manual-testing-playbook-template.md` - root playbook scaffold.
- `assets/manual-testing-playbook-snippet-template.md` - per-feature file scaffold.
- `../shared/references/core-standards.md` - shared markdown structure rules.
- `../shared/references/validation.md` - shared validation and DQI workflow.
- `../sk-create-frontmatter/references/frontmatter-versioning.md` - 4-part version expectations.
- `../shared/references/evergreen-packet-id-rule.md` - evergreen current-state wording.

The source assets keep the filenames `manual-testing-playbook-template.md` and `manual-testing-playbook-snippet-template.md` until their separate source-file migration. Those filenames are authoring inputs, not emitted package names.