Systematic Debugger

---
name: systematic-debugger
description: "MANUAL-ONLY; never auto-invoke. Drive an unknown-cause bug to its root cause through the fixed sequence reproduce → reduce → isolate → fix one thing → verify → prevent, with evidence at every step and no shotgun fixes. Use when something fails and the cause is not yet known — an intermittent failure, a regression since a release, wrong output with no obvious source, an error only in one environment. Ranks hypotheses and proves or disproves them instead of guessing; changes exactly one thing per fix attempt; never calls a symptom a root cause without evidence. Do NOT use when the cause is already isolated (implement the fix with tdd-engineer), for whole-repo health assessment (full-codebase-auditor), or to classify a CI run's failures by cause (ci-failure-classifier)."
disable-model-invocation: true
---

# Systematic Debugger

## Purpose

Replace guess-and-check debugging with an evidence chain: a reliable
reproduction, a reduced minimal case, an isolated cause proven by prediction
("if this is the cause, then X will happen when I do Y"), a single-variable
fix, verification against the original reproduction, and a prevention step so
the class of bug does not return. The output is a root cause the reader can
check — not "it works now."

## Use When

- Use when: a failure's cause is unknown — wrong output, crash, hang,
  intermittent test, environment-specific error.
- Use when: something regressed after a release, upgrade, or config change
  and nobody knows which change did it.
- Use when: a previous "fix" didn't hold — symptom returned, cause was never
  actually found.
- Do NOT use when: the root cause is already isolated — go implement the fix
  (`tdd-engineer` turns the reproduction into the regression test).
- Do NOT use when: nothing is failing and the ask is overall code health —
  that is `full-codebase-auditor` or `principal-code-analyst`.
- Do NOT use when: the "bug" is a disagreement between docs and code about
  intended behavior — `source-of-truth-reconciler` decides what's intended.
- Do NOT use when: the question is which jobs in a CI run failed and why, by
  cause class — `ci-failure-classifier` classifies first; it hands a
  `PRODUCT-BUG` with an unknown cause back here.

## Inputs to Inspect

1. The exact symptom: verbatim error text, wrong-vs-expected output, when it
   started, who/what it affects (environment, tenant, user, data shape).
2. Recent changes: commits, merges, dependency bumps, config/infra changes,
   migrations in the suspicion window (`git log`, lockfile diff, deploy log).
3. Evidence streams: logs, stack traces, metrics, traces around failing
   instances — collected before forming a favorite theory.
4. The failing path's code and tests — read as-is, not as remembered.
5. Environment deltas when it fails only somewhere: versions, env vars,
   data volume, permissions, concurrency.

## Workflow

Explicit human invocation selects this execution-capable skill; it does not
expand the allowed target or activate Task-Authorized Local Implementation
(TALI). TALI is a separate route requiring its own classification and
activation under the [skill-generation standard](../../../docs/skill-generation-standard.md#5-least-privilege--side-effects).
Use applicable existing user grants without repeated consent. Before a
state-changing command, confirm its actual files, environment and effects fit
those grants; if authority is absent, propose the operation and obtain it
before proceeding.

1. **Reproduce.** Build the smallest command/test/script that shows the
   failure on demand; record the exact invocation and failure output. For
   intermittent failures, find the frequency amplifier (loop it, add load,
   fix the seed) until reproduction is statistically reliable. If
   reproduction is impossible, say why (data gone, prod-only permissions) and
   continue on logs/traces with confidence explicitly downgraded.
2. **Reduce.** Shrink input, config, and code path until every remaining
   element is necessary for the failure. The minimal case usually names the
   suspect on its own.
3. **Isolate.** List candidate causes and rank by evidence fit and prior
   probability. Test the top hypothesis with a prediction it must satisfy
   (bisect the change window, toggle the suspect flag, pin the suspect
   dependency, instrument the suspect boundary). A hypothesis survives only
   if its prediction comes true; record disproofs — they are progress.
4. **Fix one thing.** Change exactly the isolated cause. No drive-by cleanup,
   no "also hardened X while there" — a multi-variable fix destroys the
   evidence chain and hides which change worked.
5. **Verify.** Re-run the original reproduction (not just the reduced case):
   red before the fix, green after, wider suite still green. For intermittent
   bugs, re-run enough iterations to beat the original failure rate.
6. **Prevent.** Add the regression test (the reproduction, kept), and where
   warranted an assertion, a lint rule, monitoring, or an alert. State the
   root cause in one sentence a reviewer can falsify.

## Output Format

```
DEBUGGING REPORT — <symptom>
Symptom: <verbatim error / wrong output, scope, since when>
Reproduction: <exact command + failure output> (reliability: n/n runs)
Reduction: <minimal failing case>
Hypotheses: <H1..Hn — ranked; each: prediction tested → confirmed/disproved,
            with evidence>
Root cause: <one falsifiable sentence, with file:line>
Fix: <the single change made>
Verification: <original repro red→green, suite result, iteration count if
              intermittent>
Prevention: <regression test added; monitoring/guard if any>
Unknowns kept: <what was not proven, and its risk>
```

## Validation Checklist

- [ ] Reproduction recorded with the exact command and real output (or the
      impossibility explained and confidence downgraded).
- [ ] Each hypothesis tested via a stated prediction; disproofs recorded.
- [ ] Root cause is falsifiable and cites code, not vibes ("race in X when Y"
      not "timing issue").
- [ ] Exactly one variable changed by the fix.
- [ ] Verification re-ran the ORIGINAL reproduction, plus the wider suite.
- [ ] Regression test (or explicit reason none is possible) added.
- [ ] No unrelated edits shipped with the fix.

## Gotchas

- The bug is rarely where the error surfaces; stack traces show the victim,
  not always the culprit — follow data flow upstream.
- "It stopped failing" after a change you didn't isolate is the most seductive
  false green — intermittent bugs pass runs all the time. Beat the base rate.
- Bisecting with a dirty working tree or flaky suite produces lying bisects;
  stabilize the test before trusting `git bisect`.
- Fixing the reduced case but never re-running the original reproduction
  ships a fix for the wrong layer.
- Two bugs can share one symptom; verification failing after a proven fix may
  mean bug #2, not disproof of bug #1 — re-reduce.
- Debugging output (prints, verbose flags, temp instrumentation) must be
  stripped before the fix ships; list it and remove it.

## Stop Conditions

- The skill was selected automatically rather than explicitly invoked by the
  human: do not execute it. An agent may recommend the named manual skill.
- A write, Git/network operation, database command or test side effect exceeds
  the applicable human grant: stop that operation and obtain the missing scope.
  Existing authorized operations do not need the same permission again.

- Reproduction or investigation would require production data access,
  destructive operations, or config changes in shared environments →
  `human-approval-boundary` first.
- The root cause lands in security/tenant-isolation behavior → stop and
  surface before fixing; the fix may need review as a security change.
- The isolated cause is a third-party defect → document the evidence, pin or
  work around with approval, report upstream; do not fork silently.
- Budget exhausted with hypotheses disproven and no reproduction → report the
  evidence map and unknowns honestly rather than shipping a speculative fix.

## Supporting Files

- [references/isolation-tactics.md](references/isolation-tactics.md) —
  bisection recipes, intermittent-failure amplifiers, environment-delta
  tables, and instrumentation patterns.
- `evals/evals.json` — trigger + behavior cases.
- `evals/trigger-evals.json` — discrimination against `tdd-engineer` and
  `docs-first-implementer` (implementation cluster).