Systematic Debugging

---
name: systematic-debugging
description: Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes. Especially when under time pressure, when a "quick fix" seems obvious, or when previous fix attempts have failed.
---

# Systematic Debugging

## Overview

Random fixes waste time and create new bugs. Quick patches mask underlying issues.

**Core principle:** ALWAYS find root cause before attempting fixes. Symptom fixes are failure.

**Violating the letter of this process is violating the spirit of debugging.**

## The Iron Law

```
NO FIXES WITHOUT ROOT CAUSE INVESTIGATION FIRST
```

If you haven't completed Phase 1, you cannot propose fixes.

## When to Use

Any technical issue: test failures, production bugs, unexpected behavior, performance problems, build failures, integration issues.

**Use this ESPECIALLY when:**
- Under time pressure (emergencies make guessing tempting)
- "Just one quick fix" seems obvious
- You've already tried multiple fixes
- Previous fix didn't work
- You don't fully understand the issue

**Don't skip when:**
- Issue seems simple (simple bugs have root causes too)
- You're in a hurry (rushing guarantees rework)
- Someone wants it fixed NOW (systematic is faster than thrashing)

## The Four Phases

Complete each phase before proceeding to the next.

### Phase 1: Root Cause Investigation

**BEFORE attempting ANY fix:**

1. **Read Error Messages Carefully**
   - Don't skip past errors or warnings — they often contain the exact solution
   - Read stack traces completely; note line numbers, file paths, error codes

2. **Reproduce Consistently**
   - Can you trigger it reliably? What are the exact steps? Every time?
   - If not reproducible: gather more data, don't guess

3. **Check Recent Changes**
   - Git diff, recent commits, new dependencies, config changes, environmental differences

4. **Gather Evidence in Multi-Component Systems**

   When the system has multiple components (CI -> build -> signing, API -> service -> database), add diagnostic instrumentation BEFORE proposing fixes:

   ```
   For EACH component boundary:
     - Log what data enters the component
     - Log what data exits the component
     - Verify environment/config propagation
     - Check state at each layer

   Run once to gather evidence showing WHERE it breaks,
   THEN analyze evidence to identify the failing component,
   THEN investigate that specific component.
   ```

   Example: log the secret/env var at the workflow layer, the build-script layer, and the signing layer — the logs reveal which handoff loses it.

5. **Trace Data Flow Backward (root cause tracing)**

   When the error is deep in the call stack, never fix where the error appears. Trace backward:
   - Where does the bad value originate?
   - What called this with the bad value? What called that?
   - Keep tracing up until you find the source, then fix at the source
   - If you can't trace manually, instrument: log the suspect value plus `cwd`/env plus a captured stack trace (`new Error().stack`) immediately BEFORE the dangerous operation, then run and analyze which caller passed the bad value

### Phase 2: Pattern Analysis

1. **Find Working Examples** — locate similar working code in the same codebase
2. **Compare Against References** — if implementing a pattern, read the reference implementation COMPLETELY, not skimming
3. **Identify Differences** — list every difference between working and broken, however small; don't assume "that can't matter"
4. **Understand Dependencies** — what other components, settings, config, environment does this need? What assumptions does it make?

### Phase 3: Hypothesis and Testing

1. **Form Single Hypothesis** — state clearly: "I think X is the root cause because Y." Write it down. Be specific.
2. **Test Minimally** — smallest possible change to test the hypothesis, one variable at a time
3. **Verify Before Continuing** — worked? Go to Phase 4. Didn't? Form a NEW hypothesis. DON'T stack more fixes on top.
4. **When You Don't Know** — say "I don't understand X". Don't pretend. Ask for help or research more.

### Phase 4: Implementation

1. **Create Failing Test Case** — simplest possible reproduction, automated if possible (a one-off script if there is no framework). MUST exist before fixing. Follow test-driven development for this.
2. **Implement Single Fix** — address the identified root cause. ONE change at a time. No "while I'm here" improvements, no bundled refactoring.
3. **Verify Fix** — test passes? No other tests broken? Issue actually resolved?
4. **If Fix Doesn't Work** — STOP. Count your attempts. If fewer than 3: return to Phase 1 and re-analyze with the new information. If 3 or more: STOP and question the architecture (step 5). Don't attempt fix #4 without that discussion.
5. **If 3+ Fixes Failed: Question Architecture**

   Signs of an architectural problem: each fix reveals new shared state/coupling elsewhere; fixes require massive refactoring; each fix creates new symptoms in a different place.

   STOP and question fundamentals: Is this pattern sound? Are we sticking with it through inertia? Should we refactor instead of patching symptoms? **Discuss with the user before attempting more fixes.** This is not a failed hypothesis — this is a wrong architecture.

   Before that discussion, run the getting-unstuck skill on the impasse — its generation moves (change layer, go up a level, reduce scope, invert) often surface the alternative worth proposing, and its verdict format gives the user evidence instead of "I'm stuck".

## Red Flags - STOP and Follow Process

If you catch yourself thinking:
- "Quick fix for now, investigate later"
- "Just try changing X and see if it works"
- "Add multiple changes, run tests"
- "Skip the test, I'll manually verify"
- "It's probably X, let me fix that"
- "I don't fully understand but this might work"
- "Pattern says X but I'll adapt it differently"
- "Here are the main problems: [lists fixes without investigation]"
- Proposing solutions before tracing data flow
- **"One more fix attempt" (when already tried 2+)**
- **Each fix reveals new problem in different place**

**ALL of these mean: STOP. Return to Phase 1.**

Signals from the user that you're doing it wrong: "Is that not happening?" (you assumed without verifying), "Stop guessing" (you're proposing fixes without understanding), visible frustration that you're stuck. When you see these: STOP. Return to Phase 1.

## Common Rationalizations

| Excuse | Reality |
|--------|---------|
| "Issue is simple, don't need process" | Simple issues have root causes too. Process is fast for simple bugs. |
| "Emergency, no time for process" | Systematic debugging is FASTER than guess-and-check thrashing. |
| "Just try this first, then investigate" | First fix sets the pattern. Do it right from the start. |
| "I'll write test after confirming fix works" | Untested fixes don't stick. Test first proves it. |
| "Multiple fixes at once saves time" | Can't isolate what worked. Causes new bugs. |
| "Reference too long, I'll adapt the pattern" | Partial understanding guarantees bugs. Read it completely. |
| "I see the problem, let me fix it" | Seeing symptoms != understanding root cause. |
| "One more fix attempt" (after 2+ failures) | 3+ failures = architectural problem. Question pattern, don't fix again. |

## Quick Reference

| Phase | Key Activities | Success Criteria |
|-------|---------------|------------------|
| **1. Root Cause** | Read errors, reproduce, check changes, gather evidence | Understand WHAT and WHY |
| **2. Pattern** | Find working examples, compare | Identify differences |
| **3. Hypothesis** | Form theory, test minimally | Confirmed or new hypothesis |
| **4. Implementation** | Create test, fix, verify | Bug resolved, tests pass |

## Supporting Techniques

**Defense in depth (after finding root cause):** don't stop at one validation point — make the bug structurally impossible by validating at every layer the bad data passes through:
1. Entry point validation — reject invalid input at the API boundary
2. Business logic validation — ensure data makes sense for this operation
3. Environment guards — refuse dangerous operations in specific contexts (e.g. in tests, refuse destructive ops outside temp dirs)
4. Debug instrumentation — log context (value, cwd, stack) before the dangerous operation

Different layers catch different cases: other code paths bypass entry checks, mocks bypass business checks.

**Condition-based waiting (for flaky, timing-dependent tests):** never guess at timing with arbitrary `sleep`/`setTimeout` delays. Wait for the actual condition:

```typescript
// BAD: guessing at timing
await new Promise(r => setTimeout(r, 50));
// GOOD: waiting for the condition, polling ~10ms with a timeout and clear error
await waitFor(() => getResult() !== undefined, 'result available');
```

Arbitrary timeouts are only correct when testing actual timing behavior (debounce/throttle) — and then based on known intervals, with a comment explaining why, after first waiting for the triggering condition.

## When Process Reveals "No Root Cause"

If systematic investigation reveals the issue is truly environmental, timing-dependent, or external:

1. You've completed the process
2. Document what you investigated
3. Implement appropriate handling (retry, timeout, error message)
4. Add monitoring/logging for future investigation

**But:** most "no root cause" cases are incomplete investigation. And if the conclusion is drifting toward "this can't be done at all" (missing capability, unsupported feature, blocked path — not a bug), switch to the getting-unstuck skill before reporting it.