Dev Debugging

---
name: dev-debugging
description: "Systematic debugging methodology for all orchestrated sub-agents. 4-phase root cause analysis: investigate → analyze → hypothesize → implement. Injected when encountering errors or during debugging phase."
---

# Dev-Debugging — Systematic Root Cause Analysis

This skill is the **thinking process** for fixing bugs. It enforces a structured
4-phase methodology for every technical issue — test failures, runtime errors,
build failures, performance regressions, integration bugs.

**Boundary**: This skill covers how to reason about bugs. For test harness,
reproduction frameworks, and verification tooling, see `dev-testing`. For
domain-specific context (API errors, hydration issues, query performance),
consult `dev-backend` or `dev-frontend`.

```
dev-debugging = root cause methodology (the thinking)
dev-testing   = test harness for reproducing/verifying (the tooling)
dev §2        = summary pointer to this skill (the overview)
```

---

## Core Principle

Complete root cause investigation before proposing any fix.
If Phase 1 is not done, keep investigating.

---

## When to Activate

- Test failures, runtime errors, build failures, performance regressions
- Integration issues (API, database, third-party), CI pipeline failures
- **Especially**: when under time pressure or when "just one quick fix" seems obvious — that's when methodology matters most

---

## The Four Phases

### Phase 1: Root Cause Investigation

**Complete these before attempting any fix:**

1. **Read the full error** — stack trace, line numbers, error code, surrounding
   context. Do not skim. The answer is often in the error message itself.

2. **Reproduce consistently** — exact steps to trigger the bug. If intermittent,
   document frequency, conditions, and environment state. A bug you cannot
   reproduce is a bug you cannot verify as fixed.

3. **Check recent changes** — run `git log --oneline -10` and `git diff`. Check
   new dependencies, config changes, environment variables. Bugs correlate with
   recent changes most of the time.

4. **Trace data flow** — where does the bad value originate? Trace backward from
   the failure point through the call stack until you find the source. Fix at the
   source, not the symptom.

5. **Instrument component boundaries** — for multi-layer systems (API → service →
   database, CI → build → deploy), log input/output at each boundary BEFORE
   proposing fixes.

```
For EACH component boundary:
  - Log what data enters the component
  - Log what data exits the component
  - Verify environment/config propagation
Run once → analyze evidence → identify failing layer → investigate THAT layer
```

Work through these steps; skip only if clearly irrelevant to the problem at hand.

### Phase 2: Pattern Analysis

1. **Find working examples** — similar working code in the same codebase. If it
   worked before, use `git bisect` to find the breaking commit (see
   `references/tool-guides.md`).

2. **Compare systematically** — list every difference between working and broken
   code. No matter how small. Resist assuming "that can't matter."

3. **Read reference docs completely** — official documentation for the library,
   API, or framework involved. Don't skim — read the full relevant section.

4. **Check known issues** — GitHub Issues, changelogs, migration guides. Someone
   may have hit the same bug. Search with the exact error message.

### Phase 3: Hypothesis and Testing

1. **State hypothesis explicitly** — "X is the root cause because evidence Y
   shows Z." Write it down. If you can't articulate it clearly, you don't
   understand it yet.

2. **Design a test to disprove** — falsification is stronger than confirmation.
   What would you expect to see if your hypothesis is wrong?

3. **Test one variable** — smallest possible change, one variable at a time.
   Never fix multiple things at once.

4. **If it fails** → form a new hypothesis. Revert the failed change and
   start from clean state. Stacking fixes obscures the root cause.

5. **Admit ignorance** — "I don't understand X" is a valid finding. Research
   further rather than guessing. Record the open question explicitly.

### Phase 4: Implementation

1. **Write a failing test first** — the test reproduces the bug. It should fail
   before the fix. Use `dev-testing` for TDD patterns and test harness setup.

2. **Make the minimal fix** — address the root cause, not symptoms. One logical
   change only.

3. **Verify**: the test passes, no regressions (run the full test suite:
   `npm test` / `pytest` / equivalent).

4. **Check for similar patterns** — does the same bug class exist elsewhere in
   the codebase? Search for it. Fix all instances, not just the one you found.

5. **Document** — commit message explains root cause AND fix. Not "fixed bug"
   but "fix: race condition in session middleware caused by missing await on
   Redis write."

---

## Red Flags — Return to Phase 1

If you catch yourself doing any of these, pause — root cause investigation
was likely skipped.

| Red Flag | Why It Fails |
|----------|-------------|
| "Quick fix for now, investigate later" | First fix sets the pattern. Tech debt compounds. You won't investigate later. |
| "Just try changing X and see" | Guessing guarantees rework. You'll be back here within the hour. |
| "Add multiple changes, run tests" | Can't isolate cause if multiple variables changed. Revert, change ONE thing. |
| "It's probably X, let me fix that" | "Probably" without evidence = Phase 1 not done. Go back and trace it. |
| "I don't fully understand but this might work" | Seeing symptoms ≠ understanding root cause. Your "fix" hides the real bug. |
| "One more fix attempt" (after repeated failures) | After repeated failures, pause and reassess architecture/assumptions. See escalation below. |
| "It works on my machine" | Reproduce in the SAME environment as the failure. Local success proves nothing. |
| "Let me add a try/catch around it" | Suppressing errors is not fixing them. Find WHY it throws. |

**Repeated Failure Rule**: After repeated failed fix attempts, pause entirely.
Each fix revealing a new problem in a different place is a sign of
**architectural issues**, not simple bugs. Discuss with the user before
attempting more fixes.

---

## Slop Debugging Patterns

Slop debugging is spray-and-pray: guess, patch, pray, repeat.

| Instead of… | Use… |
|-------------|------|
| Proposing fixes before investigation | Complete Phase 1 checklist first |
| "Might be X" without evidence | "Evidence shows X because [log/trace/diff]" |
| Multiple simultaneous changes | One change at a time, revert between attempts |
| Skimming stack traces | Read every line of stack trace, note line numbers |
| Silent `catch` blocks that suppress errors | Log with context (`[module] error.message`), re-throw or handle |
| Modifying failing tests to pass | Fix the code, not the test — a failing test is evidence |
| Claiming "fixed" without running verification | Run full test suite, show green output, verify the original symptom |
| Copy-pasting a fix without understanding | Understand why the fix works, then adapt to your codebase |
| Wrapping `try/catch` around the crash site | Fix at the source — trace upstream to where the bad data originates |
| Guessing at types, nulls, or undefined values | Add diagnostic logging, inspect actual runtime values |
| "It works now" after changing something unrelated | Correlation ≠ causation — revert the change and test again |

---

## Concrete Debugging Scenarios

### Scenario A: API Returns 500

Root cause pattern: Missing input validation lets undefined values propagate into business logic. Instrument controller/service/repository boundaries to find where the bad value enters. Compare with a working endpoint that validates input with a schema. Fix: add schema validation at the entry point, write a test that sends invalid input and expects 400.

### Scenario B: React Hydration Mismatch

Root cause pattern: Server renders a value (e.g., date, locale string) that differs from client-side rendering due to environment differences (UTC vs. local timezone). Compare with components that defer environment-dependent rendering to useEffect. Fix: move environment-dependent formatting into a client component.

### Scenario C: N+1 Query Performance

Root cause pattern: List endpoint lazy-loads related records per item (1 query + N queries). Enable query logging to count queries, then compare with an endpoint that uses eager loading. Fix: add include/joinedload, write a test asserting bounded query count.

### Scenario D: Flaky Test (Intermittent Failure)

Root cause pattern: Test passes in isolation but fails in suite due to shared mutable state (database rows, global variables, uncleared mocks). Compare with stable tests that use transaction rollback in beforeEach/afterEach. Fix: add proper test isolation, then search for other tests missing cleanup.

---

## When to Escalate vs When to Keep Digging

### Keep Digging When:

- You have untested hypotheses from Phase 2
- You haven't read the full error message or stack trace
- You haven't checked recent changes (`git log`, `git diff`)
- You haven't found working comparison code yet
- The bug is in YOUR code (not a third-party library)
- You still have untested approaches to try

### Escalate When:

- **Repeated fix attempts failed** — likely architectural; needs human judgment
- **Undocumented library behavior** — file an issue upstream, work around it
- **Environment-specific** — requires access you don't have (prod DB, cloud IAM)
- **Security-sensitive** — don't debug auth/crypto/payment alone; flag for human review
- **Multi-team dependency** — bug is in another team's service or API contract
- **Stalled**: if investigation stalls, reassess approach

### How to Escalate Well

Don't just say "I'm stuck." Provide: **symptom** (exact error), **reproduction
steps**, **evidence gathered** (logs, traces, bisect results), **hypotheses
tested** (what you tried, why it failed), **remaining hypotheses** (untested),
and a **recommendation** for next steps.

---

## Post-Mortem Discipline

After resolving any bug that:
- Was user/customer-impacting
- Took >1 hour to diagnose
- Involved repeated failed fix attempts
- Revealed a systemic issue (same bug class exists elsewhere)

Fill out `references/postmortem-template.md` and include it in the PR or commit.
The goal is **learning, not blame**. Every postmortem must produce at least one
action item that prevents the same class of bug from recurring.

---

## Modular References

| File | When to Read | What It Covers |
|------|-------------|----------------|
| `references/tool-guides.md` | When you need stack-specific debugger commands | Node.js inspector, Python pdb/debugpy, Chrome DevTools, git bisect, database EXPLAIN |
| `references/postmortem-template.md` | After resolving a significant incident | Blameless postmortem template with filled example |

---

## Integration with Other Skills

| Skill | Relationship |
|-------|-------------|
| `dev` §2 | Summary of this methodology. This skill is the full version. |
| `dev-testing` | Phase 4 "write failing test first" → use `dev-testing` for test patterns and harness. `dev-testing` provides the tooling; this skill provides the thinking. |
| `dev-backend` | Server-side debugging context: API errors, database issues, middleware chains. |
| `dev-frontend` | Client-side debugging context: hydration, rendering, DevTools, layout shifts. |
| `dev-code-reviewer` | Code review catches bugs before they ship — prevention beats debugging. |

---

## Compact Summary

When context is limited, preserve: (1) Core principle — no fixes without root cause,
(2) 4 Phases — investigate → analyze → hypothesize → implement,
(3) Repeated Failure Rule — after repeated failures, reassess, (4) one variable at a time,
(5) evidence over intuition, (6) failing test first.