Diagnose

---
name: diagnose
description: Disciplined diagnosis loop for hard bugs and performance regressions. Reproduce → minimise → hypothesise → instrument → fix → regression-test. Use when user says "diagnose this" / "debug this", reports a bug, says something is broken/throwing/failing, or describes a performance regression.
tags: [domain:debugging, intent:debug, intent:fix, intent:diagnose, type:skill]
---

# Diagnose

A discipline for hard bugs. Skip phases only when explicitly justified.

When exploring the codebase, use the project's domain glossary to get a clear mental model of the relevant modules, and check ADRs in the area you're touching.

## Phase 1 — Build a feedback loop

**This is the skill.** Everything else is mechanical. If you have a fast, deterministic, agent-runnable pass/fail signal for the bug, you will find the cause — bisection, hypothesis-testing, and instrumentation all just consume that signal. If you don't have one, no amount of staring at code will save you.

Spend disproportionate effort here. **Be aggressive. Be creative. Refuse to give up.**

### Ways to construct one — try them in roughly this order

1. **Failing test** at whatever seam reaches the bug — unit, integration, e2e.
2. **Curl / HTTP script** against a running dev server.
3. **CLI invocation** with a fixture input, diffing stdout against a known-good snapshot.
4. **Headless browser script** (Playwright / Puppeteer) — drives the UI, asserts on DOM/console/network.
5. **Replay a captured trace.** Save a real network request / payload / event log to disk; replay it through the code path in isolation.
6. **Throwaway harness.** Spin up a minimal subset of the system (one service, mocked deps) that exercises the bug code path with a single function call.
7. **Property / fuzz loop.** If the bug is "sometimes wrong output", run 1000 random inputs and look for the failure mode.
8. **Bisection harness.** If the bug appeared between two known states (commit, dataset, version), automate "boot at state X, check, repeat" so you can `git bisect run` it.
9. **Differential loop.** Run the same input through old-version vs new-version (or two configs) and diff outputs.
10. **HITL bash script.** Last resort. If a human must click, drive _them_ with `scripts/hitl-loop.template.sh` so the loop is still structured. Captured output feeds back to you.

Build the right feedback loop, and the bug is 90% fixed.

### Iterate on the loop itself

Treat the loop as a product. Once you have _a_ loop, ask:

- Can I make it faster? (Cache setup, skip unrelated init, narrow the test scope.)
- Can I make the signal sharper? (Assert on the specific symptom, not "didn't crash".)
- Can I make it more deterministic? (Pin time, seed RNG, isolate filesystem, freeze network.)

A 30-second flaky loop is barely better than no loop. A 2-second deterministic loop is a debugging superpower.

### Non-deterministic bugs

The goal is not a clean repro but a **higher reproduction rate**. Loop the trigger 100×, parallelise, add stress, narrow timing windows, inject sleeps. A 50%-flake bug is debuggable; 1% is not — keep raising the rate until it's debuggable.

### When you genuinely cannot build a loop

Stop and say so explicitly. List what you tried. Ask the user for: (a) access to whatever environment reproduces it, (b) a captured artifact (HAR file, log dump, core dump, screen recording with timestamps), or (c) permission to add temporary production instrumentation. Do **not** proceed to hypothesise without a loop.

Do not proceed to Phase 2 until you have a loop you believe in.

## Phase 2 — Reproduce

Run the loop. Watch the bug appear.

Confirm:

- [ ] The loop produces the failure mode the **user** described — not a different failure that happens to be nearby. Wrong bug = wrong fix.
- [ ] The failure is reproducible across multiple runs (or, for non-deterministic bugs, reproducible at a high enough rate to debug against).
- [ ] You have captured the exact symptom (error message, wrong output, slow timing) so later phases can verify the fix actually addresses it.

Do not proceed until you reproduce the bug.

## Phase 3 — Hypothesise

Generate **3–5 ranked hypotheses** before testing any of them. Single-hypothesis generation anchors on the first plausible idea.

Each hypothesis must be **falsifiable**: state the prediction it makes.

> Format: "If <X> is the cause, then <changing Y> will make the bug disappear / <changing Z> will make it worse."

If you cannot state the prediction, the hypothesis is a vibe — discard or sharpen it.

**Check Stack Overflow for the specific error signature** (exact exception message, stack trace shape, or symptom) before finalizing the ranked list — not instead of it. If the signature matches a well-corroborated SO answer (accepted, high-voted, version-consistent with your stack), fold it in as a ranked hypothesis with a note on its source; it doesn't skip Phase 1/2 or get treated as confirmed until it survives your own feedback loop. A generic search for the bug's *symptom* rather than its exact signature is usually noise — narrow the query to what makes this failure specific.

**Show the ranked list to the user before testing.** They often have domain knowledge that re-ranks instantly ("we just deployed a change to #3"), or know hypotheses they've already ruled out. Cheap checkpoint, big time saver. Don't block on it — proceed with your ranking if the user is AFK.

**MARVIN-specific gotcha classes** — check these early when the symptom fits, before deeper instrumentation:
- *Script works manually, fails/no-ops silently under launchd or cron*: the scheduled job's PATH doesn't include the interactive shell's additions. A script that shells out to a CLI (e.g. `claude`) can resolve to nothing and have its error swallowed into the very output it's supposed to produce — confirmed 2026-07-02, where this masked a broken daily-digest cron for days because the "digest" was literally the error string.
- *A hook fires on unrelated files/edits repo-wide*: Claude Code hook matchers key on tool name (Write/Edit), not path — they do not scope by file location unless the hook script does its own path filtering. Confirmed 2026-07-02 on `rebuild-manifest.py`, which ran on every Write/Edit anywhere until path-scoped explicitly.
- *Derived state (manifest, index, cache) goes stale after a raw deletion*: hooks only fire on tool calls (Write/Edit), not on `rm` or other out-of-band filesystem changes — a manual `rm` on a tracked file leaves derived state pointing at something that no longer exists until a manual resync runs. Confirmed 2026-07-02: `manifest.json` stayed stale after deleting a skill file, with no hook to catch it.
- *A trust/config flag in `~/.claude.json` won't stick despite correctly re-accepting the dialog*: a long-running `claude` process loads that file into memory at startup and periodically flushes its own state back to disk on its own schedule — this silently clobbers any out-of-band change (manual edit, or a dialog accepted in a *different* session) with the stale in-memory snapshot. The fix is ending the long-running session so a fresh process reads the current file, not repeating the accept dialog. Confirmed 2026-07-03, root-caused via PID/timeline correlation while debugging why `claude remote-control` workspace trust wouldn't persist.
- *A deploy/install script silently drops files it should have copied*: a copy step built around a glob like `dir/*` only reaches what's inside that one subdirectory — sibling top-level files and symlinks are missed with no error. Files `cp`'d rather than symlinked to their git-tracked original also drift silently and stop feeding cross-machine sync. Confirmed 2026-07-03 setting up a second machine — `install_skills()` missed `brain-map/`, `retrospective-log.md`, and resume-tailor's deps entirely; `retrospective-log.md` on the deployed machine was also a bare `cp` outside git until fixed to a symlink.
- *A background/menu-bar app briefly flashes visible despite requesting `.accessory` activation policy*: if it's a bare compiled binary rather than a proper `.app` bundle, `NSApplication.setActivationPolicy(.accessory)` runs too late — macOS has already registered the process under the default policy by the time the runtime call executes. Fix is a real `.app` bundle with `LSUIElement` set in `Info.plist`, not a runtime workaround. Confirmed 2026-07-03 on DesktopLive.
- *A scheduled `claude -p` script produces a report that reads like a narrated failure ("I tried to write X but couldn't...") instead of the intended output*: an unrestricted `claude -p` call for a read-only reporting/generation task can still attempt a Write or Edit, get silently denied by the harness, and then narrate that denial into the very stdout it's supposed to produce as clean output. Fix: pass `--disallowedTools "Write,Edit"` (or the minimal tool set actually needed) on any non-interactive `claude -p` invocation whose job is to *produce* text, not modify files. Confirmed 2026-07-06 in both `daily_digest.py` and `research_digest.py` — same root cause hit twice independently, fixed identically in both.
- *Building anything (eval harness, client, integration) against a local model server's "OpenAI-compatible" API*: compatibility is usually partial, not full — check which specific params are actually implemented before designing around the assumption. Confirmed 2026-07-07: `mlx_lm.server` has no `echo` support at all, which `lm-eval`'s `local-completions` backend needs for log-likelihood tasks (`leaderboard_mmlu_pro`); the whole server-based harness was a dead end, discovered only after building it. Fix was dropping the HTTP layer and writing a small in-process adapter wrapping the library directly instead of assuming the server exposed everything the eval tool expected.
- *A regex parses one log/state-file entry using a greedy `.+` under `re.DOTALL`*: DOTALL makes `.` match newlines too, so the greedy `.+` keeps consuming past the intended entry boundary and bleeds into the next entry — corrupting every entry after the first, often silently. Confirmed 2026-07-14 in `check_sync_log()`: a greedy header regex under DOTALL swallowed subsequent log entries. Fix: use a negated-newline class (`[^\n]+`) for any field that should stay single-line, reserving DOTALL-sensitive greediness for the part of the pattern that actually needs to span lines.
- *A monitoring check scans a log's text for a keyword (e.g. "CONFLICT") to detect an ongoing problem*: once written, the keyword persists in the log forever, so the check keeps firing as a false positive long after the problem was resolved — text logs are a historical record, not current state. Confirmed 2026-07-14 building the SessionStart hook's conflict detector: `sync-log.md` still contained "CONFLICT" from an already-resolved merge, and a real stray stash from that old conflict was found sitting unresolved. Fix: query the live authoritative source directly (`git status --porcelain` conflict codes, `MERGE_HEAD` existence, `git stash list`) instead of grepping historical logs for keywords.
- *A prose checklist in CLAUDE.md/SKILL.md gets converted into a deterministic hook that runs the same checks*: leaving the old step-by-step prose in place alongside the new hook causes the model to re-perform checks the hook already ran and injected into context — wasted tool calls/context, and a real risk of silent disagreement between the hook's output and the model's own recheck. Fix: collapse the prose to a one-line pointer + an explicit "don't re-perform these by hand" instruction, and move full step rationale to a separate reference doc that tells editors to update it in the same commit as the script. Confirmed 2026-07-14 converting MARVIN's 11-step session-start checklist into `session_start_report.py`.
- *A themed/dark CSS background silently doesn't render, but "looks right" in one specific browser*: CSS custom properties scoped to a descendant selector (e.g. `.viz-root { --page-plane: ... }`) are invisible to an *ancestor* rule referencing them (e.g. `html, body { background: var(--page-plane) }`) — custom-property inheritance flows down the DOM tree, not up, so the ancestor rule silently resolves to nothing with no console error. Compounding factor: a browser with automatic dark-mode repainting (e.g. Comet) can incidentally produce the intended-looking dark background anyway, masking the bug entirely until viewed in a standards-compliant browser. Fix: scope custom properties at `:root` (or the highest consumer), not a scoped wrapper class. Confirmed 2026-07-17 on `~/.agents/brain-map/template.html`.
- *A JS animation library's high-level API (e.g. Motion's `animate()`) silently no-ops or looks frozen in a wallpaper-mode/embedded canvas rendering context*: high-level animation helpers are built on `requestAnimationFrame`, which macOS suspends for windows rendered at the desktop-wallpaper window level — even though a separate native render loop (e.g. a Swift `NSTimer`-driven one) keeps calling into the JS layer on schedule. The library isn't broken; rAF itself just never fires there. Fix: use the library's low-level physics primitive directly (e.g. Motion's `spring()` generator) and sample it manually from the existing native-driven loop instead of the high-level rAF-based API. Confirmed 2026-07-17 on `~/.agents/brain-map`/DesktopLive — hit twice independently (an earlier camera-rotation bug, then again for pulse/camera-nudge/`fitScale` spring animations), same root cause both times, same fix both times.
- *A "cause" animation (a highlight/pulse) and its "effect" animation (a camera move, a transition) are tuned independently and one finishes well before the other*: if the cause's visible duration is shorter than the effect's, the signal fades out mid-effect, so the effect reads as unmotivated ("why did it move — nothing happened") even though there was a real trigger. Fix: tune the cause's fade to comfortably outlast the effect's full duration, not just approximately match it. Confirmed 2026-07-17 on `~/.agents/brain-map`/DesktopLive: a ~2.5s pulse fade against a 4.2s camera-ease left the camera moving with no visible cause for the last ~1.7s.
- *Widening a shared instrumentation trigger's conditions for one consumer's benefit silently changes behavior for every other consumer keyed off the same signal*: MARVIN's generic "heartbeat" pulse (from `skill_activity.py`) was widened to fire on nearly every tool call so the pulse itself would feel more alive, but an unrelated consumer (camera-nudge logic) listening to that same signal started re-triggering on every call too, fighting its own auto-rotate instead of only reacting to named-skill activity. The signal wasn't broken, it just had more than one purpose riding on it. Fix: split consumers onto the specific condition each actually needs (named-skill activity for the camera nudge, any activity for the pulse) rather than sharing one trigger across purposes. Confirmed 2026-07-17 on `~/.agents/brain-map`/DesktopLive.
- *Real algorithmic performance work (parallelization, caching, cancellation) is applied to a compiled-language hot path but the wins are smaller than expected, or a rewrite in a faster language is being floated as the next step*: before optimizing further, confirm the build actually being measured is an optimized/release build, not the default debug build — an unoptimized compiled binary (Swift, C++, Rust) can dominate measured time by an order of magnitude more than any algorithmic change, making real optimizations look like they aren't working. Confirmed 2026-08-26 on killer-sudoku: parallel batch generation and cooperative solver cancellation were both implemented and measured against a ~10-38s baseline, and a C++ rewrite was being considered as the next lever, before discovering the entire session had been profiling the debug build — switching the run script to a release build alone dropped runtime to ~1s, dwarfing every prior optimization. Fix: at the start of any perf-diagnosis session on compiled code, confirm the invoked build/run command actually uses the optimized configuration before touching algorithms.
- *An enum backs data that gets persisted (SwiftData, `Codable`, `UserDefaults`) and relies on implicit `rawValue`s (case order)*: removing, inserting, or reordering a case silently reassigns every subsequent case's raw value, corrupting already-persisted records with no compile error and no crash — the app just starts reading/writing the wrong case for existing rows. Confirmed 2026-08-26 on killer-sudoku (ADR 0011): removing `.beginner` from a `Difficulty` enum with implicit raw values silently broke previously persisted `SolveRecord`s. Fix: any enum backing persisted state must declare explicit `rawValue`s up front, and case removal must leave the remaining values untouched (deprecate/reserve the slot rather than closing the gap).
- *A Compose/Kotlin canvas animation (pulse, progress bar, transition) builds and runs with no errors but visually never advances*: computing elapsed time by mixing a monotonic clock (`System.nanoTime()`) for one side of the subtraction with a wall-clock/epoch one (`System.currentTimeMillis()`) for the other produces a nonsense magnitude, usually already outside the animation's clamp range — so it silently renders in its finished (or never-started) state forever, with no exception to point at the mismatch. Confirmed 2026-08-26 on killer-sudoku's Kotlin/Compose Multiplatform port (`BoardCanvas.kt`): every board animation was dead because `pulseProgress` compared a `nanoTime()`-based `now` against an epoch-based start time; found via temporary `println` instrumentation only after two wrong guesses (font family, then a first mis-fix). Fix: pick one clock source for a given elapsed-time calculation — prefer `System.nanoTime()`, since it's monotonic — and use it consistently for both the captured start and every later `now` read.
- *Keyboard input silently does nothing in a Compose Desktop/Multiplatform window, with no error or console output*: requesting focus once at launch (e.g. a single `focusRequester.requestFocus()` in a `LaunchedEffect(Unit)`) doesn't survive the window losing and regaining focus, or the user clicking elsewhere in the same window — Compose Desktop doesn't automatically restore focus to that composable afterward. Confirmed 2026-08-26 on killer-sudoku's Kotlin port: the board never accepted key events after the very first interaction because focus was only requested once at launch. Fix: re-request focus on every relevant user interaction (e.g. on each board tap), not just at startup.
- *Verifying visual/pixel parity between a UI port and its original across different rendering toolkits (Skia vs CoreGraphics, Compose vs SwiftUI, etc.)*: eyeballing side-by-side screenshots is unreliable for this — small offsets (a grid line a few pixels off, slightly different glyph metrics) look "close enough" but are real regressions, and a visual read can also give a false-positive parity verdict. Confirmed 2026-08-26 on killer-sudoku's Kotlin port: an eyeballed screenshot comparison gave a wrong verdict once. Fix: use pixel-level analysis (Pillow + numpy — grid-line detection, glyph bounding-box extraction) for cross-toolkit visual parity checks instead of relying on visual inspection alone.
- *A long-uptime local daemon (rendering loop, live wallpaper, background service) drifts into a bad state over weeks in a way that's expensive to root-cause live (needs multi-day instrumentation) but already runs under a launchd agent with `KeepAlive`*: rather than blocking the fix on live instrumentation, add a second, separate launchd agent whose only job is a scheduled `pkill` of the target process — the target's existing `KeepAlive` relaunches it within seconds, forcing a periodic restart with zero changes to the original agent's plist or code. This is a mitigation, not a root-cause fix — pair it with a scoped backlog entry (with the actual instrumentation plan) rather than treating "restart fixes it" as closing the investigation. Confirmed 2026-08-26 on `~/.agents/brain-map`/DesktopLive: A/B'd two screenshots of the 22-day-old process 4s apart (genuine motion, not frozen, just zoomed in tight) against a fresh restart (immediately clean) to confirm drift vs. freeze before committing to this fix; root cause (suspected `fitVelocity` spring-physics accumulation) deliberately deferred, instrumentation plan logged in `~/.claude/suggestions.md`.
- *New code calls a real side-effecting module (ticket stage tracker, log writer, state file) and existing tests start writing to the REAL `~/.claude/` files*: when the new call isn't dependency-injected, any pre-existing test that uses realistic-looking inputs (real ticket numbers, real paths) silently exercises it against live state, polluting production data with no test failure. Confirmed 2026-10-01 on MARVIN's ticket-stage work (`ticket_stages.py/.js`): hit repeatedly in one session, each time fixed after the fact (autouse pytest fixture redirecting the home/log dir; a `vi.mock` on the JS side). Fix: when adding a call to a side-effecting module, immediately add an autouse fixture/mock at the test-suite level (or inject the path), and grep existing tests that reach that code path, rather than waiting for the next pollution to surface.
- *A classifier/lookup/search feature "works" but results are mediocre, and the primary backend (vector DB, embedding index) may be empty or unreachable*: a silent fallback to a weaker path (e.g. keyword matching) hides the failure — no error, just lower accuracy. Confirmed 2026-10-01: `route.py`'s embedding classifier had an empty ChromaDB collection and silently fell back to ~28%-accurate keyword matching; found only while benchmarking it against another approach (re-seeded via `intent_classify.py --seed`). Fix: when a feature has a fallback path, check which path actually served the result before trusting measured quality, and give the fallback a loud signal (log/Health check) so an emptied primary can't go unnoticed.
- *A merge/CI gate runs tests in a fresh scratch worktree and reports a "massive error" on a PR that has no conflicts*: a new worktree has no `node_modules`, so the test command fails wholesale on missing deps rather than on anything in the PR. Confirmed 2026-10-01 in `webhook-server/merge.js` (`_defaultRunTests`): PR #119 sat 30 days blamed on a merge conflict; real cause was no `npm install` before `npx vitest run`. Fix: install deps in the scratch worktree before running tests; when a gate fails on an unchanged-looking PR, read the gate's actual output before assuming conflict.
- *An automated measure/metric (test counts, pass-rate, before/after delta) reads flat, zero, or "no change" for days or weeks and nobody questions it*: suspect the measurer before the thing measured. A crashed or unparseable measurement coerced to `0` is indistinguishable from a real zero, so a broken harness looks like a healthy, unchanging system. Confirmed 2026-10-02 on MARVIN's ticket-pipeline measure, blind for ~a month with five independent causes stacked: child process fd limit (256) crashing the runner; a diff-scoped test command that skipped most tests; crash counted as 0 instead of raised; a parser that took the first "N passed" line instead of the final summary line; and a `#` in the worktree dir name crashing vitest. Fix: make measurement failure a distinct state (a `MeasureError`, not a number), report per-suite counts, parse the summary line only, run a fixed whole-suite command rather than a diff-derived one, and sanity-check the harness against a known-nonzero baseline before trusting any delta.
- *A scheduled job installed on more than one machine "should be working" because health is green, or because the identical job works on the other machine*: check which machine's health check actually reads that job's log. A central health check usually reads only its own machine's local logs, so a per-machine copy of the job elsewhere can fail every run and never appear in any report, and a working twin with the same plist hides it further. Confirmed 2026-10-06: the laptop's tidy-agent had thrown `PermissionError` every night since 2026-07-13 (about 12 weeks, 38 files waiting), while health-check ran only on the mac-mini, whose identical tidy-agent was fine. It was found by chance during storage work. Fix: for any job in `health_checks.JOB_PLACEMENT` that runs on more than one machine, make sure each machine's log is read somewhere (a per-machine check, or ship the log to the checker). Until then, read the job's own `*.err.log` on that machine directly before assuming it works.
- *A blocking failure persisted for days even though a check detected it and printed it every session*: look at how the alarm was shown, not only whether it fired. If a "this is stuck" line has the same visual weight as routine backlog counts (suggestions, queue, quarantine), it reads as one more number and gets skimmed, so in practice it is silent. Confirmed 2026-10-14: `session_start_report.py` printed the `~/.claude` sync-conflict line every session for 4 days (766 `code_sync` refusals, 6 memories and 2 handoffs never reached the Mini) as one bullet among 52/182/73 backlog counts. Fix: blocking states go first in the report, set apart from backlog counts, and include how long they've been stuck and how many times they've failed, so the size of the problem is visible. Not-yet-verified: the fix was scoped but not built at review time.
- *A status/sync log says an action succeeded, but the remote/target never got it*: check where the log line is written in relation to the outcome it reports. A log entry written *before* the action finishes (for example, so it is swept into the same commit, which is how `code_sync.py` `push()` avoids self-describing-log conflicts) can only record what was intended. Confirmed 2026-10-14: the sync log said "committed + pushed 16 file(s)" at 16:37:30, but the push was rejected. The next entry showed a CONFLICT in the merge after the rejection, and `origin/main` never got the commit. Fix: record success only after a confirmed result (exit code plus remote ref check), log rejection or conflict as its own state, and make that state count as a blocker for the alarm above.

## Phase 4 — Instrument

Each probe must map to a specific prediction from Phase 3. **Change one variable at a time.**

Tool preference:

1. **Debugger / REPL inspection** if the env supports it. One breakpoint beats ten logs.
2. **Targeted logs** at the boundaries that distinguish hypotheses.
3. Never "log everything and grep".

**Tag every debug log** with a unique prefix, e.g. `[DEBUG-a4f2]`. Cleanup at the end becomes a single grep. Untagged logs survive; tagged logs die.

**Perf branch.** For performance regressions, logs are usually wrong. Instead: establish a baseline measurement (timing harness, `performance.now()`, profiler, query plan), then bisect. Measure first, fix second.

## Phase 5 — Fix + regression test

Write the regression test **before the fix** — but only if there is a **correct seam** for it.

A correct seam is one where the test exercises the **real bug pattern** as it occurs at the call site. If the only available seam is too shallow (single-caller test when the bug needs multiple callers, unit test that can't replicate the chain that triggered the bug), a regression test there gives false confidence.

**If no correct seam exists, that itself is the finding.** Note it. The codebase architecture is preventing the bug from being locked down. Flag this for the next phase.

If a correct seam exists:

1. Turn the minimised repro into a failing test at that seam.
2. Watch it fail.
3. Apply the fix.
4. Watch it pass.
5. Re-run the Phase 1 feedback loop against the original (un-minimised) scenario.

## Phase 6 — Cleanup + post-mortem

Required before declaring done:

- [ ] Original repro no longer reproduces (re-run the Phase 1 loop)
- [ ] Regression test passes (or absence of seam is documented)
- [ ] All `[DEBUG-...]` instrumentation removed (`grep` the prefix)
- [ ] Throwaway prototypes deleted (or moved to a clearly-marked debug location)
- [ ] The hypothesis that turned out correct is stated in the commit / PR message — so the next debugger learns

**Then ask: what would have prevented this bug?** If the answer involves architectural change (no good test seam, tangled callers, hidden coupling) hand off to the `/improve-codebase-architecture` skill with the specifics. Make the recommendation **after** the fix is in, not before — you have more information now than when you started.