Skip to content
Back to skills

Debug

ASecurity

Debug methodology — triage running-system evidence, guarded data reads, parallel investigation, and synthesis.

  • 10 stars
  • 0 votes
  • 0 copies
  • 0 views
  • Added November 13, 2026
developmentgoshellbashsqlnextjsrefactoringgitapidatabasefrontend

Works with

  • cli
  • api
  • mcp

Security analysis

A100/100

Pro scans all 7 files and shows the line behind each finding

Scanned November 13, 2026

npx -y skills add reliant-labs/forge --skill debug --agent claude-code

Installs into .claude/skills of the current project.

Are you the author of Debug?

Add the live security badge to your README. It updates with every re-scan.

Security grade badge for Debug
[![Security: A — Skills Directory](https://www.skillsdirectory.com/api/skills/reliant-labs-debug/badge)](https://www.skillsdirectory.com/skills/reliant-labs-debug)

More formats (shields.io, HTML) on the badges page. Keep it an A: scan every change in CI with Pro.

Download with Pro
SKILL.md
---
name: debug
description: Debug methodology — triage running-system evidence, guarded data reads, parallel investigation, and synthesis.
emit: both
applies: any
---

# Debug Methodology

## Triage First

Classify the bug before diving in:

- **Crash / panic** → check logs, attach a debugger.
- **Wrong behavior** → trace the code path, form hypotheses, then verify with a targeted test.
- **Only in multi-service / multi-process flows** → reproduce with an end-to-end test against the full stack.
- **Flaky test** → run in a loop with stress / race detection enabled.
- **Works locally, fails in CI** → diff the environment (versions, env vars, filesystem, time, network) before assuming a code bug.

## Parallel Investigation

If your environment supports parallel agents, split the work into three independent tracks:

- **Researcher** — trace code paths, form hypotheses, check git log for recent changes that touch the affected area.
- **Tester** — write a failing test that isolates the bug. Top-down bisection: start at the outermost reproducing layer (e2e), then narrow to the unit.
- **Reproducer** — add diagnostic logging, exercise the path in a running system, collect runtime evidence (request IDs, error chains, timing).

Working solo? Run the three passes sequentially: research → reproduce → isolate.

## Synthesis

Combine findings from all tracks before proposing a fix:

- **Root cause** with confidence level (High / Medium / Low).
- **Evidence** from each investigation track — which path, which test, which log line.
- **Recommended fix** approach — hand off to an implementer; don't fix in debug mode.

## Discipline

- **Root cause over symptom.** Trace the failing call through the *real* code path and read the actual server / pod / service logs — don't pattern-match the error message and guess. The first plausible explanation is often a symptom one layer down from the cause.
- **Reproduce before you guess.** A bug you can't trigger on demand is one you can't verify a fix for.
- **One hypothesis per test.** A failing test that "exercises the bug area" doesn't prove the hypothesis; one that fails for exactly one reason does.
- **Check connectivity and credentials first.** For anything that crosses a process or cluster boundary, the top failure mode is stale connection state, not a logic bug: can the caller actually reach the target API/endpoint? Is the kubeconfig context current and pointed at the right cluster? Are the credentials live? Confirm the wire before you debug the payload.
- **Fail fast — isolate and name the one blocker.** When stuck, don't loop re-running the same failing command with small tweaks. Identify the single specific thing that's blocking (a 403, an unreachable host, an undefined symbol) and state it plainly. A precise blocker is actionable; "it's broken" is not.
- **Don't widen the diff.** Touch only what the evidence indicts; refactoring "while you're in there" turns a 1-line fix into a 200-line PR that needs its own review.

<!-- @forge-only:start -->
## Forge-Specific Triage

On top of the generic triage above, common forge-shaped bug classes:

- **Stale generated code / config drift** → if symbols are "undefined" or a build breaks *right after* a proto / forge.yaml / schema change, suspect stale gen before you chase the code. Run `forge generate`, then `forge project audit` to surface drift, then retest. Forge generates from proto + forge.yaml; stale gen masquerades as a bug in hand-written code.
- **Can't reach a service / cross-cluster flow fails** → verify connectivity and credentials before assuming a logic bug. Each workload deploys to the kubectl context its `forge.OnCluster {target}` names (multi-cluster is native); a "service unreachable" or auth failure is usually a stale or wrong kubectl context, an expired credential, or a service that isn't up — not a code path. Confirm the context is current and the target responds first.
- **Broken DI wiring** → check the explicit composition (`internal/app/compose.go` `NewComponents`, off the owned `internal/app/providers.go` `OpenInfra`). Deps are interface-typed fields resolved by type off `Infra`, so a missing required provider is a compile error (`NewComponents` won't build) or a `forge generate` "no provider" report, not a silent nil — but a wrong fill (an unintended optional dep left nil) can still surface as a nil-pointer panic deep in handler code. There is no generated `wire_gen.go`/`bootstrap.go` to inspect; the wiring is `NewComponents` filling typed `Deps` off `Infra`.
- **Mock vs real divergence** → tests pass with generated mocks but the real adapter fails. Re-run the integration suite (`task test:integration`) before chasing a unit-test ghost.
- **Proto-DB drift** → entity types and DB schema evolve independently; `forge project audit` flags the mismatch. If the symptom is "column X not found" in a handler that names the right struct field, audit first.

## Forge Debug Tools

```
forge project introspect handlers      # every RPC path the binary registers — localize the fault
forge api curl <svc.method>    # build a copy-pasteable Connect curl from the shell
forge cluster logs --service X # kubectl-backed log tail for one service (single cluster)
forge debug start <svc>        # debug-build <svc>, run it under Delve with env dev's config
forge debug start --attach PID --run # attach Delve to a live process; --run keeps it running
forge debug connect <addr>     # drive a dlv you started (e.g. the delve runner: 127.0.0.1:2345)
forge debug break/continue/eval# breakpoints, wait for a hit (continue --timeout 60s), evaluate
forge debug stop               # kills only what start launched; detaches/releases everything else
task test -- -v ./internal/handlers/<name>/... # verbose isolated test runs
task test:e2e                 # full-stack reproduction
task test              # run tests with race detector
forge generate                 # regenerate code (use when stale gen is suspected)
```

### Ask the running system before reading code

Start with `forge obs status <env> --json`, then follow the failing request,
not a broad production scan. Load `forge skill load observability/query` for
the full verb/JSON reference and `forge skill load db/query` for application
rows. Each step should narrow the hypothesis:

| Evidence needed | Read-only command |
|---|---|
| Scope, telemetry freshness, failing services and release | `forge obs status prod --since 15m --json` |
| Error lines in that window | `forge obs logs prod 'body:timeout' --service api --since 15m --limit 5 --json` |
| Failed requests, then the call tree | `forge obs traces prod --errors --service api --since 15m --limit 5 --json`, then `forge obs trace <trace-id> prod` |
| Grouped cause, call site and latest event: every ERROR and every WARN server failure (`error_class=server`, e.g. a retryable Aborted), not a caller's timeout | `forge obs issues prod --since 15m --limit 5 --json`, then `forge obs issue <id> prod` |
| Logs for that request | `forge obs logs prod --trace <trace-id> --since 15m --limit 5 --json` |
| Available metrics, then values | `forge obs metrics list prod --contains rpc --since 15m`, then `forge obs metrics query prod rpc.server.call.duration --agg quantile --quantile 0.95 --since 15m` |
| A question beyond the structured verbs | `forge obs endpoints dev --explain --json`, then adapt its SQL with `forge obs query dev --sql-file investigation.sql` |
| A dashboard panel's data | `forge obs query dev --panel panel.json --since 15m` |
| A dashboard that looks empty | `forge obs dashboards audit prod --dashboard <id> --json` (recipe below) |
| Continuous CPU/heap evidence | `forge obs profile --env dev --service api --since 15m -o /tmp/api.pb.gz` (local ClickStack; see below) |

`api` is an example **telemetry service.name**, not necessarily a workload or
binary name. Use the name status actually returned. Prefer complete `suggested`
commands from the result, but verify their env/backend before running them.
Use `forge env releases at <env> <time-or-trace-id>` to relate evidence to the
release live then. `forge obs check <env>` judges a deploy, not an arbitrary
application-flow invariant. Reading issues is not triage: do not resolve,
ignore, assign or comment during a read-only investigation.

#### Bind the project, environment and authority first

Run the project's pinned Forge; check `forge version` and heed version-skew
warnings. An embedded `reliant forge` can differ from the project's standalone
binary. Run in the project root (or select it with `-C <project-root>`), and
name the env explicitly: most obs verbs default to dev, **db query never does**.
Servers need not share your filesystem; `--sql-file` and `--panel` read files
on the CLI machine, not a remote API server.

The env declaration should resolve the telemetry location: local ClickStack
or the control plane. Normal use is env-first, without a backend flag. Check
the returned `scope` (env, backend, cloud env_id, services and time window).
On an older binary that routes the env incorrectly, `--backend cloud` or
`--backend local` is a troubleshooting override, not required configuration;
report the routing/version mismatch instead of normalizing it. Start with a
short window, then fix `--from`/`--to` for comparable evidence.
`--explain` prints SQL/parameters instead of executing the telemetry query;
it is not a connectivity or authorization test.

Cloud reads require the credential's role-bound `obs:read`; release reads may
also need `deploy:read`, and hosted database reads need `db:query`. Under
Reliant use the credential helper, not an interactive `forge login`. Elsewhere
use the project's documented login. HTTP 403/missing scope on a valid token
is an authorization blocker, not evidence of an app outage. Report it to the
owner/admin; do not mint broader tokens, read secret values or bypass policy.
A backend that is itself broken (for example ClickHouse code 33) may also break
its own observability: use an already-authorized, read-only out-of-band source
such as the owning cluster's pod logs. Coordinate with the incident
investigator rather than duplicating scans or escalating privileges.

#### Unknown is not healthy

With `--json`, inspect `ok`, `exit_code`, `error`, `scope`, and, where present,
`unread`, `warnings`, `not_evaluated`, `truncated`, `omitted`, `next_page_token`.
**Status exits 0 when any section was read**, even if every telemetry section
is unread and only the release ledger was available. An empty array means
read-and-empty; an absent/unread section means unknown. A capped result is
incomplete: narrow or page it, never infer absence from it.

For obs reads, exit 1 means a bad/refused request; exit 2 means Forge could not
look (auth, quota, unreachable/off telemetry or an unsupported backend call).
Database reads similarly distinguish refused SQL/policy (1) from unavailable,
rate-limited or timed-out reads (2). Preserve the exact error/reason: missing
scope, quota and a SQL error need different actions. Do not retry blindly or
turn a failed query into “no matching errors”.

#### Query telemetry and application data separately

`forge obs query` is **ClickHouse telemetry SQL**, not the app's Postgres.
Start with `--explain` from a structured verb. For example, count logs in the
bound window without interpolating values into the statement:

```bash
forge obs query dev --since 15m --service api --sql \
  'SELECT count() AS n FROM otel_logs
   WHERE Timestamp >= {from:DateTime64(9)} AND Timestamp < {to:DateTime64(9)}
   AND (empty({services:Array(String)}) OR has({services:Array(String)}, ServiceName))'
```

The parameters bind time/services; keep the predicates when adapting SQL.
The cloud backend enforces the credential's environment scope. The guard
allows one read-only SELECT, not DDL, writes, system tables or table functions.
`--panel` takes **one panels[] element**, not a whole dashboard document;
load `observability/dashboards` for authoring and `observability/query` for
`--param`, `--step` and the JSON contract.

`forge db query` reads the workload's **application database** instead:

```bash
forge db query dev 'select 1 as reachable' --max-rows 1 --timeout 5s --json
# For an existing, authorized route to a non-hosted cluster DB:
forge db query prod 'select 1 as reachable' --addr localhost:15432 \
  --confirm-env prod --reason 'incident connectivity check' --max-rows 1 --json
```

Use `--workload` (default `api`) and, for several managed DBs, `--database` to
select the database that owns the records. A database forge does not run
(Cloud SQL behind a pooler) is reachable when the env declares it as a
`forge.ExternalDatabase`: `--database <name>` then tunnels to its service
with your kube context, on a real env too (load `db/query`). `--addr host:port` supplies only
**your existing VPN/tunnel/bastion route**; it creates no route and bypasses
neither policy nor guards. Hosted envs refuse `--addr`: the control plane
executes/audits the query. Cluster routes use `forge_ro`, its kube Secret and
CA without printing credentials; host/compose keeps the workload's URL user
with host/port replaced, not necessarily `forge_ro`. Keep default cluster TLS
verification; do not disable it to get past a failed connection.

Every query is one statement in a timed READ ONLY transaction under row/byte
caps. Real envs normally require `--confirm-env` and `--reason`; `deny` stays
denied. Hosted audited access is person/role-bound, not a machine-token escape
hatch. The managed read-only role excludes secret columns; select explicit
safe columns or aggregates, not `SELECT *`. Never reset, seed, migrate or write
production data to investigate it. `select 1` proves connectivity only, not
that app records or schema are correct. Full transport/policy rules: `db/query`.

#### An empty dashboard

Run `forge obs dashboards audit <env>` to replay the console's panels. Empty
panels are probed to distinguish absent data from data outside the window,
environment or filters; failed probes are not evidence of absence. Load
`forge skill load debug/dashboards` for verdicts, failure classes and remedies.

#### Tool API versus CLI

Forge's stdio MCP server is `forge mcp` (embedded: `reliant forge mcp`).
Register the entrypoint your installed binary actually exposes; load `debug`
with `skill_load` using `name: "debug"` and `dir: "<project-root>"` to select
the project. Its current catalog exposes project/skill/lint/generate tools,
**not obs or db query tools**. Use the harness's shell for those CLI reads;
do not invent MCP tool names. See `docs/mcp.md` in the Forge repository for
registration and the tool list. Older embedded binaries may not expose `mcp`;
check `--help` before registering.

**Pick the tool for the job — and know where each stops short:**

- **`forge project introspect handlers` to localize.** Prints every RPC path the assembled
  binary serves. If the failing RPC isn't there, the fault is a downstream/remote
  hop, not this binary — that one check collapses the search space.
- **`forge api curl <service.method>` to exercise an endpoint.** Builds and runs a
  Connect request from the shell, but it **stops at the auth interceptor** — it
  does not mint a token. Use it for shape/connectivity; supply your own credential
  for an authed call.
- **`forge cluster logs --service <name>` for the symptom.** kubectl-backed, no
  file-grepping — but it tails only the **owner cluster** (the one cluster name in
  config). For a multi-cluster app, set the context and run
  `kubectl --context <other> -n <ns> logs <pod>` for the other half.
- **`forge debug` (Delve): pass `--run` so the target keeps running.** `start <svc>` builds
  `-gcflags=all=-N -l`, launches it with the workload's configuration from
  `--env` (default `dev`, resolved like `forge env exec`), and writes its output to
  `.forge/logs/<env>/debug_<svc>.log`. When `forge env up` already serves the
  workload's PORT, either pass `--app-port <free>` or attach to the running
  process: `start --attach <pid>` (`forge env status <env> --json` reports each
  host service's `pid`). The loop that never blocks an agent:
  `forge debug break <Rpc>` (a short handler method name, `file.go:42`, or a full
  Delve function name), trigger the request in the background, then
  `forge debug continue --timeout 60s` — it returns the hit (or exits non-zero
  with the target still running). `eval`/`locals`/`stack` inspect;
  `continue --no-wait` resumes without waiting. `stop` kills only a process
  `start` launched: an attached process is detached with its breakpoints
  removed and keeps serving, and a `connect`ed dlv keeps running with the
  session's breakpoints cleared. A workload on the delve runner
  (`forge.OnHost {runner = "delve"}`) already runs under dlv on
  127.0.0.1:2345 (`delve_port`): `forge debug connect 127.0.0.1:2345`.
  Without `--run`, `start` leaves the target halted at entry (or where attach
  found it) until the first `continue`.

**Smoke/status are NOT app-flow proofs.** A green `forge env smoke` / `forge env status <env>`
checks listeners, local compose, and telemetry — never app-flow invariants. They
(and `forge cluster status`, green whenever pods are `Running`) can all be green
while the actual flow is broken. The **only** things that prove an app-flow fix:

1. a declarative, exit-coded app-health assertion (model: a project `doctor:<flow>`
   task that fails non-zero when the invariant is violated), and
2. a full `task test:e2e`.

Generic forge tools localize and surface evidence; they do not certify the fix.

## Frontend logs are ON DISK in dev — read them before reaching for a browser

**The browser console is not the only sink.** A scaffolded frontend installs
`installDevLogging()` (from `@reliantlabs/forge-web-runtime`) in dev, which
mirrors every `console.log/info/warn/error/debug` — plus every **uncaught
error and unhandled promise rejection** — to the dev server, which prints them
to its stdout. `forge env up` already tees that to a file:

```
.forge/logs/<env>/frontend_<name>.log     # ← browser lines land here, prefixed [browser:<level>]
.forge/logs/<env>/<service>.log           # ← backend services
```

So for a frontend bug, **grep the log first** — it costs one command and needs
no browser:

```bash
grep '\[browser:' .forge/logs/dev/frontend_web.log | tail -40
grep '\[browser:error\]' .forge/logs/dev/*.log          # errors across every frontend
tail -f .forge/logs/dev/frontend_web.log                 # follow while reproducing
```

The uncaught-error capture is the part worth internalizing: a `TypeError` that
crashes a render, or a rejected promise, produces **no `console` call at all**
and is invisible to a code search. It still lands in this file, with its stack.

**Every failed RPC is an error line too** (the runtime's interceptors write it,
whatever the page did with the error), naming the procedure, the Connect code
and the trace — the server's `traceresponse`, else the browser's `traceparent`:

```
[browser:error] rpc failed procedure=/shop.v1.OrdersService/ArchiveOrder code=deadline_exceeded message="deadline exceeded" trace_id=6cdcc2b8d409e05528b82b2687a58af2
```

`grep -r <trace_id> .forge/logs/dev/` finds it beside the server's `rpc
failed` line. Cancelled calls are left out. No such line? The frontend's
runtime predates it (needs the release after 0.13.10).

Three caveats that will otherwise mislead you:

- **Dev only.** The receiving endpoint is a `apply: "serve"` Vite plugin (or a
  Next.js route that 404s in production), and `installDevLogging` no-ops
  outside dev. An empty log in a deployed environment is expected, not a bug.
- **A project may have opted out.** It is ordinary scaffolded code the project
  owns — one call in `src/main.tsx` / `src/app/providers.tsx` and one plugin
  entry. If the file has no `[browser:` lines at all, check those two places
  before concluding the code never ran.
- **Lines arrive in batches, up to ~250ms late.** Forwarding coalesces console
  calls into one POST per quarter-second (errors flush immediately, and the
  buffer is beaconed on page unload), so `tail -f` lags slightly behind the
  browser. That is the fix for one-request-per-line saturating the browser's
  six-connections-per-origin pool and slowing the app's own RPCs.

### If the log shows empty `[browser:...]` lines, the endpoint is out of date

The endpoint is a **scaffold-once file the project owns**, so `forge generate`
will not update it — and a `@reliantlabs/forge-web-runtime` upgraded through npm
can outrun it. A pre-batch endpoint parses `{"entries":[…]}` as a single line,
finds no `msg`, and prints one empty line for the whole batch.

The client detects this (a 2xx with no `X-Forge-Devlog` response header), falls
back to one request per line so nothing is lost, and says so once:

```
[browser:warn] [forge-devlog] this dev server's /__forge/log endpoint predates batched posts; update it
```

To fix it, make the endpoint understand both shapes — a `{"entries":[…]}` batch
and a bare `{"level","msg"}` — print one `[browser:<level>] <msg>` line per
entry in a single `console.log`, and answer `X-Forge-Devlog: 2`. The current
version is what `forge scaffold frontend` writes into a new project:
`src/lib/devlog-plugin.ts` (Vite) or `src/app/%5F_forge/log/route.ts`
(Next.js). Copy it from a freshly scaffolded frontend.

Related failure modes that produce the same silence: a malformed or oversized
post now prints `[browser:warn] [forge-devlog] dropped …` rather than being
swallowed, and a client that could not deliver a batch prepends
`[forge-devlog] dropped N lines (<reason>)` to the next one. If you see none of
those and no `[browser:` lines at all, the forwarder is not installed.

Use chrome-devtools MCP tools (snapshots, console, network) when you need to
DRIVE the UI or inspect live DOM/network state — the log file tells you what
already happened; devtools tells you what is happening now.

## Sub-Skills (forge)

The three investigation tracks above each have a dedicated forge sub-skill with concrete patterns:

- `reproduce` — runtime evidence collection and e2e reproduction.
- `isolate` — top-down bisection from e2e to unit test.
- `investigate` — hypothesis formation and code tracing.
- `multi-cluster-e2e` — a workload stuck `Pending` / not-ready, or a flow that fails only in a k3d multi-cluster e2e env: read the pod status before the network, hold the pod up on failure, and the nested-cluster image-pull / host-gateway-DNS / path-MTU gotchas.

For local ClickStack endpoints and live pprof, load `forge skill load debug/telemetry`.
Continuous profiles: `forge skill load observability/profiles`.

<!-- @forge-only:end -->

Files in this skill

  • SKILL.md22.3 KB
  • dashboards/SKILL.md3.9 KB
  • investigate/SKILL.md3.5 KB
  • isolate/SKILL.md1.7 KB
  • multi-cluster-e2e/SKILL.md9.5 KB
  • reproduce/SKILL.md3.6 KB
  • telemetry/SKILL.md2.7 KB

Attribution

Is this your skill, or is something wrong with this listing? Request removal or report an issue. Author removals are honored within 72 hours.

View all in development →

Comments

Loading comments…

Newsletter

Get the weekly list of safe new skills, and the ones we pulled.