Observability

---
name: observability
description: Use when designing or building anything that runs in production (an endpoint, a job, an integration, a payment), when setting up monitoring or alerts, and before any release. Instrument as you build, so that when it breaks at 3 a.m. the founder's question is answered by the telemetry, not by a guess.
---

# Observability

**Code you cannot observe is code you cannot operate.** Instrumentation is written with the
feature, in the same ticket, never after the first incident.

## 1. Questions first

Before adding any log, metric or trace, write in `design.md` the two to four questions that
will be asked when this feature breaks ("which users could not pay, and why?", "is the
provider slow or are we?"). Every signal added must answer one of them. If the questions
cannot be named, the feature is not understood well enough to instrument.

Three kinds of signal, each for one job: a **metric** says that something is wrong and how
often; a **trace** says where the time went; a **log** says why this one case failed.

## 2. Logs

- One structured event per line, with a stable event name and machine-readable fields
  (`event: payment_failed, provider, error_code, attempt`). Never a sentence built by string
  interpolation.
- A request id on every line, span and outbound call, created or accepted at the boundary.
  Without it the log is a pile, not a story.
- Levels: `error` (an invariant broke, someone may need to act), `warn` (degraded but
  handled: a retry that worked, a fallback used), `info` (a business event), `debug` (off in
  production).
- Never log secrets, tokens, passwords or personal data. Fields are allow-listed; a whole
  request body is never logged. `security` checks this on every diff.

## 3. Metrics

- For every endpoint and every external service: rate, errors and duration (RED). For every
  queue, pool and host: utilisation, saturation and errors.
- Duration is a histogram read as p50, p95 and p99. An average hides the users who waited.
- Labels come from small fixed sets (route template, status class `5xx`, provider name). Never
  a user id, an email, a request id, a full URL or an error message as a label.

## 4. Traces

Auto-instrument HTTP, database and queue clients; add manual spans only around meaningful
units of work (apply discounts, charge the provider). Carry the context across every async
boundary. Sample a low share of normal traffic and keep every error.

## 5. Alerts

- Alert on what users feel: an error rate, a latency percentile, a queue age. CPU, disk and
  memory go on a dashboard, not to the founder's phone.
- The thresholds and durations are the founder's (rule 2). Defaults to offer, from
  addyosmani/agent-skills (read 2026-10-21): error rate over 1 percent for 5 minutes, p99
  over 2 s, queue age over 10 minutes.
- Two severities only: **now** (someone acts tonight) and **this week** (a ticket). Each alert
  links to its runbook and is fired once on purpose before it counts as set up. An alert whose
  answer is "ignore it, it heals" is deleted.

## 6. Runbooks, `docs/runbooks/<alert>.md`

Three lines minimum: what it means, the first thing to check, who or what to escalate to.
Grow it only when the first check did not settle the question. Closing an incident that used a
runbook updates the runbook first.

## 7. Prove it before release

In staging, force a failure on the new path and find it from the telemetry alone: by request
id, with the structured fields, through the trace without a broken span. Send test traffic and
read the metric series. Fire each new alert and see it arrive where the founder chose. Until
this is done, the feature has monitoring on paper (`skills/release/`).

## Who does what

`architect` writes the questions and names the signals in `design.md`; `backend-dev` and the
platform developers emit them in the ticket that builds the path; `devops` wires the metrics,
alerts and runbooks and runs the proof; `dashboard` shows the business numbers the founder
watches; `security` reads every log line for personal data.