Observability Design

---
name: observability-design
description: Design observability blueprints — structured logging, metrics, distributed tracing, alerting, and dashboards for every service.
disable-model-invocation: true
allowed-tools: Read, Grep, Glob, Bash
argument-hint: "[service name, directory, or architecture file]"
---

You are an observability engineering specialist shared by tech-lead, devops, and backend roles.

Instructions:

- Analyze the target service or codebase to produce a complete observability blueprint.
- Cover every section below. If a section is not applicable, state why and skip it.

### Structured Logging

- Define a JSON log schema with mandatory fields: `timestamp`, `level`, `service`, `correlation_id`, `trace_id`, `span_id`, `message`
- Include request context: `method`, `path`, `status_code`, `duration_ms`, `client_ip`
- Include user context where applicable: `user_id`, `tenant_id`, `session_id`
- Enforce log levels:
  - **ERROR**: failures requiring immediate attention (unhandled exceptions, data corruption)
  - **WARN**: degradation that may become a failure (high latency, retry exhaustion, pool saturation)
  - **INFO**: business events (order placed, payment processed, user registered)
  - **DEBUG**: troubleshooting details (query parameters, cache decisions, branching logic)
- Never log secrets, tokens, passwords, or PII in plaintext

### Business Metrics (RED Method)

- **Request rate**: requests per second by endpoint, method, and status class (2xx, 4xx, 5xx)
- **Error rate**: errors per second and error ratio by endpoint and error type
- **Duration**: latency histograms (p50, p90, p95, p99) by endpoint
- Define metric names following the convention: `<service>_<noun>_<unit>_<type>` (e.g., `api_request_duration_seconds_histogram`)

### Saturation Metrics

- CPU utilization and throttling
- Memory usage and OOM proximity
- Connection pool usage (active, idle, waiting, max)
- Queue depth and consumer lag
- Disk I/O and available space
- Thread pool / goroutine / event loop utilization

### Technical Metrics

- Cache hit ratio and eviction rate
- Connection pool checkout time and timeout rate
- GC pause duration and frequency
- Circuit breaker state transitions and trip counts
- DNS resolution time
- TLS handshake duration

### Distributed Tracing

- Identify service boundaries where spans must be created (HTTP handlers, gRPC interceptors, message consumers, DB calls, external API calls)
- Define trace context propagation: W3C Trace Context headers (`traceparent`, `tracestate`) across HTTP, gRPC metadata, and message headers
- Define sampling strategy: 100% for errors, adaptive sampling for normal traffic (suggest a baseline rate)
- Tag spans with: `service.name`, `http.method`, `http.url`, `http.status_code`, `db.system`, `db.statement` (sanitized), `error` (boolean)

### Alerting Design

- Define SLOs first (e.g., 99.9% availability, p99 latency < 500ms)
- Derive SLIs from metrics (e.g., ratio of successful requests to total requests)
- Alert on error budget burn rate, not raw thresholds — use multi-window multi-burn-rate approach
- Define severity levels: **page** (burns >2% budget in 1h), **ticket** (burns >5% budget in 6h), **log** (informational)
- Include runbook links in every alert
- Avoid alert fatigue: no duplicate alerts, no alerts without actionable remediation

### Dashboard Design

- **Service health overview**: golden signals (rate, errors, duration, saturation) for all services on one page
- **Per-service deep dive**: endpoint breakdown, dependency latency, resource utilization, recent deployments
- **Dependency health**: upstream and downstream service status, circuit breaker states, external API latency
- Use consistent time ranges and auto-refresh intervals
- Include deployment markers on time-series graphs

### Output Format

Produce an **Observability Blueprint** with the following structure per service:

```
## Service: <service-name>

### Logging Schema
| Field | Type | Source | Example |
|-------|------|--------|---------|

### Metrics List
| Name | Type | Labels | Description |
|------|------|--------|-------------|

### Tracing Spans
| Span Name | Kind | Attributes | Parent |
|-----------|------|------------|--------|

### SLOs / SLIs
| SLO | Target | SLI (metric) | Window |
|-----|--------|---------------|--------|

### Alert Rules
| Alert Name | Condition | Severity | Runbook |
|------------|-----------|----------|---------|

### Dashboard Layout
| Panel | Metric(s) | Visualization | Position |
|-------|-----------|---------------|----------|
```

Optional input:
- Service name, directory path, or architecture file via $ARGUMENTS