Monitoring

---
name: monitoring
description: Observability stack — Prometheus/Grafana, Datadog, metrics, alerting.
---

# Monitoring & Observability

## Core Principle

Instrument first → aggregate second → alert third. Never set up alerts before you know what to measure.

---

## Stack Selection

| Scenario | Recommended |
|----------|-------------|
| VPS / single server, cost-sensitive | Prometheus + Grafana (self-hosted via Docker) |
| AWS-native stack | CloudWatch (Metrics + Logs + Alarms) |
| GCP-native stack | Cloud Monitoring + Cloud Logging |
| Azure-native stack | Azure Monitor + Log Analytics |
| Multi-cloud or mixed stack | Datadog (SaaS) or Grafana Cloud |
| High log volume, self-hosted | Loki + Grafana (avoid ELK unless already invested) |

For full setup details, load the matching reference:

| Stack | Reference |
|-------|-----------|
| Prometheus + Grafana (+ Loki) | `references/prometheus-grafana.md` |
| AWS CloudWatch | `references/cloudwatch.md` |
| Datadog | `references/datadog.md` |

---

## Application Instrumentation Requirements

Confirm with the backend developer before setting up any monitoring stack:

| Language | Library |
|----------|---------|
| Node.js | `prom-client` |
| Python | `prometheus-client` |
| Go | `prometheus/client_golang` |
| Java | Micrometer |
| PHP | `promphp/prometheus_client_php` |

Minimum metrics every app must expose:
- HTTP request count (by method, path, status)
- HTTP request duration (histogram — P50/P95/P99)
- Active connections or queue depth
- Error count (by type)

---

## Before Declaring Done

- [ ] Application exposes `/metrics` with the minimum metrics above
- [ ] All critical services have a `ServiceDown` alert
- [ ] Error rate and latency alerts defined and routing to a notification channel
- [ ] Disk usage alert on every host > 80%
- [ ] Dashboard created: request rate, error rate, latency P95, CPU, memory
- [ ] Log aggregation collecting from all services
- [ ] Log retention policy set — never unlimited
- [ ] Runbook URL in alert annotations
- [ ] **If Prometheus**: AlertManager configured and test alert delivered to Slack/PagerDuty
- [ ] **If Loki**: label cardinality reviewed — no high-cardinality labels on streams
- [ ] **If Datadog**: Unified Service Tagging applied; APM traces visible for at least one critical flow
- [ ] **If CloudWatch**: retention policy on every log group; Log Insights tested with a sample query