Overview

---
name: overview
description: "Vendor-neutral observability strategy guidance across metrics, logs, and traces: monitoring strategy, tool selection, SLI/SLO/SLA design, alerting philosophy, and the three pillars of observability. Use for \"monitoring\", \"observability\", \"metrics\", \"logs\", \"traces\", \"alerting\", \"which monitoring tool\", \"dashboard strategy\", \"SLO\", \"SLI\", \"error budget\", \"alert fatigue\", \"observability stack\", or comparisons like \"Prometheus vs Datadog\". Do NOT use for tool-specific syntax or config — use the tool's own skill (`prometheus`, `grafana`, `elk`, `opentelemetry`, `datadog`, `newrelic`, `dynatrace`, `splunk`, `zabbix`, `nagios`, `pagerduty`)."
license: MIT
---

# Monitoring & Observability Strategic Expert

This skill covers monitoring and observability strategy spanning open-source, managed, and enterprise tooling: vendor-neutral guidance on:

- The three pillars of observability (metrics, logs, traces)
- Tool selection (open-source vs managed vs enterprise vs legacy)
- Monitoring strategy (USE method, RED method, 4 Golden Signals)
- SLI/SLO/SLA design and error budgets
- Alerting philosophy and fatigue prevention
- Cardinality management and cost control

This skill is strategic. For tool-specific implementation, use the appropriate technology skill.

## How to Approach Tasks

When you receive a request:

1. **Classify** the request type:
   - **Observability strategy** -- Use the three pillars framework and monitoring methodologies below
   - **Tool selection** -- Use the tool selection framework
   - **Concepts and theory** -- Load `references/concepts.md`
   - **Technology-specific** -- Use the relevant technology skill (Prometheus, Grafana, ELK, OpenTelemetry)

2. **Understand context** -- Ask about existing tooling, team size, budget, data volume, and compliance requirements before recommending a stack.

3. **Be honest about trade-offs** -- Every tool can monitor every workload. The question is which makes a given workload easier, cheaper, or more reliable.

4. **Include cost context** -- Observability spend can exceed 30% of infrastructure cost. Always address cost implications of tool choices, cardinality decisions, and retention policies.

## Three Pillars of Observability

Observability is the ability to understand the internal state of a system by examining its external outputs. The three pillars produce complementary data:

### Metrics

Numeric measurements aggregated over time. Lightweight, cheap to store, ideal for dashboards and alerting.

**When to use:** System health at a glance, trend analysis, capacity planning, SLO tracking, alerting on thresholds.

**Examples:** CPU utilization, request rate, error count, p99 latency, queue depth, disk usage.

**Key tools:** Prometheus (pull-based, open-source standard), Datadog Metrics, CloudWatch Metrics, InfluxDB, VictoriaMetrics.

### Logs

Timestamped records of discrete events. Rich in context, expensive at scale, essential for debugging.

**When to use:** Root cause analysis, audit trails, debugging specific errors, compliance, security forensics.

**Examples:** Application error logs, access logs, system logs, audit logs, deployment events.

**Key tools:** Grafana Loki (label-indexed, cost-effective), Elasticsearch/ELK (full-text indexed), Datadog Logs, Splunk, CloudWatch Logs.

### Traces

End-to-end request paths across distributed services. Show causality and latency breakdown.

**When to use:** Debugging distributed systems, identifying slow dependencies, understanding request flow, service dependency mapping.

**Examples:** HTTP request spanning API gateway, auth service, database, and cache. Each hop is a span; the collection is a trace.

**Key tools:** Grafana Tempo, Jaeger, Zipkin, Datadog APM, Elastic APM, Dynatrace. OpenTelemetry is the standard instrumentation layer.

### How the Pillars Connect

```
Metrics ──(exemplars)──> Traces ──(trace ID in logs)──> Logs
   ^                                                       |
   └──────────(log-derived metrics)────────────────────────┘
```

- **Metrics to traces:** Prometheus exemplars embed trace IDs in metric samples. Click a spike to see the exact traces causing it.
- **Traces to logs:** Trace IDs propagated into log lines. Click a span to see correlated log entries.
- **Logs to metrics:** Derive metrics from log patterns (count of errors per service, latency from access logs).

**Rule of thumb:** Start with metrics for detection, use traces for localization, use logs for root cause.

## Tool Selection Framework

### Open-Source Stack (Prometheus + Grafana + Loki + Tempo)

**Best for:** Cloud-native teams, Kubernetes-first environments, cost-sensitive organizations, teams comfortable with self-management.

**Strengths:** No license cost, massive community, Kubernetes-native service discovery, PromQL is the industry standard query language, unified Grafana UI for all three pillars, OpenTelemetry-first ecosystem.

**Watch out for:** Operational burden (scaling, HA, retention), Prometheus is single-node by default (need Thanos/Mimir for long-term storage), Loki requires careful label design, no built-in RBAC without Grafana Enterprise.

**When to choose:** Team has Kubernetes expertise, budget is limited, data sovereignty matters, vendor lock-in is a concern.

### Managed Platforms (Datadog / New Relic / Dynatrace)

**Best for:** Organizations wanting turnkey observability, teams without dedicated platform engineers, rapid time-to-value, full-stack APM with auto-instrumentation.

**Strengths:** Zero operational overhead, built-in APM with code-level visibility, AI-powered root cause analysis (Dynatrace Davis, Datadog Watchdog), unified billing, enterprise support.

**Watch out for:** Cost at scale (Datadog custom metrics pricing, per-host APM fees), vendor lock-in (proprietary query languages, agents), data residency limitations, egress charges.

| Platform | Pricing Model | Sweet Spot |
|----------|--------------|------------|
| **Datadog** | Per-host + per-metric + per-GB logs | Broad coverage, 700+ integrations, strong K8s |
| **New Relic** | Per-user + per-GB ingested | Generous free tier (100 GB/mo), full-stack |
| **Dynatrace** | Per-host (full-stack) | Auto-discovery, AI root cause, enterprise |

**When to choose:** Small ops team, need fast setup, budget allows SaaS pricing, compliance with SaaS data handling is acceptable.

### Enterprise Stack (Splunk / ELK)

**Best for:** Large enterprises with heavy log analytics, security use cases (SIEM), compliance requirements, existing Elastic or Splunk investments.

**Strengths:** Full-text search at scale, security analytics (Splunk SIEM, Elastic SIEM), mature RBAC and audit, on-premises deployment for data sovereignty.

**Watch out for:** ELK cluster management complexity (shard tuning, heap management), Splunk licensing cost (per-GB ingested), steep learning curves.

**When to choose:** Security and compliance are primary drivers, heavy log analytics workload, existing enterprise license agreements.

### Legacy Monitoring (Zabbix / Nagios)

**Best for:** Traditional infrastructure (bare-metal, VMware), network device monitoring, SNMP-based monitoring, organizations with established Zabbix/Nagios deployments.

**Strengths:** Proven reliability for infrastructure monitoring, SNMP support, agent-based or agentless, extensive template libraries.

**Watch out for:** Poor fit for cloud-native/microservices, limited distributed tracing support, dated UIs, scaling challenges.

**When to choose:** Monitoring physical infrastructure, network devices, legacy systems that expose SNMP. Not recommended for greenfield cloud-native projects.

### Decision Tree

```
Cloud-native / Kubernetes-first?
  YES --> Team can operate open-source?
    YES --> Prometheus + Grafana + Loki + Tempo + OpenTelemetry
    NO  --> Grafana Cloud (managed LGTM stack)
  NO --> Heavy log analytics or security/SIEM?
    YES --> ELK (self-hosted) or Splunk (managed)
    NO --> Want turnkey full-stack APM?
      YES --> Budget for SaaS?
        YES --> Datadog / New Relic / Dynatrace
        NO  --> Elastic APM or Grafana Cloud free tier
      NO --> Traditional infrastructure / SNMP?
        YES --> Zabbix (or Prometheus + SNMP exporter)
        NO  --> Prometheus + Grafana (start simple)
```

## Monitoring Strategy

### USE Method (Brendan Gregg)

For every **resource** (CPU, memory, disk, network, GPU), measure:

| Signal | Question | Example |
|--------|----------|---------|
| **Utilization** | What percentage of capacity is consumed? | CPU at 85% |
| **Saturation** | How much work is queued or waiting? | Run queue length 12 |
| **Errors** | How many error events occurred? | Disk I/O errors: 3 |

**Best for:** Infrastructure-centric dashboards (node health, database hosts, storage systems).

### RED Method (Tom Wilkie)

For every **service** (API, microservice, function), measure:

| Signal | Question | Example |
|--------|----------|---------|
| **Rate** | How many requests per second? | 1,200 req/s |
| **Errors** | What fraction of requests fail? | 2.3% error rate |
| **Duration** | How long do requests take? | p99 = 450ms |

**Best for:** Service-centric dashboards (API health, microservice performance).

### 4 Golden Signals (Google SRE)

A superset combining RED with saturation awareness:

1. **Latency** -- Time to serve a request. Distinguish successful request latency from error latency (errors are often fast).
2. **Traffic** -- Demand on the system. Requests per second, queries per second, sessions.
3. **Errors** -- Rate of failed requests. Explicit (HTTP 5xx) and implicit (200 with wrong content, slow responses treated as errors by SLO).
4. **Saturation** -- How full the service is. CPU, memory, I/O, queue depth. Measures how close to capacity the service is running.

**Best for:** SRE teams defining SLOs and building service-level dashboards.

### Dashboard Hierarchy

Design dashboards in layers for progressive drill-down:

| Level | Purpose | Content | Audience |
|-------|---------|---------|----------|
| **L1 Overview** | Is everything healthy? | Service grid with red/green status, top-level error rates | On-call, management |
| **L2 Service** | Which service has a problem? | RED/USE metrics for one service, SLO burn rate | On-call engineer |
| **L3 Diagnostic** | What exactly is broken? | Detailed metrics, correlated logs, trace waterfall | Debugging engineer |

Link dashboards together: L1 links to L2 via data links, L2 links to L3 via drill-down.

## Alerting Philosophy

### Symptoms vs Causes

**Alert on symptoms, not causes.** A symptom is user-visible impact (high error rate, slow responses). A cause is internal state (CPU high, disk full). Cause-based alerts generate noise because high CPU does not always mean user impact.

| Alert Type | Example | Priority |
|-----------|---------|----------|
| Symptom | Error rate > 5% for 5 minutes | Page (critical) |
| Symptom | p99 latency > 2s for 10 minutes | Page (warning) |
| Cause | CPU > 90% for 15 minutes | Ticket (warning) |
| Cause | Disk predicted full in 4 hours | Ticket (warning) |
| Informational | Deployment completed | Slack notification |

### Severity Levels

| Severity | Action | Response Time | Channel |
|----------|--------|--------------|---------|
| **Critical** | Wake someone up | Minutes | PagerDuty / phone |
| **Warning** | Investigate next business day | Hours | Ticket / Slack |
| **Info** | Awareness only | None | Dashboard / Slack |

### Anti-Patterns to Avoid

- **Alerting on every metric** -- Only alert on user-facing symptoms and critical infrastructure
- **No `for` clause** -- Transient spikes cause alert storms. Minimum `for: 1m`, typical `for: 5m`
- **Missing runbook URLs** -- Every alert must link to a runbook explaining what to check and how to fix
- **Duplicate alerts** -- One symptom should trigger one alert, not one per instance
- **Never-resolved alerts** -- If an alert fires and nobody acts, delete or tune it
- **Alerting on percentages without volume** -- 100% error rate on 1 request is noise

## Technology Skills

For tool-specific implementation guidance, use:

- the `prometheus` skill -- PromQL, scrape configuration, recording rules, alerting rules, Alertmanager, TSDB management, native histograms, service discovery, federation, remote write, Thanos/Mimir integration.
- the `grafana` skill -- Dashboard design, panel types, variables, transformations, provisioning, Unified Alerting, Loki/LogQL, Tempo/TraceQL, data source configuration, RBAC.
- the `elk` skill -- Elasticsearch cluster management, Elastic Agent/Fleet, ingest pipelines, KQL/ES|QL, ILM, APM agents, distributed tracing, log management, data streams.
- the `opentelemetry` skill -- Collector configuration (receivers, processors, exporters), auto-instrumentation, manual instrumentation, context propagation, sampling strategies, OTLP protocol, backend integrations.
- the `datadog`, `newrelic`, `dynatrace`, `splunk`, `zabbix`, `nagios`, `pagerduty` skills -- respective tool-specific implementation.

## Reference Files

Load these when you need deep knowledge for a specific area:

- `references/concepts.md` -- Three pillars of observability, SLI/SLO/SLA definitions, error budgets, cardinality, sampling strategies, USE/RED/Golden Signals methodology, alert design principles. Read for conceptual and strategic questions.