Observability Maturity Check

---
name: observability-maturity-check
enabled: true
description: |
  Use when performing observability maturity check — evaluates an organization's
  observability maturity across the three pillars (metrics, logs, traces) plus
  alerting, dashboards, and AIOps capabilities. Identifies gaps in visibility,
  assesses signal quality, and produces a roadmap for achieving full-stack
  observability.
required_connections:
  - prefix: monitoring
    label: "Monitoring Platform"
config_fields:
  - key: monitoring_stack
    label: "Current Monitoring Stack"
    required: true
    placeholder: "e.g., Datadog, Prometheus+Grafana, New Relic, Splunk"
  - key: service_count
    label: "Number of Services in Production"
    required: true
    placeholder: "e.g., 50"
  - key: primary_concern
    label: "Primary Observability Concern"
    required: false
    placeholder: "e.g., alert fatigue, blind spots, slow troubleshooting"
features:
  - DEVOPS
  - OBSERVABILITY
  - MONITORING
---

# Observability Maturity Check

## Phase 1: Metrics Assessment
1. Evaluate metrics collection and usage
   - [ ] Infrastructure metrics (CPU, memory, disk, network)
   - [ ] Application metrics (request rate, error rate, latency - RED)
   - [ ] Business metrics (transactions, revenue, user activity)
   - [ ] Custom metrics for domain-specific KPIs
   - [ ] Metrics retention and resolution appropriate
   - [ ] Metrics naming conventions standardized
   - [ ] Service-level indicators (SLIs) derived from metrics
2. Score: 1 (Basic infra only) to 5 (Full-stack with business metrics)

### Metrics Coverage

| Layer | Coverage | Gaps | Quality |
|-------|----------|------|---------|
| Infrastructure | % of hosts/containers | | High/Med/Low |
| Application (RED) | % of services | | |
| Database | % of instances | | |
| External dependencies | % of integrations | | |
| Business KPIs | % defined | | |

## Phase 2: Logging Assessment
1. Evaluate logging practices
   - [ ] Centralized log aggregation
   - [ ] Structured logging (JSON) across all services
   - [ ] Consistent log levels (DEBUG, INFO, WARN, ERROR)
   - [ ] Request/correlation IDs in all log entries
   - [ ] PII/sensitive data redaction in logs
   - [ ] Log retention policies defined
   - [ ] Log search performance adequate
   - [ ] Log-based alerting configured
2. Score: 1 (Scattered file logs) to 5 (Centralized, structured, searchable)

## Phase 3: Distributed Tracing Assessment
1. Evaluate tracing implementation
   - [ ] Tracing instrumented across services
   - [ ] Trace propagation across service boundaries
   - [ ] Span attributes include relevant context
   - [ ] Sampling strategy defined (head, tail, adaptive)
   - [ ] Trace-to-log and trace-to-metric correlation
   - [ ] Service dependency map generated from traces
   - [ ] Trace data used in incident investigation
2. Score: 1 (No tracing) to 5 (Full distributed tracing with correlation)

### Three Pillars Coverage

| Pillar | Coverage (%) | Quality (1-5) | Tool | Key Gap |
|--------|-------------|--------------|------|---------|
| Metrics | % | | | |
| Logs | % | | | |
| Traces | % | | | |
| **Combined** | **%** | **/5** | | |

## Phase 4: Alerting Quality Assessment
1. Evaluate alerting effectiveness
   - [ ] Alerts are actionable (every alert requires human action)
   - [ ] Alert severity levels match impact
   - [ ] On-call rotation and escalation configured
   - [ ] Alert noise ratio acceptable (signal vs. noise)
   - [ ] Symptom-based alerts (not cause-based)
   - [ ] Runbooks linked to alerts
   - [ ] Alert fatigue measured and managed
   - [ ] SLO-based alerting (burn rate alerts)
2. Measure alert quality metrics

### Alert Quality Metrics

| Metric | Current | Target | Status |
|--------|---------|--------|--------|
| Alerts/week | | < | |
| Actionable alerts % | % | > 80% | |
| Alerts with runbooks % | % | 100% | |
| MTTA (time to acknowledge) | min | < 5 min | |
| False positive rate | % | < 10% | |
| Duplicate/correlated alerts | % | < 5% | |

## Phase 5: Dashboards & Visualization
1. Evaluate dashboard practices
   - [ ] Service-level dashboards for each team
   - [ ] On-call dashboard (single pane of glass)
   - [ ] SLO status dashboards
   - [ ] Infrastructure overview dashboards
   - [ ] Business metrics dashboards
   - [ ] Dashboard naming and organization standards
   - [ ] Dashboard-as-code (version controlled)
2. Assess dashboard discoverability and usefulness

## Phase 6: Advanced Capabilities
1. Evaluate advanced observability features
   - [ ] Anomaly detection (automated baseline comparison)
   - [ ] Root cause analysis automation
   - [ ] Service dependency visualization
   - [ ] Change correlation (deploy/config changes vs. incidents)
   - [ ] Continuous profiling (CPU, memory, lock)
   - [ ] Real User Monitoring (RUM) / frontend observability
   - [ ] Synthetic monitoring for critical paths
   - [ ] OpenTelemetry adoption for vendor neutrality

### Maturity Scorecard

| Dimension | Score (1-5) | Current State | Target State |
|-----------|-----------|---------------|-------------|
| Metrics | | | |
| Logging | | | |
| Tracing | | | |
| Alerting | | | |
| Dashboards | | | |
| Advanced (AI/ML) | | | |
| **Overall** | **/5** | | |

## Counter-Rationalizations

| Shortcut | Counter | Why |
|----------|---------|-----|
| "We can skip some steps for this case" | Adapt the workflow steps, don't skip them | Skipped steps are where incidents and oversights originate |
| "The user seems to already know what to do" | Complete all workflow phases with the user | The workflow catches blind spots that experience alone misses |
| "This is a minor case, full process is overkill" | Scale the process down, don't turn it off | Minor cases become major when unstructured; the process scales, not disappears |
| "I'll fill in the details later" | Complete each section before moving on | Deferred details are forgotten; real-time capture is more accurate |
| "The template output isn't necessary" | Always produce the structured output format | Structured output enables comparison, audit trails, and handoff to other teams |

## Output Format
- **Pillar Assessment**: Metrics, logs, traces coverage and quality
- **Alert Quality Report**: Noise analysis and improvement recommendations
- **Coverage Gaps**: Services and layers lacking observability
- **Tool Assessment**: Current stack evaluation and recommendations
- **Improvement Roadmap**: Phased plan to advance observability maturity

## Action Items
- [ ] Assess coverage across all three pillars
- [ ] Audit alert quality and reduce noise
- [ ] Implement structured logging across all services
- [ ] Roll out distributed tracing to uninstrumented services
- [ ] Standardize dashboards and make discoverable
- [ ] Evaluate and pilot advanced capabilities
- [ ] Schedule quarterly observability review