Harness Observability

# Harness Observability

> Structured logging, metrics, distributed tracing, and alerting strategy. The three pillars of observability, assessed and designed for production readiness.

## When to Use

- When designing or reviewing observability instrumentation for a service
- When auditing logging, metrics, or tracing coverage gaps
- When defining SLIs, SLOs, and alerting strategies for a new feature
- NOT for application performance benchmarking (use harness-perf)
- NOT for security-focused log analysis (use harness-security-review)
- NOT for incident response procedures (use harness-incident-response)

> See also: `harness-pulse` is the read-side companion to this skill. Pulse READS observability data (and external analytics/error signals) to produce daily product-pulse reports; this skill DESIGNS the instrumentation that produces those signals in the first place.

## Process

### Phase 1: DETECT -- Identify Existing Instrumentation

1. **Scan for observability libraries.** Check package manifests for instrumentation dependencies:
   - **Logging:** winston, pino, bunyan, log4js, structlog, serilog, zap, logrus
   - **Metrics:** prom-client, micrometer, statsd-client, @opentelemetry/sdk-metrics
   - **Tracing:** @opentelemetry/sdk-trace-node, dd-trace, jaeger-client, zipkin
   - **Platforms:** @datadog/browser-rum, newrelic, @sentry/node, elastic-apm-node

2. **Locate instrumentation code.** Search for logger, metrics, and tracing initialization:
   - Logger configuration files (log levels, formatters, transports)
   - Metrics registry and custom metric definitions
   - Tracer initialization and span creation patterns
   - Middleware for automatic HTTP instrumentation

3. **Detect collector and exporter configuration.** Look for:
   - OpenTelemetry Collector config (`otel-collector-config.yaml`)
   - Prometheus scrape configuration (`prometheus.yml`)
   - Grafana dashboards (`grafana/dashboards/`)
   - Datadog agent configuration (`datadog.yaml`)
   - Fluentd or Fluent Bit configuration for log forwarding

4. **Identify alerting configuration.** Search for:
   - Prometheus alerting rules (`alert.rules.yml`)
   - Grafana alert definitions
   - PagerDuty, Opsgenie, or Slack integration configs
   - SLO definitions in monitoring-as-code format

5. **Present detection summary:**

   ```
   Observability Detection:
     Logging: pino (structured JSON) -- 12 logger instances found
     Metrics: prom-client -- 8 custom metrics defined
     Tracing: @opentelemetry/sdk-trace-node -- initialized in src/tracing.ts
     Collector: OpenTelemetry Collector -> Grafana Cloud
     Alerting: 3 Prometheus alert rules, Slack integration
   ```

---

### Phase 2: AUDIT -- Evaluate Coverage and Quality

1. **Audit logging quality.** Evaluate each logger usage for:
   - **Structured format:** JSON output with consistent field names, not string concatenation
   - **Log levels:** Appropriate use of error, warn, info, debug (not everything at info)
   - **Context fields:** Request ID, user ID, operation name included in log entries
   - **Correlation IDs:** Trace ID propagated through log entries for cross-service correlation
   - **Sensitive data:** No PII, credentials, or tokens logged
   - **Error logging:** Stack traces included for errors, not just error messages

2. **Audit metrics coverage.** Check for standard metrics:
   - **RED metrics:** Request rate, error rate, duration for every HTTP endpoint
   - **USE metrics:** Utilization, saturation, errors for system resources
   - **Custom business metrics:** Domain-specific counters and gauges
   - **Histogram buckets:** Appropriate bucket boundaries for latency histograms
   - **Label cardinality:** No high-cardinality labels (user ID, request ID as metric labels)

3. **Audit tracing implementation.** Verify:
   - **Span creation:** Entry points create root spans, downstream calls create child spans
   - **Context propagation:** Trace context flows across HTTP boundaries (W3C or B3 headers)
   - **Span attributes:** Meaningful attributes set on spans (not empty spans)
   - **Error recording:** Exceptions recorded on spans with status codes
   - **Sampling strategy:** Configured and appropriate (not sampling 100% in production)

4. **Audit alerting effectiveness.** Check each alert for:
   - **Actionability:** Does the alert tell the on-call engineer what to do?
   - **Signal quality:** Based on symptoms (error rate) not causes (CPU usage)
   - **Thresholds:** Based on SLO burn rate, not arbitrary values
   - **Runbook link:** Each alert has a link to a runbook or troubleshooting guide
   - **Routing:** Alerts route to the correct team and escalation path

5. **Score each pillar and identify gaps:**

   ```
   Observability Audit:
     Logging:  7/10 -- structured, but missing correlation IDs in 4 services
     Metrics:  5/10 -- RED metrics partial, no business metrics
     Tracing:  8/10 -- good coverage, sampling needs tuning
     Alerting: 3/10 -- only 3 rules, no SLO-based alerts, no runbooks
   ```

---

### Phase 3: DESIGN -- Recommend Instrumentation Strategy

1. **Design logging strategy.** Recommend:
   - Standard log format with required fields (timestamp, level, service, traceId, message)
   - Logger factory that injects correlation context automatically
   - Log level configuration per environment (debug in dev, info in production)
   - Log aggregation pipeline (application -> collector -> storage -> query)
   - Retention policy per environment

2. **Design metrics strategy.** Recommend:
   - Standard RED metrics for every HTTP endpoint using middleware
   - Custom business metrics aligned with product KPIs
   - Histogram bucket configuration based on expected latency distribution
   - Metric naming convention following Prometheus best practices (`http_request_duration_seconds`)
   - Dashboard templates for standard service views

3. **Design tracing strategy.** Recommend:
   - Automatic instrumentation for HTTP, database, and message queue operations
   - Manual span creation for business-critical code paths
   - Sampling strategy (head-based for development, tail-based for production)
   - Baggage propagation for cross-cutting concerns (tenant ID, feature flags)
   - Trace-to-log correlation via trace ID injection

4. **Define SLIs and SLOs.** For each service endpoint:
   - **Availability SLI:** Proportion of successful requests (non-5xx)
   - **Latency SLI:** Proportion of requests faster than threshold (p99 < 500ms)
   - **SLO target:** 99.9% availability, 99% latency within budget
   - **Error budget:** Calculation and burn rate alerting
   - **SLO dashboard:** Visual budget remaining over rolling window

5. **Design alerting strategy.** Recommend:
   - Multi-window, multi-burn-rate alerts based on SLOs
   - Page-worthy alerts (high burn rate) vs. ticket-worthy alerts (slow burn)
   - Alert grouping to reduce noise (one alert per service, not per instance)
   - Escalation paths with timeouts
   - Regular alert review cadence (monthly) to retire noisy alerts

---

### Phase 4: VALIDATE -- Verify Instrumentation Correctness

1. **Validate log output.** Check that logs from a test run:
   - Parse as valid JSON (if structured logging is configured)
   - Contain required fields (timestamp, level, service name)
   - Include trace IDs when tracing is active
   - Do not contain sensitive data (grep for common patterns: password, token, secret)
   - Use appropriate log levels (no error-level logs for normal operations)

2. **Validate metric exposition.** Verify:
   - Metrics endpoint (`/metrics`) is accessible and returns Prometheus format
   - All declared custom metrics appear in the output
   - Histogram buckets are populated with reasonable values
   - No high-cardinality label combinations (check label count)
   - Counter values increment correctly (not resetting unexpectedly)

3. **Validate trace propagation.** Verify end-to-end:
   - A request to the entry service generates a trace visible in the tracing backend
   - Child spans appear for downstream service calls
   - Span attributes are populated with expected values
   - Error spans have appropriate status and exception recording
   - Sampling is working as configured (not dropping all traces or keeping all)

4. **Validate alerting rules.** Check:
   - Alert expressions are syntactically valid (PromQL, LogQL)
   - Alert thresholds fire on historical data that represents real incidents
   - Alert routing sends to the correct channel or pager
   - Runbook links resolve to actual documentation

5. **Generate observability report:**

   ```
   Observability Validation: [PASS/WARN/FAIL]

   Logging: PASS (structured, correlated, no PII detected)
   Metrics: WARN (RED metrics present, missing business metrics)
   Tracing: PASS (propagation verified, sampling at 10%)
   Alerting: FAIL (no SLO-based alerts, 2 of 3 rules missing runbooks)

   Priority actions:
     1. Define SLOs for /api/orders and /api/payments endpoints
     2. Add multi-burn-rate alerts based on SLO error budget
     3. Write runbooks for existing alerts
     4. Add order_total and payment_success_rate business metrics
   ```

---

## Harness Integration

- **`harness skill run harness-observability`** -- Primary invocation for observability audit.
- **`harness validate`** -- Run after instrumentation changes to verify project health.
- **`harness check-deps`** -- Verify observability library dependencies are installed.
- **`emit_interaction`** -- Present audit results and SLO design recommendations.

## Success Criteria

- All three observability pillars (logging, metrics, tracing) are assessed
- Coverage gaps are identified with specific remediation recommendations
- SLI/SLO definitions are provided for critical service endpoints
- Alerting strategy is evaluated for actionability and signal quality
- No sensitive data detected in log output
- Correlation between logs and traces is verified

## Examples

### Example: Express.js API with OpenTelemetry

```
Phase 1: DETECT
  Logging: pino with pino-http middleware
  Metrics: @opentelemetry/sdk-metrics -> Prometheus
  Tracing: @opentelemetry/sdk-trace-node -> Jaeger
  Collector: OTel Collector (otel-collector-config.yaml)
  Alerting: 5 Prometheus rules in monitoring/alerts.yml

Phase 2: AUDIT
  Logging: 8/10 -- structured JSON, trace IDs present, missing request body size
  Metrics: 6/10 -- http_request_duration_seconds present, missing queue depth
             and business metrics (orders_created_total)
  Tracing: 9/10 -- auto-instrumented HTTP + pg + Redis, manual spans on
             checkout flow
  Alerting: 4/10 -- static thresholds, no SLO burn rate, 2 missing runbooks

Phase 3: DESIGN
  SLOs recommended:
    - POST /api/orders: 99.9% availability, p99 < 800ms
    - GET /api/products: 99.95% availability, p99 < 200ms
  Alerting: Replace static "error rate > 5%" with multi-window burn rate
  Metrics: Add orders_created_total, cart_abandonment_rate gauges
  Logging: Add request/response body size for capacity planning

Phase 4: VALIDATE
  Log output: PASS (valid JSON, no PII)
  Metrics endpoint: PASS (all custom metrics present)
  Trace propagation: PASS (end-to-end verified)
  Alert rules: WARN (valid PromQL, but thresholds not SLO-based)
  Result: WARN -- alerting strategy needs SLO alignment
```

### Example: Go Microservices with Datadog

```
Phase 1: DETECT
  Logging: zap (structured) across 4 services
  Metrics: Datadog dogstatsd client
  Tracing: dd-trace-go with automatic HTTP/gRPC instrumentation
  Collector: Datadog Agent (datadog.yaml in k8s/)
  Alerting: 12 monitors in Datadog (Terraform-managed)

Phase 2: AUDIT
  Logging: 9/10 -- consistent structured format, correlation IDs, no PII
  Metrics: 7/10 -- RED metrics present, custom counters for business events,
             but missing histogram for gRPC call duration
  Tracing: 8/10 -- HTTP and gRPC instrumented, database spans present,
             Redis spans missing
  Alerting: 6/10 -- good coverage but static thresholds, no error budgets

Phase 3: DESIGN
  1. Add dd-trace-go Redis integration for complete trace picture
  2. Add grpc_server_handling_seconds histogram
  3. Define SLOs in Datadog for top 5 endpoints
  4. Convert 4 highest-priority monitors to SLO burn rate alerts
  5. Add Datadog SLO dashboard for team visibility

Phase 4: VALIDATE
  Log output: PASS
  Metrics: WARN (missing gRPC histogram)
  Traces: WARN (Redis spans missing)
  Alerts: WARN (no SLO-based alerts)
  Result: WARN -- 3 instrumentation gaps, alerting needs SLO alignment
```

## Rationalizations to Reject

| Rationalization                                                                        | Reality                                                                                                                                                                                                                                                                                                     |
| -------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| "We can see what's happening in CloudWatch logs — we don't need structured logging"    | Unstructured log lines cannot be queried, aggregated, or correlated across services. When an incident spans three services, searching for a request ID across unstructured logs is manual forensics. Structured logging is not a nicety — it is the foundation for incident response.                       |
| "We'll add alerting once we've seen a few incidents and know what to alert on"         | The first incident is the worst time to define alerting. SLO-based burn rate alerts can be defined from traffic patterns before any incidents occur. Waiting for incidents to define thresholds means every early failure goes undetected.                                                                  |
| "User ID is a useful label for the latency metric — it helps us debug per-user issues" | User ID as a metric label creates one time series per user, which at 100,000 users means 100,000 label combinations. High-cardinality labels exhaust metric storage, cause query timeouts, and make the entire metrics system unstable. Use logs for per-user debugging; use metrics for aggregate signals. |
| "The tracing library is initialized, so we have distributed tracing"                   | Initializing the library creates root spans but does not propagate context across HTTP boundaries, instrument database calls, or connect traces to logs. Trace initialization without verified end-to-end propagation produces disconnected, useless traces.                                                |
| "We have alerts — they're just not linked to runbooks yet"                             | An alert that fires at 3am without a runbook link requires the on-call engineer to start debugging from scratch. The absence of a runbook is not a documentation gap; it is a mean-time-to-recover multiplier.                                                                                              |

## Gates

- **No sensitive data in logs.** If PII, credentials, or tokens are detected in log output, it is a blocking finding. The logging configuration must sanitize or redact sensitive fields before any other improvements are made.
- **No high-cardinality metric labels.** Metric labels with unbounded values (user IDs, request IDs, timestamps) cause storage explosion and query timeouts. This is a blocking finding.
- **No alerting without runbooks.** Production alerts that lack a runbook link are incomplete. Every page-worthy alert must have actionable documentation.
- **No tracing without context propagation.** Traces that do not propagate context across service boundaries provide incomplete pictures and mislead investigations. Broken propagation is a blocking finding.

## Escalation

- **When observability libraries conflict:** Some combinations (e.g., dd-trace and OTel auto-instrumentation) cause duplicate spans or metric conflicts. Recommend choosing one provider and removing the other. If migration is needed, present a phased approach.
- **When sampling drops critical traces:** If tail-based sampling is misconfigured, critical error traces may be dropped. Recommend always-sample rules for error responses and critical code paths, with probabilistic sampling for normal traffic.
- **When metric cardinality is already out of control:** Do not attempt to fix retroactively in the metrics backend. Recommend adding new metrics with correct labels and deprecating the high-cardinality ones. Set a timeline for removal.
- **When no observability infrastructure exists:** This is a design-from-scratch scenario. Start with structured logging (lowest barrier), then add metrics (RED for HTTP endpoints), then tracing. Do not attempt all three at once.