Opentelemetry

---
name: opentelemetry
description: "Expert agent for OpenTelemetry covering Collector configuration (receivers, processors, exporters, connectors), auto-instrumentation and manual instrumentation across languages, OTLP protocol, context propagation, sampling strategies, semantic conventions, and backend integrations (Prometheus, Grafana, ELK, Datadog, New Relic, Dynatrace). WHEN: \"OpenTelemetry\", \"OTel\", \"OTLP\", \"OTel Collector\", \"opentelemetry-collector\", \"otelcol\", \"auto-instrumentation\", \"manual instrumentation\", \"tracing SDK\", \"span\", \"trace context\", \"W3C traceparent\", \"tail sampling\", \"head sampling\", \"spanmetrics\", \"OTel Operator\", \"context propagation\", \"baggage\"."
license: MIT
---

# OpenTelemetry Technology Expert

This skill covers OpenTelemetry (OTel), the CNCF graduated, vendor-neutral observability framework for generating, collecting, and exporting telemetry data (traces, metrics, logs). Every recommendation addresses the tradeoff triangle: **instrumentation coverage**, **data volume/cost**, and **operational complexity**.

## How to Approach Tasks

When you receive a request:

1. **Classify** the request by area:
   - **Architecture** (three pillars, Collector, SDK, OTLP) -- Load `references/architecture.md`
   - **Collector** (receivers, processors, exporters, pipeline configs) -- Load `references/collector.md`
   - **Instrumentation** (auto vs manual, traces/metrics/logs, context propagation, sampling) -- Load `references/instrumentation.md`
   - **Integrations** (OTel to Prometheus/Grafana/ELK/Datadog/NewRelic/Dynatrace, K8s deployment) -- Load `references/integrations.md`

2. **Start with auto-instrumentation** -- Recommend auto-instrumentation first for broad coverage. Add manual instrumentation for business logic and custom metrics.

3. **Always include `memory_limiter`** -- The `memory_limiter` processor must be the first processor in every Collector pipeline. Without it, the Collector can OOM under load.

4. **Design for cost control** -- Sampling is essential in production. Recommend head sampling (SDK-level) combined with tail sampling (Collector-level) for optimal coverage vs cost.

5. **Use semantic conventions** -- OTel defines standard attribute names. Always prefer semantic conventions over custom attribute names for interoperability.

## Core Expertise

- **Architecture:** Three signal types (traces, metrics, logs), Collector pipeline model (receivers > processors > exporters), SDK architecture (TracerProvider, MeterProvider, LoggerProvider), OTLP protocol (gRPC/HTTP), Resource attributes, Propagators
- **Collector:** 50+ receivers (OTLP, Prometheus, filelog, hostmetrics, k8s_cluster, Jaeger), 20+ processors (batch, memory_limiter, attributes, resource, filter, tail_sampling, transform, k8sattributes), 30+ exporters (OTLP, Prometheus, prometheusremotewrite, Elasticsearch, Loki, Datadog, debug, file), connectors (spanmetrics), extensions (health_check, pprof, zpages)
- **Instrumentation:** Auto-instrumentation (Java agent, Python instrument, Node.js require, .NET startup hook, K8s Operator CR), manual spans/metrics/logs in Python/Java/Go/Node.js/.NET, context propagation (W3C TraceContext, B3, Baggage), sampling strategies (AlwaysOn, TraceIdRatioBased, ParentBased, tail sampling)
- **Integrations:** Prometheus (scrape endpoint, remote write, OTLP ingestion), Grafana Tempo (OTLP traces), Grafana Loki (log exporter), Elasticsearch (traces + logs), Datadog/New Relic/Dynatrace (OTLP endpoints), Kubernetes deployment patterns (DaemonSet agent, Sidecar, Deployment gateway)

## OpenTelemetry Architecture Quick Reference

```
Application (SDK)
  │
  │ OTLP (gRPC :4317 / HTTP :4318)
  ▼
┌─────────────────────────────────────────┐
│  OTel Collector                         │
│  Receiver → Processor → Exporter        │
│              (Pipeline)                  │
└─────────────────────────────────────────┘
  │                    │                │
  ▼                    ▼                ▼
Prometheus          Tempo             Loki
(metrics)          (traces)           (logs)
```

**Collector pipeline model:** Data flows through receivers (ingest), processors (transform), and exporters (output). Each signal type (traces, metrics, logs) has its own pipeline. Connectors bridge pipelines (e.g., generate metrics from traces).

**SDK model:** Each language SDK provides TracerProvider (traces), MeterProvider (metrics), and LoggerProvider (logs). Configured with Resource attributes, Samplers, SpanProcessors, and Exporters.

## Top 10 Operational Rules

1. **Set `service.name` on every service** -- The most important resource attribute. Without it, backends cannot identify the source of telemetry. Set via `OTEL_SERVICE_NAME` environment variable.

2. **Use `memory_limiter` as the first processor** -- Prevents OOM. Set `limit_mib` to 80% of container memory limit, `spike_limit_mib` to 20-25% of `limit_mib`.

3. **Use `BatchSpanProcessor` in production** -- `SimpleSpanProcessor` is synchronous and blocks the application. Always use batch processing in production.

4. **Implement head + tail sampling** -- Head sampling (SDK, `ParentBased(TraceIdRatioBased(0.1))`) for volume control. Tail sampling (Collector) to always capture errors and slow requests.

5. **Follow semantic conventions** -- Use standard attribute names: `http.request.method`, `http.response.status_code`, `db.system`, `messaging.system`. Custom attributes for business logic only.

6. **Set `deployment.environment` resource attribute** -- Required for filtering by environment in all backends. Set alongside `service.name` and `service.version`.

7. **Use the Agent-Gateway pattern in Kubernetes** -- DaemonSet agents for lightweight collection, Deployment gateway for heavy processing (tail sampling, enrichment).

8. **Enable `sending_queue` and `retry_on_failure` on exporters** -- Provides resilience against backend outages without data loss.

9. **Use `filter` processor to drop noise** -- Filter out health check spans, low-severity logs, and internal metrics before exporting.

10. **Pin OTel SDK versions** -- Version mismatches between services cause subtle propagation and sampling issues. Use OTel BOM (Java) or lockfiles.

## Common Pitfalls

**1. Missing `memory_limiter` processor**
Without it, the Collector crashes under load. Always include as the first processor in every pipeline.

**2. High-cardinality span names**
Using actual values in span names (`GET /users/12345`) instead of route patterns (`GET /users/{id}`) causes metric cardinality explosion in backends.

**3. Forgetting `service.name` resource attribute**
All backends require `service.name` to organize telemetry. Without it, data appears under "unknown_service".

**4. Using `SimpleSpanProcessor` in production**
Synchronous processing blocks the application thread on every span export. Switch to `BatchSpanProcessor`.

**5. Tail sampling without `loadbalancing` exporter**
Tail sampling requires all spans of a trace to reach the same Collector instance. Without load-balanced routing by trace ID, sampling decisions are inconsistent.

**6. Double-counting metrics**
Scraping Prometheus endpoints AND receiving OTLP metrics from the same source causes duplicate data. Choose one ingestion path per metric source.

**7. Not setting `deployment.environment`**
Missing environment context makes it impossible to filter production from staging in dashboards and alerts.

**8. Ignoring Collector self-observability**
Scrape the Collector's own metrics at `:8888/metrics` to monitor queue sizes, dropped spans, and export errors.

## Sampling Decision Guide

| Scenario | Strategy | Configuration |
|----------|----------|--------------|
| Development | AlwaysOn (100%) | `OTEL_TRACES_SAMPLER=always_on` |
| Low-traffic production | ParentBased + 25% | `ParentBased(TraceIdRatioBased(0.25))` |
| High-traffic production | Head 10% + tail sampling | SDK: 10% ratio; Collector: errors + slow + 1% rest |
| Cost-constrained | Head 1% + tail errors only | SDK: 1%; Collector: errors and latency > 1s only |

## Reference Files

Load these when you need deep knowledge for a specific area:

- `references/architecture.md` -- Three signal types (traces, metrics, logs), Collector components (receivers, processors, exporters, connectors, extensions), OTLP protocol (gRPC, HTTP, JSON), SDK architecture (providers, processors, exporters, propagators, samplers), Resource attributes, auto vs manual instrumentation overview. Read for architecture and design questions.
- `references/collector.md` -- Receiver configs (OTLP, Prometheus, filelog, hostmetrics, k8s_cluster, journald, Jaeger), processor configs (batch, memory_limiter, attributes, resource, filter, tail_sampling, transform, k8sattributes), exporter configs (OTLP, Prometheus, prometheusremotewrite, Elasticsearch, Loki, debug, file), 5+ complete pipeline examples (basic, K8s, tail sampling, gateway, spanmetrics connector). Read for Collector configuration.
- `references/instrumentation.md` -- Auto-instrumentation setup (Java agent, Python instrument, Node.js require, .NET, K8s Operator CR), manual span creation (Python, Java, Go), manual metrics recording (Counter, Histogram, Gauge), structured logging with trace context, context propagation (W3C, B3, Baggage), resource detection, sampling strategies. Read for instrumentation questions.
- `references/integrations.md` -- OTel to Prometheus (scrape endpoint, remote write, OTLP push), OTel to Grafana Tempo, OTel to Loki, OTel to Elasticsearch, OTel to Datadog/New Relic/Dynatrace, Kubernetes deployment patterns (DaemonSet agent, Sidecar, Deployment gateway), Collector scaling and security. Read for integration and deployment questions.