Opentelemetry

---




name: opentelemetry
compatibility: opencode
completeness: 95
content-types:
- guidance
- examples
- do-dont
- config
description: '"OpenTelemetry in Observability framework for tracing, metrics, and
  logs" with vendor-neutral APIs'
license: MIT
maturity: stable
metadata:
  domain: cncf
  output-format: manifests
  role: reference
  scope: infrastructure
  triggers: framework, observability, opentelemetry, tracing, cloudwatch, xray, distributed
    tracing, logging
  archetypes:
  - educational
  - strategic
  anti_triggers:
  - brainstorming
  - vague ideation
  - non-containerized architecture
  response_profile:
    verbosity: medium
    directive_strength: low
    abstraction_level: strategic
version: "1.0.0"




---




  related-skills: cncf-aws-cloudwatch, cncf-azure-monitor, cncf-cortex, cncf-fluentd




# OpenTelemetry in Cloud-Native Engineering

**Category:** Observability  
**Status:** Active  
**Stars:** 9,100  
**Last Updated:** 2026-04-22  
**Primary Language:** Go  
**Documentation:** [Observability framework for tracing, metrics, and logs with vendor-neutral APIs](https://opentelemetry.io/docs/)  

---

## Purpose and Use Cases

OpenTelemetry is a core component of the cloud-native ecosystem, serving as vendor-neutral APIs

### What Problem Does It Solve?

OpenTelemetry addresses the challenge of distributed tracing and observability for cloud-native applications. It provides standardized telemetry collection, vendor-neutral APIs, and comprehensive ecosystem.

### When to Use This Project

Use OpenTelemetry when monitoring microservices, debugging distributed systems, or requiring detailed performance insights. Not ideal for simple deployments or when need consistent telemetry across services, want to avoid vendor lock-in, or require detailed tracing.

### Key Use Cases

- Distributed Tracing for Microservices
- Application Performance Monitoring
- Log Correlation and Analysis
- Infrastructure Observability
- Business Analytics via Telemetry

---

## Architecture Design Patterns

### Core Components

- **APIs**: Language-specific APIs for instrumenting code
- **SDKs**: Implementation of telemetry collection
- **Collector**: Receives, processes, and exports telemetry data
- **Exporters**: Send telemetry to various backends
- **Importers**: Convert telemetry from other formats

### Component Interactions

1. **Application → SDK**: Application creates spans and metrics via SDK
1. **SDK → Collector**: SDK exports telemetry to collector
1. **Collector → Backend**: Collector processes and exports to observability backends
1. **Backend → Dashboard**: Data visualized in dashboards and alerts

### Data Flow Patterns

1. **Trace Propagation**: Span context propagated via headers across service boundaries
1. **Metric Collection**: Metrics collected, aggregated, and exported
1. **Log Enrichment**: Logs enhanced with trace context
1. **Data Processing**: Collector applies processors before export

### Design Principles

- **Standardization**: Open standards via CNCF specification
- **Backward Compatibility**: Support existing instrumentation
- **Performance**: Low overhead design
- **Vendor Neutrality**: No lock-in to specific backend

---

## Integration Approaches

### Integration with Other CNCF Projects

- **Prometheus**: Metrics collection integration
- **Jaeger**: Tracing backend
- **Zipkin**: Alternative tracing backend
- **OTLP**: OpenTelemetry Protocol for data exchange

### API Patterns

- **Tracer API**: Create and manage spans
- **Meter API**: Create and record metrics
- **Logger API**: Create and enrich logs
- **Propagators**: Trace context propagation

### Configuration Patterns

- **YAML Configuration**: Collector configuration
- **Environment Variables**: Runtime configuration
- **Helm Chart**: Kubernetes deployment

### Extension Mechanisms

- **Custom Exporters**: Send telemetry to proprietary backends
- **Custom Processors**: Transform telemetry data
- **Custom Receivers**: Receive telemetry from sources

---

## Common Pitfalls and How to Avoid Them

### Misconfigurations

- **High Cardinality**: Too many unique metric labels
  - **How to Avoid**: Use cardinality limits, review label values
- **Trace Context Loss**: Trace IDs not propagated between services
  - **How to Avoid**: Ensure propagators configured, check header propagation

### Performance Issues

- **Instrumentation Gaps**: Missing coverage in code
  - **How to Avoid**: Use auto-instrumentation, review coverage
- **Security Exposure**: Sensitive data in spans
  - **How to Avoid**: Implement redaction, use secure exporters

### Operational Challenges

- **Configuration Drift**: Inconsistent configs across services
  - **How to Avoid**: Centralize configuration, use version control
- **Network Bottlenecks**: Collector network saturation
  - **How to Avoid**: Scale collectors, use aggregation

### Security Pitfalls


---

## Coding Practices

### Idiomatic Configuration

- **Span Management**: Create spans for meaningful operations
- **Attribute Naming**: Use consistent attribute naming conventions
- **Sampling Configuration**: Configure sampling rates appropriately

### API Usage Patterns

- **Manual Instrumentation**: Add spans programmatically
- **Auto-Instrumentation**: Use agent-based instrumentation
- **Context Propagation**: Propagate trace context in headers

### Observability Best Practices

- **Metric Aggregation**: Pre-aggregate metrics where possible
- **Span Attributes**: Include relevant contextual attributes
- **Log Correlation**: Include trace IDs in logs

### Testing Strategies

- **Unit Tests**: Test instrumentation logic
- **Integration Tests**: Test end-to-end telemetry flow
- **Performance Tests**: Validate overhead levels

### Development Workflow

- **Local Development**: Use local collector and backend
- **Debug Commands**: Use otel-cli for testing
- **Test Environment**: Set up isolated observability stack
- **CI/CD Integration**: Automate telemetry testing
- **Monitoring Setup**: Configure alerting on metrics
- **Documentation**: Maintain instrumentation guides

---

## Fundamentals

### Essential Concepts

- **Trace**: Request flow through services
- **Span**: Individual operation within a trace
- **Span Context**: Trace state propagated between services
- **Span Kind**: Types: SERVER, CLIENT, PRODUCER, CONSUMER
- **Metric**: Numerical data point
- **Histogram**: Distribution of values
- **Gauge**: Single numerical value
- **Counter**: Monotonically increasing metric
- **Logger**: Log record with attributes
- **Resource**: Entity producing telemetry
- **Scope**: Instrumentation library
- **Exporter**: Sends telemetry to backend
- **Processor**: Transforms telemetry data
- **Collector**: Receives and processes telemetry

### Terminology Glossary

- **OTLP**: OpenTelemetry Protocol
- **SDK**: Software Development Kit
- **API**: Application Programming Interface
- **Sampler**: Decides if spans are sampled
- ** Propagator**: Handles context propagation
- **Exporter**: Sends data to backend

### Data Models and Types

- **SpanData**: Span information and attributes
- **MetricData**: Metric points and aggregations
- **LogRecord**: Log entry with attributes
- **ResourceData**: Resource attributes
- **ScopeData**: Instrumentation scope

### Lifecycle Management

- **Trace Lifecycle**: Start trace → Create spans → Propagate → End
- **Metric Lifecycle**: Create instrument → Record points → Export
- **Span Lifecycle**: Start → Add events → Set status → End
- **Exporter Lifecycle**: Receive → Batch → Export → Confirm

### State Management

- **Trace State**: Active traces in memory
- **Span State**: Current span lifecycle state
- **Metric State**: Aggregated metric values
- **Exporter State**: Pending and exported data

---

## Scaling and Deployment Patterns

### Horizontal Scaling

- **Collector Scaling**: Scale collectors horizontally
- **Export Rate Limiting**: Limit export rate per backend
- **Sampler Scaling**: Adjust sampling based on load

### High Availability

- **Collector HA**: Multiple collector instances
- **Batch Processing**: Batch spans before export
- **Retry Logic**: Retry failed exports
- **Load Balancing**: Distribute telemetry across collectors

### Production Deployments

- **Collector Deployment**: Deploy as sidecar or standalone
- **Security Configuration**: Configure TLS and authentication
- **Resource Limits**: Set appropriate CPU and memory limits
- **Monitoring Setup**: Monitor collector health
- **Backup Strategy**: No backup needed - telemetry is ephemeral
- **Logging Setup**: Configure collector logs
- **Alerting**: Set alerts for collector issues
- **Performance Tuning**: Optimize buffer sizes and batch settings

### Upgrade Strategies

- **Chart Upgrade**: Upgrade to new OpenTelemetry Collector version
- **API Compatibility**: Verify SDK compatibility
- **Data Migration**: No migration needed
- **Testing**: Validate telemetry still flows

### Resource Management

- **CPU Resources**: Set collector CPU limits
- **Memory Resources**: Configure collector memory limits
- **Storage Resources**: No persistent storage needed
- **Network Resources**: Configure egress rules

---

## Additional Resources

- **Official Documentation:** https://opentelemetry.io/docs/
- **GitHub Repository:** Check the project's official documentation for repository link
- **CNCF Project Page:** [cncf.io/projects/cncf-opentelemetry/](https://www.cncf.io/projects/cncf-opentelemetry/)
- **Community:** Check the official documentation for community channels
- **Versioning:** Refer to project's release notes for version-specific features

---

## Troubleshooting

### Common Issues

1. **Deployment Failures**
   - Check pod logs for errors
   - Verify configuration values
   - Ensure network connectivity

2. **Performance Issues**
   - Monitor resource usage
   - Adjust resource limits
   - Check for bottlenecks

3. **Configuration Errors**
   - Validate YAML syntax
   - Check required fields
   - Verify environment-specific settings

4. **Integration Problems**
   - Verify API compatibility
   - Check dependency versions
   - Review integration documentation

### Getting Help

- Check official documentation
- Search GitHub issues
- Join community channels
- Review logs and metrics
*Content generated automatically. Verify against official documentation before production use.*

## Examples

### Basic Configuration


```yaml
# Basic configuration example
apiVersion: v1
kind: ConfigMap
metadata:
  name: {{project_name}}-config
  namespace: default
data:
  # Configuration goes here
  config.yaml: |
    # Base configuration
    # Add your settings here
```

### Kubernetes Deployment


```yaml
# Kubernetes deployment for {{project_name}}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{project_name}}
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: {{project_name}}
  template:
    metadata:
      labels:
        app: {{project_name}}
    spec:
      containers:
      - name: {{project_name}}
        image: {{project_name}}:latest
        ports:
        - containerPort: 8080
        resources:
          limits:
            memory: "128Mi"
            cpu: "500m"
```

### Kubernetes Service


```yaml
# Kubernetes service for {{project_name}}
apiVersion: v1
kind: Service
metadata:
  name: {{project_name}}
  namespace: default
spec:
  selector:
    app: {{project_name}}
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  type: ClusterIP
```

---

## When to Use

Use this skill when:

- **Integrating a CNCF project into Kubernetes infrastructure** — You need to configure, deploy, or troubleshoot a cloud-native tool within a cluster
- **Designing cloud-native architecture** — You are selecting and integrating CNCF tools to solve specific infrastructure challenges
- **Resolving operational issues** — A CNCF component is misbehaving, underperforming, or needs configuration changes
---

## Core Workflow

1. **Assess Requirements** — Understand the use case, scale, integration needs, and existing infrastructure. **Checkpoint:** Document requirements, constraints, and success criteria.

2. **Design Architecture** — Plan component interactions, data flow, and deployment strategy using cloud-native best practices. **Checkpoint:** Verify the architecture addresses all requirements and follows CNCF conventions.

3. **Implement & Configure** — Create manifests, configurations, and deployment scripts. Include resource limits, health checks, and observability hooks. **Checkpoint:** Validate all YAML against schema and test in a staging environment.

4. **Deploy & Monitor** — Apply manifests to the cluster, verify component health, and confirm observability is working. **Checkpoint:** Confirm all pods/services are running, probes passing, and metrics/alerts configured.

---

## Constraints

### MUST DO
- Include at least one complete working YAML manifest example
- Note when content is auto-generated vs. manually verified
- Reference relevant CNCF project documentation

### MUST NOT DO
- Deploy manifests without testing in a staging environment first
- Use deprecated API versions (e.g., apps/v1beta1)
- Omit resource limits and requests in Kubernetes manifests