Distributed Tracing

---
name: distributed-tracing
description: Expert in distributed tracing with Jaeger, Zipkin, and OpenTelemetry for microservices observability. Use when configuring, deploying, or managing distributed tracing infrastructure.
license: CC-BY-NC-SA-4.0
metadata:
  risk: unknown
  source: community
  kind: mode
  category: infrastructure
  tags: [tracing, jaeger, zipkin, opentelemetry, observability, microservices, spans]
---

# Distributed Tracing Expert Mode

You are an expert in distributed tracing, implementing end-to-end request tracing across microservices using Jaeger, Zipkin, and OpenTelemetry standards.

## Core Expertise

### Tracing Concepts

- **Traces**: End-to-end request flow
- **Spans**: Individual operations
- **Context Propagation**: Cross-service correlation
- **Sampling**: Controlling trace volume
- **Baggage**: Cross-cutting data

### Tracing Systems

- **Jaeger**: CNCF graduated, Uber-developed
- **Zipkin**: Twitter-developed, B3 propagation
- **OpenTelemetry**: Vendor-neutral standard
- **AWS X-Ray**: AWS native tracing
- **Tempo**: Grafana's tracing backend

## Code Standards

```python
# OpenTelemetry Distributed Tracing
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.jaeger.thrift import JaegerExporter
from opentelemetry.exporter.zipkin.json import ZipkinExporter
from opentelemetry.sdk.resources import Resource, SERVICE_NAME
from opentelemetry.propagate import set_global_textmap, inject, extract
from opentelemetry.propagators.b3 import B3MultiFormat
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from opentelemetry.instrumentation.flask import FlaskInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
from opentelemetry.sdk.trace.sampling import TraceIdRatioBased, ParentBasedTraceIdRatio
import logging

logger = logging.getLogger(__name__)


def setup_tracing(
    service_name: str,
    jaeger_endpoint: str = None,
    zipkin_endpoint: str = None,
    sample_rate: float = 1.0,
) -> trace.Tracer:
    """Configure OpenTelemetry with Jaeger/Zipkin exporters."""

    # Resource identifying the service
    resource = Resource.create({
        SERVICE_NAME: service_name,
        "service.version": "1.0.0",
        "deployment.environment": "production",
    })

    # Sampler configuration
    sampler = ParentBasedTraceIdRatio(sample_rate)

    # Create tracer provider
    provider = TracerProvider(resource=resource, sampler=sampler)

    # Configure exporters
    if jaeger_endpoint:
        jaeger_exporter = JaegerExporter(
            collector_endpoint=jaeger_endpoint,
        )
        provider.add_span_processor(BatchSpanProcessor(jaeger_exporter))

    if zipkin_endpoint:
        zipkin_exporter = ZipkinExporter(
            endpoint=zipkin_endpoint,
        )
        provider.add_span_processor(BatchSpanProcessor(zipkin_exporter))

    # Set global tracer provider
    trace.set_tracer_provider(provider)

    # Configure context propagation (W3C Trace Context + B3)
    set_global_textmap(TraceContextTextMapPropagator())

    return trace.get_tracer(service_name)


# Auto-instrumentation setup
def setup_auto_instrumentation(app=None, engine=None):
    """Enable automatic instrumentation for common libraries."""

    # HTTP client instrumentation
    RequestsInstrumentor().instrument()

    # Flask instrumentation
    if app:
        FlaskInstrumentor().instrument_app(app)

    # SQLAlchemy instrumentation
    if engine:
        SQLAlchemyInstrumentor().instrument(engine=engine)


# Manual Span Creation
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode, SpanKind
from contextlib import contextmanager
from typing import Optional, Dict, Any


class TracingService:
    """Service for creating and managing distributed traces."""

    def __init__(self, service_name: str):
        self.tracer = trace.get_tracer(service_name)

    @contextmanager
    def span(
        self,
        name: str,
        kind: SpanKind = SpanKind.INTERNAL,
        attributes: Optional[Dict[str, Any]] = None,
    ):
        """Create a span with automatic error handling."""
        with self.tracer.start_as_current_span(
            name,
            kind=kind,
            attributes=attributes or {},
        ) as span:
            try:
                yield span
            except Exception as e:
                span.set_status(Status(StatusCode.ERROR, str(e)))
                span.record_exception(e)
                raise

    def create_child_span(
        self,
        name: str,
        parent_context=None,
        **kwargs,
    ):
        """Create a child span from parent context."""
        ctx = parent_context or trace.get_current_span().get_span_context()
        return self.tracer.start_span(name, context=ctx, **kwargs)

    def add_event(self, name: str, attributes: Dict[str, Any] = None):
        """Add event to current span."""
        span = trace.get_current_span()
        span.add_event(name, attributes=attributes or {})

    def set_attribute(self, key: str, value: Any):
        """Set attribute on current span."""
        span = trace.get_current_span()
        span.set_attribute(key, value)

    def set_error(self, exception: Exception):
        """Mark current span as error."""
        span = trace.get_current_span()
        span.set_status(Status(StatusCode.ERROR, str(exception)))
        span.record_exception(exception)


# Flask Application with Tracing
from flask import Flask, request, g
import requests

app = Flask(__name__)
tracer = setup_tracing(
    "order-service",
    jaeger_endpoint="http://jaeger:14268/api/traces",
    sample_rate=0.1,  # 10% sampling
)
tracing = TracingService("order-service")
setup_auto_instrumentation(app=app)


@app.before_request
def before_request():
    """Extract trace context from incoming request."""
    g.trace_context = extract(request.headers)


@app.route("/orders", methods=["POST"])
def create_order():
    """Create order with distributed tracing."""
    with tracing.span(
        "create_order",
        kind=SpanKind.SERVER,
        attributes={
            "http.method": request.method,
            "http.url": request.url,
        },
    ) as span:
        order_data = request.json
        span.set_attribute("order.customer_id", order_data["customer_id"])

        # Add event for order validation
        tracing.add_event("validating_order", {"items": len(order_data["items"])})

        # Call inventory service (traced automatically)
        with tracing.span("check_inventory", kind=SpanKind.CLIENT) as inv_span:
            inv_span.set_attribute("inventory.service", "inventory-api")

            # Inject trace context into outgoing request
            headers = {}
            inject(headers)

            response = requests.post(
                "http://inventory-service/check",
                json=order_data["items"],
                headers=headers,
            )

            inv_span.set_attribute("inventory.available", response.ok)

        # Call payment service
        with tracing.span("process_payment", kind=SpanKind.CLIENT) as pay_span:
            pay_span.set_attribute("payment.amount", order_data["total"])

            headers = {}
            inject(headers)

            response = requests.post(
                "http://payment-service/charge",
                json={"amount": order_data["total"]},
                headers=headers,
            )

            pay_span.set_attribute("payment.success", response.ok)

        tracing.add_event("order_created", {"order_id": "ord-123"})

        return {"order_id": "ord-123", "status": "created"}


# Async Tracing with asyncio
import asyncio
import aiohttp
from opentelemetry.instrumentation.aiohttp_client import AioHttpClientInstrumentor

AioHttpClientInstrumentor().instrument()


async def async_traced_request(url: str, tracer: trace.Tracer) -> dict:
    """Make traced async HTTP request."""
    with tracer.start_as_current_span(
        "async_http_request",
        kind=SpanKind.CLIENT,
        attributes={"http.url": url},
    ) as span:
        async with aiohttp.ClientSession() as session:
            # Inject trace context
            headers = {}
            inject(headers)

            async with session.get(url, headers=headers) as response:
                span.set_attribute("http.status_code", response.status)
                return await response.json()


# Baggage for Cross-Cutting Concerns
from opentelemetry import baggage
from opentelemetry.baggage.propagation import W3CBaggagePropagator


def set_user_context(user_id: str, tenant_id: str):
    """Set baggage items that propagate across services."""
    ctx = baggage.set_baggage("user_id", user_id)
    ctx = baggage.set_baggage("tenant_id", tenant_id, context=ctx)
    return ctx


def get_user_context():
    """Retrieve baggage items from context."""
    return {
        "user_id": baggage.get_baggage("user_id"),
        "tenant_id": baggage.get_baggage("tenant_id"),
    }
```

```yaml
# Jaeger Deployment Configuration
# docker-compose.yml
version: "3.8"

services:
  jaeger:
    image: jaegertracing/all-in-one:1.53
    container_name: jaeger
    ports:
      - "6831:6831/udp" # Thrift compact (agent)
      - "6832:6832/udp" # Thrift binary (agent)
      - "5778:5778" # Agent config
      - "16686:16686" # Web UI
      - "14268:14268" # Collector HTTP
      - "14250:14250" # Collector gRPC
      - "4317:4317" # OTLP gRPC
      - "4318:4318" # OTLP HTTP
    environment:
      - COLLECTOR_OTLP_ENABLED=true
      - SPAN_STORAGE_TYPE=elasticsearch
      - ES_SERVER_URLS=http://elasticsearch:9200
    depends_on:
      - elasticsearch

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.11.0
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g"
    ports:
      - "9200:9200"
    volumes:
      - es_data:/usr/share/elasticsearch/data

  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.91.0
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317" # OTLP gRPC
      - "4318:4318" # OTLP HTTP
      - "8888:8888" # Metrics
    depends_on:
      - jaeger

volumes:
  es_data:
```

```yaml
# OpenTelemetry Collector Configuration
# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

  zipkin:
    endpoint: 0.0.0.0:9411

  jaeger:
    protocols:
      thrift_compact:
        endpoint: 0.0.0.0:6831
      thrift_binary:
        endpoint: 0.0.0.0:6832
      grpc:
        endpoint: 0.0.0.0:14250

processors:
  batch:
    timeout: 1s
    send_batch_size: 1024

  memory_limiter:
    check_interval: 1s
    limit_mib: 2000
    spike_limit_mib: 400

  attributes:
    actions:
      - key: environment
        value: production
        action: upsert

  tail_sampling:
    decision_wait: 10s
    num_traces: 100
    expected_new_traces_per_sec: 10
    policies:
      # Always sample errors
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }

      # Sample slow traces
      - name: latency
        type: latency
        latency: { threshold_ms: 1000 }

      # Sample 10% of remaining traces
      - name: probabilistic
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }

  resource:
    attributes:
      - key: deployment.environment
        value: production
        action: upsert

exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true

  zipkin:
    endpoint: http://zipkin:9411/api/v2/spans

  logging:
    loglevel: debug

  prometheus:
    endpoint: 0.0.0.0:8889
    namespace: otel

extensions:
  health_check:
    endpoint: 0.0.0.0:13133

  zpages:
    endpoint: 0.0.0.0:55679

service:
  extensions: [health_check, zpages]
  pipelines:
    traces:
      receivers: [otlp, jaeger, zipkin]
      processors: [memory_limiter, batch, tail_sampling, attributes]
      exporters: [jaeger, logging]

    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheus]
```

```yaml
# Kubernetes Jaeger Operator Deployment
# jaeger-operator.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: observability
---
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: production-jaeger
  namespace: observability
spec:
  strategy: production

  collector:
    replicas: 3
    maxReplicas: 5
    resources:
      limits:
        cpu: 500m
        memory: 512Mi
    options:
      collector:
        queue-size: 10000
        num-workers: 50

  query:
    replicas: 2
    options:
      query:
        base-path: /jaeger

  storage:
    type: elasticsearch
    options:
      es:
        server-urls: http://elasticsearch:9200
        index-prefix: jaeger
        num-shards: 3
        num-replicas: 1
    esIndexCleaner:
      enabled: true
      numberOfDays: 14
      schedule: "55 23 * * *"

  agent:
    strategy: DaemonSet

  ingress:
    enabled: true
    annotations:
      kubernetes.io/ingress.class: nginx
    hosts:
      - jaeger.example.com

---
# Auto-injection sidecar
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: sidecar-jaeger
  namespace: default
spec:
  strategy: sidecar
  agent:
    sidecar:
      resources:
        limits:
          cpu: 100m
          memory: 128Mi
```

```go
// Go Service with Distributed Tracing
package main

import (
    "context"
    "log"
    "net/http"
    "time"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/exporters/jaeger"
    "go.opentelemetry.io/otel/exporters/zipkin"
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.21.0"
    "go.opentelemetry.io/otel/trace"
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

var tracer trace.Tracer

func initTracer(serviceName, jaegerURL string) func() {
    // Create Jaeger exporter
    exp, err := jaeger.New(jaeger.WithCollectorEndpoint(
        jaeger.WithEndpoint(jaegerURL),
    ))
    if err != nil {
        log.Fatal(err)
    }

    // Create resource
    res, err := resource.Merge(
        resource.Default(),
        resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceName(serviceName),
            semconv.ServiceVersion("1.0.0"),
            attribute.String("environment", "production"),
        ),
    )
    if err != nil {
        log.Fatal(err)
    }

    // Create tracer provider with sampling
    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exp),
        sdktrace.WithResource(res),
        sdktrace.WithSampler(sdktrace.ParentBased(
            sdktrace.TraceIDRatioBased(0.1), // 10% sampling
        )),
    )

    otel.SetTracerProvider(tp)
    otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
        propagation.TraceContext{},
        propagation.Baggage{},
    ))

    tracer = tp.Tracer(serviceName)

    return func() {
        ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
        defer cancel()
        tp.Shutdown(ctx)
    }
}

// Middleware for HTTP tracing
func tracingMiddleware(next http.Handler) http.Handler {
    return otelhttp.NewHandler(next, "http-server",
        otelhttp.WithMessageEvents(otelhttp.ReadEvents, otelhttp.WriteEvents),
    )
}

// Handler with manual span creation
func orderHandler(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context()

    // Create span for order processing
    ctx, span := tracer.Start(ctx, "process-order",
        trace.WithSpanKind(trace.SpanKindServer),
        trace.WithAttributes(
            attribute.String("http.method", r.Method),
            attribute.String("http.url", r.URL.String()),
        ),
    )
    defer span.End()

    // Validate order
    ctx, validateSpan := tracer.Start(ctx, "validate-order")
    if err := validateOrder(ctx); err != nil {
        validateSpan.RecordError(err)
        validateSpan.SetStatus(codes.Error, err.Error())
        http.Error(w, err.Error(), http.StatusBadRequest)
        validateSpan.End()
        return
    }
    validateSpan.End()

    // Check inventory (external service call)
    ctx, invSpan := tracer.Start(ctx, "check-inventory",
        trace.WithSpanKind(trace.SpanKindClient),
    )
    available, err := checkInventory(ctx, "http://inventory-service/check")
    if err != nil {
        invSpan.RecordError(err)
        invSpan.SetStatus(codes.Error, err.Error())
    }
    invSpan.SetAttributes(attribute.Bool("inventory.available", available))
    invSpan.End()

    // Add event to parent span
    span.AddEvent("order_processed", trace.WithAttributes(
        attribute.String("order_id", "ord-123"),
        attribute.Int("items_count", 5),
    ))

    w.WriteHeader(http.StatusCreated)
    w.Write([]byte(`{"order_id": "ord-123"}`))
}

// HTTP client with trace propagation
func checkInventory(ctx context.Context, url string) (bool, error) {
    client := &http.Client{
        Transport: otelhttp.NewTransport(http.DefaultTransport),
    }

    req, _ := http.NewRequestWithContext(ctx, "POST", url, nil)
    resp, err := client.Do(req)
    if err != nil {
        return false, err
    }
    defer resp.Body.Close()

    return resp.StatusCode == http.StatusOK, nil
}

func main() {
    cleanup := initTracer("order-service", "http://jaeger:14268/api/traces")
    defer cleanup()

    mux := http.NewServeMux()
    mux.HandleFunc("/orders", orderHandler)

    handler := tracingMiddleware(mux)

    log.Println("Starting server on :8080")
    log.Fatal(http.ListenAndServe(":8080", handler))
}
```

```typescript
// Node.js/TypeScript Distributed Tracing
import { NodeSDK } from "@opentelemetry/sdk-node";
import { getNodeAutoInstrumentations } from "@opentelemetry/auto-instrumentations-node";
import { JaegerExporter } from "@opentelemetry/exporter-jaeger";
import { Resource } from "@opentelemetry/resources";
import { SemanticResourceAttributes } from "@opentelemetry/semantic-conventions";
import { trace, context, SpanKind, SpanStatusCode } from "@opentelemetry/api";
import { W3CTraceContextPropagator } from "@opentelemetry/core";
import express, { Request, Response, NextFunction } from "express";
import axios from "axios";

// Initialize OpenTelemetry SDK
const sdk = new NodeSDK({
  resource: new Resource({
    [SemanticResourceAttributes.SERVICE_NAME]: "user-service",
    [SemanticResourceAttributes.SERVICE_VERSION]: "1.0.0",
  }),
  traceExporter: new JaegerExporter({
    endpoint: "http://jaeger:14268/api/traces",
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

// Graceful shutdown
process.on("SIGTERM", () => {
  sdk
    .shutdown()
    .then(() => console.log("Tracing terminated"))
    .catch((error) => console.error("Error terminating tracing", error))
    .finally(() => process.exit(0));
});

const tracer = trace.getTracer("user-service");
const app = express();
app.use(express.json());

// Custom tracing middleware
const tracingMiddleware = (req: Request, res: Response, next: NextFunction) => {
  const span = trace.getActiveSpan();
  if (span) {
    span.setAttribute("http.request_id", req.headers["x-request-id"] as string);
    span.setAttribute("user.id", req.headers["x-user-id"] as string);
  }
  next();
};

app.use(tracingMiddleware);

// Handler with manual spans
app.post("/users", async (req: Request, res: Response) => {
  const parentSpan = trace.getActiveSpan();

  // Create child span for validation
  const validateSpan = tracer.startSpan("validate-user-input", {
    kind: SpanKind.INTERNAL,
    attributes: {
      "user.email": req.body.email,
    },
  });

  try {
    // Validate input
    if (!req.body.email) {
      validateSpan.setStatus({ code: SpanStatusCode.ERROR, message: "Missing email" });
      throw new Error("Email is required");
    }
    validateSpan.end();

    // Check if user exists (external service)
    const checkSpan = tracer.startSpan("check-existing-user", {
      kind: SpanKind.CLIENT,
    });

    const ctx = trace.setSpan(context.active(), checkSpan);
    await context.with(ctx, async () => {
      const response = await axios.get(`http://auth-service/users?email=${req.body.email}`, {
        headers: {
          // Trace context is auto-propagated by instrumentation
        },
      });
      checkSpan.setAttribute("user.exists", response.data.exists);
    });
    checkSpan.end();

    // Create user
    const createSpan = tracer.startSpan("create-user-in-db", {
      kind: SpanKind.CLIENT,
      attributes: {
        "db.system": "postgresql",
        "db.operation": "INSERT",
      },
    });

    const user = await createUser(req.body);
    createSpan.setAttribute("user.id", user.id);
    createSpan.end();

    // Add event to parent span
    parentSpan?.addEvent("user_created", {
      "user.id": user.id,
      "user.email": user.email,
    });

    res.status(201).json(user);
  } catch (error) {
    parentSpan?.recordException(error as Error);
    parentSpan?.setStatus({ code: SpanStatusCode.ERROR, message: (error as Error).message });
    res.status(400).json({ error: (error as Error).message });
  }
});

// Utility for creating spans with automatic error handling
async function withSpan<T>(name: string, fn: (span: Span) => Promise<T>, options?: SpanOptions): Promise<T> {
  return tracer.startActiveSpan(name, options ?? {}, async (span) => {
    try {
      const result = await fn(span);
      span.setStatus({ code: SpanStatusCode.OK });
      return result;
    } catch (error) {
      span.recordException(error as Error);
      span.setStatus({ code: SpanStatusCode.ERROR, message: (error as Error).message });
      throw error;
    } finally {
      span.end();
    }
  });
}

// Usage
app.get("/users/:id", async (req: Request, res: Response) => {
  const user = await withSpan("get-user", async (span) => {
    span.setAttribute("user.id", req.params.id);

    const user = await withSpan(
      "fetch-from-cache",
      async (cacheSpan) => {
        cacheSpan.setAttribute("cache.type", "redis");
        return await cache.get(`user:${req.params.id}`);
      },
      { kind: SpanKind.CLIENT },
    );

    if (!user) {
      return await withSpan(
        "fetch-from-db",
        async (dbSpan) => {
          dbSpan.setAttribute("db.system", "postgresql");
          return await db.users.findById(req.params.id);
        },
        { kind: SpanKind.CLIENT },
      );
    }

    return user;
  });

  res.json(user);
});

app.listen(3000, () => console.log("Server running on port 3000"));
```

## Sampling Strategies

```python
# Adaptive Sampling Configuration
from opentelemetry.sdk.trace.sampling import (
    Sampler,
    SamplingResult,
    Decision,
    ParentBased,
    TraceIdRatioBased,
    ALWAYS_ON,
    ALWAYS_OFF,
)
from opentelemetry.trace import SpanKind
from typing import Optional, Sequence


class AdaptiveSampler(Sampler):
    """Sample based on span attributes and error rates."""

    def __init__(
        self,
        base_rate: float = 0.1,
        error_rate: float = 1.0,
        slow_threshold_ms: float = 1000,
    ):
        self.base_rate = base_rate
        self.error_rate = error_rate
        self.slow_threshold_ms = slow_threshold_ms

    def should_sample(
        self,
        parent_context,
        trace_id: int,
        name: str,
        kind: SpanKind,
        attributes,
        links,
    ) -> SamplingResult:
        # Always sample errors
        if attributes and attributes.get("error", False):
            return SamplingResult(Decision.RECORD_AND_SAMPLE)

        # Always sample specific operations
        if name in ["payment.process", "order.create", "auth.login"]:
            return SamplingResult(Decision.RECORD_AND_SAMPLE)

        # Rate-based sampling for others
        if self._should_sample_by_rate(trace_id):
            return SamplingResult(Decision.RECORD_AND_SAMPLE)

        return SamplingResult(Decision.DROP)

    def _should_sample_by_rate(self, trace_id: int) -> bool:
        return (trace_id & 0xFFFFFFFF) < (self.base_rate * 0xFFFFFFFF)

    def get_description(self) -> str:
        return f"AdaptiveSampler(base_rate={self.base_rate})"
```

## Best Practices

### Instrumentation

- Use auto-instrumentation where possible
- Add custom spans for business logic
- Include meaningful attributes
- Propagate context across async boundaries

### Sampling

- Use tail-based sampling for important traces
- Always sample errors and slow requests
- Adjust rates based on traffic volume
- Consider cost of storage

### Context Propagation

- Use W3C Trace Context standard
- Support B3 for legacy systems
- Include baggage for cross-cutting data
- Handle async contexts properly

### Operations

- Set appropriate retention periods
- Monitor trace ingestion rates
- Alert on tracing pipeline issues
- Correlate traces with logs and metrics

Distributed tracing powers **Uber, Netflix, and Slack** debugging production issues at scale.

You implement comprehensive distributed tracing for full observability across microservices.