Datadog

---
name: datadog
description: "Expert agent for Datadog managed observability platform covering Agent deployment, integrations, DogStatsD custom metrics, APM distributed tracing, log management, monitors, SLOs, and cost optimization. Provides deep expertise with real pricing context, tagging strategy, and cardinality management. WHEN: \"Datadog\", \"datadog\", \"DogStatsD\", \"APM Datadog\", \"Datadog Agent\", \"Datadog monitor\", \"Datadog logs\", \"Datadog APM\", \"Datadog metrics\", \"Datadog SLO\", \"DD_API_KEY\", \"Datadog integration\", \"Datadog cost\", \"custom metrics Datadog\", \"Datadog Kubernetes\", \"Datadog Helm\", \"Datadog Cluster Agent\", \"Unified Service Tagging\"."
license: MIT
---

# Datadog Technology Expert

This skill covers Datadog's managed observability platform: infrastructure monitoring, APM, log management, synthetics, security monitoring, and cost optimization. Every recommendation addresses the tradeoff triangle: **observability depth**, **cost**, and **operational complexity**.

Datadog pricing is per-unit (host, GB, metric). Always remind users to verify current pricing at https://www.datadoghq.com/pricing/.

## How to Approach Tasks

When you receive a request:

1. **Classify** the request by domain:
   - **Architecture / Agent deployment** -- Load `references/architecture.md`
   - **Metrics, APM, logs, monitors, SLOs** -- Load `references/features.md`
   - **Cost optimization** -- Load `references/cost.md`

2. **Include cost context** -- Never recommend enabling a product (APM, NPM, logs, profiling) without addressing its billing model and alternatives. Provide concrete monthly estimates where possible.

3. **Enforce tagging discipline** -- Every recommendation should reference Unified Service Tagging (`env`, `service`, `version`). Tagging failures are the root cause of most Datadog usability and cost problems.

4. **Challenge cardinality** -- Before recommending custom metrics, ask whether the tag set will create a cardinality explosion. High-cardinality tags (user IDs, request IDs) are the single biggest cost trap.

5. **Recommend the right product** -- Datadog has many overlapping products. Guide users to the simplest, cheapest option that meets the need.

## Core Expertise

You have deep knowledge across these Datadog areas:

- **Agent:** Agent architecture (Collector, DogStatsD, Trace Agent, Process Agent, Security Agent), Kubernetes DaemonSet and Cluster Agent deployment, Helm chart configuration, Datadog Operator, autodiscovery
- **Infrastructure:** 600+ integrations, cloud integrations (AWS, Azure, GCP), host maps, container maps, live processes, infrastructure list
- **Metrics:** DogStatsD protocol (counter, gauge, histogram, distribution, set), metric query language, aggregation functions, rollups, Metrics without Limits
- **APM:** Tracing libraries (Java, Python, .NET, Node.js, Go, Ruby, PHP), distributed tracing, service catalog, service map, flame graphs, trace analytics, continuous profiler, ingestion controls, retention filters
- **Log Management:** Agent log collection, log processing pipelines (Grok parser, remappers, processors), indexes, exclusion filters, archives, rehydration, log-based metrics, Live Tail
- **Monitors & Alerting:** Metric monitors, log monitors, APM monitors, composite monitors, anomaly/forecast/outlier detection, Watchdog, SLOs (monitor-based, metric-based), burn-rate alerts, downtime scheduling
- **Synthetics:** API tests, browser tests, private locations
- **Security:** Cloud SIEM, CWS, CSPM, ASM
- **Cost:** Per-host pricing, custom metric cardinality, log cost optimization, APM trace sampling, estimated usage metrics, Usage Attribution, Metrics without Limits

## Agent Architecture

The Datadog Agent is the primary data collection component. It runs as a process on hosts or as a DaemonSet in Kubernetes.

**Agent sub-components:**

| Component | Purpose | Port |
|-----------|---------|------|
| Collector | System metrics + integration checks (15s default) | -- |
| DogStatsD | Custom metrics via StatsD protocol | UDP 8125 |
| Trace Agent | APM traces from tracing libraries | HTTP 8126 |
| Process Agent | Live process and container data | -- |
| Security Agent | CWS kernel events via eBPF | -- |
| NPM Agent | Network flow data via eBPF | -- |

**Agent config:** `/etc/datadog-agent/datadog.yaml`

```yaml
api_key: <DD_API_KEY>
site: datadoghq.com
logs_enabled: true
apm_config:
  enabled: true
process_config:
  enabled: true
```

### Kubernetes Deployment

```bash
helm repo add datadog https://helm.datadoghq.com
helm install datadog-agent datadog/datadog \
  --set datadog.apiKey=<DD_API_KEY> \
  --set datadog.logs.enabled=true \
  --set datadog.apm.portEnabled=true \
  --set agents.image.tag=7
```

The Cluster Agent (Deployment, not DaemonSet) provides cluster-level metadata, External Metrics Provider for HPA, and Admission Controller for automatic library injection.

## Unified Service Tagging

The three reserved tags that power correlation across all Datadog products:

```bash
DD_ENV=production
DD_SERVICE=checkout-api
DD_VERSION=1.4.2
```

Apply via environment variables on every container. Without UST, service maps, trace-to-log correlation, and deployment tracking break silently.

## Custom Metrics (DogStatsD)

| Type | Syntax | Behavior |
|------|--------|----------|
| Counter | `name:value\|c` | Summed per flush, reported as rate/s |
| Gauge | `name:value\|g` | Last value per flush interval |
| Histogram | `name:value\|h` | Per-agent avg, count, max, p95 |
| Distribution | `name:value\|d` | Global percentiles across all agents |
| Set | `name:value\|s` | Unique value count per flush |

**Key rule:** Distributions compute accurate global percentiles but count as multiple custom metrics (one per configured percentile). Use distributions for latency; use counters for throughput.

## Metric Query Language

```
aggregation:metric_name{tag_filter} by {group_by}
```

Examples:
```
avg:system.cpu.user{env:production} by {host}
sum:aws.elb.request_count{service:checkout}.as_rate()
p99:trace.web.request.duration{service:api,env:prod}
```

Functions: `avg`, `sum`, `min`, `max`, `count`. Rollup: `.rollup(sum, 60)`. Math: `abs()`, `log2()`, `cumsum()`, `diff()`, `top()`.

## Top 10 Cost Rules

1. **Audit custom metric cardinality monthly** -- Check `Metrics > Summary`, sort by distinct tag combinations. One metric with unbounded tags costs more than 100 hosts.
2. **Never tag metrics with user IDs, session IDs, or request IDs** -- These create millions of time series.
3. **Use exclusion filters on log indexes** -- Drop health checks and DEBUG logs before indexing. Keep 5-10% sample for debugging.
4. **Create tiered log indexes** -- 3-day retention for INFO, 15-day for ERROR. Default 15-day-everything is expensive.
5. **Set APM ingestion controls** -- `DD_TRACE_SAMPLE_RATE=0.1` for high-traffic services. Tracing Without Limits keeps 100% of errors regardless.
6. **Use Metrics without Limits** -- Configure which tag combinations to index, reducing custom metric count without losing raw data.
7. **Monitor estimated usage metrics** -- Build dashboards on `datadog.estimated_usage.*` metrics before the bill arrives.
8. **Beware container tag cardinality** -- Pod name and container ID as metric tags create unique time series per container restart.
9. **Use log-based metrics instead of indexing** -- Extract counts/measures from high-volume logs as custom metrics rather than indexing every event.
10. **Archive everything, index selectively** -- Ship all logs to S3/GCS for compliance. Only index what operators need to search interactively.

## Common Pitfalls

**1. Cardinality explosion from container tags**
Container orchestrators add pod name, replica set, and container ID tags to metrics. Each container restart creates new time series. Use `DD_TAGS` and `DD_CHECKS_TAG_CARDINALITY` to limit tag scope.

**2. Enabling every product on every host**
Each host running Infra + APM + NPM + CWS multiplies the per-host cost 3-4x. Enable APM only on application hosts. Enable NPM only where network visibility is needed.

**3. Lambda APM surprise billing**
Lambda functions are billed per invocation (150 invocations = 1 APM host-equivalent). High-invocation functions can cost more than EC2 hosts.

**4. Log rehydration without budget**
Rehydrating archives is billed per GB. Unplanned incident investigation of large archives generates unexpected charges. Set archive rehydration alerts.

**5. Missing Unified Service Tagging**
Without `env`, `service`, `version` tags, APM service maps show unknown services, trace-to-log linking fails, and deployment tracking is blind.

**6. Running DogStatsD metrics from buggy code**
A runaway loop emitting metrics with random tag values can spike custom metric count to millions within minutes. Monitor `datadog.estimated_usage.custom_metrics` with an alert.

## Key Reference

| Component | Port | Protocol |
|-----------|------|----------|
| DogStatsD | 8125 | UDP/UDS |
| Trace Agent | 8126 | HTTP |
| Agent IPC | 5001 | HTTP (local) |
| Datadog intake | 443 | HTTPS |

### Agent CLI

```bash
datadog-agent status          # full agent status
datadog-agent check <name>    # run integration check manually
datadog-agent flare           # collect diagnostics for support
datadog-agent configcheck     # validate configuration files
```

## Reference Files

Load these when you need deep knowledge for a specific area:

- `references/architecture.md` -- Agent internals, Kubernetes deployment (Helm/Operator/Cluster Agent), DogStatsD, API, integrations, autodiscovery, tagging strategy. Read for deployment and integration questions.
- `references/features.md` -- Metrics (query language, DogStatsD types), log management (pipelines, indexes, exclusion filters, archives), APM (tracing libraries, sampling, profiler), monitors (types, configuration), SLOs (metric-based, monitor-based, burn-rate alerts). Read for feature and configuration questions.
- `references/cost.md` -- Pricing model (per-host, per-metric, per-GB), custom metric cardinality, log cost optimization, APM trace sampling, estimated usage metrics, Usage Attribution, Metrics without Limits, common cost surprises. Read for cost and billing questions.