Elk

---
name: elk
description: "Covers the Elastic Stack (ELK) 8.x/9.x for observability use cases: Elasticsearch cluster management, Elastic Agent/Fleet, Kibana, ingest pipelines, KQL/ES|QL, Index Lifecycle Management, APM agents, distributed tracing, log management, data streams, and diagnostics. Use for \"ELK\", \"Elastic Stack\", \"Elasticsearch observability\", \"Kibana\", \"Elastic Agent\", \"Fleet\", \"Filebeat\", \"Metricbeat\", \"Elastic APM\", \"KQL\", \"ES|QL\", \"ILM\", \"index lifecycle\", \"ingest pipeline\", \"data stream\", \"Logstash\", \"ECS\", \"Elastic Common Schema\", \"searchable snapshots\". Do NOT use for Elastic SIEM/security detection content — that's the `elastic-security` skill in the `security` plugin. Do NOT use for Elasticsearch/OpenSearch as a general-purpose database engine (indexing internals, query DSL for app data) — that's the `elasticsearch`/`opensearch` skills in the `database` plugin."
license: MIT
---

# ELK Stack Observability Expert

This skill covers the Elastic Stack (Elasticsearch + Kibana 8.x/9.x) for observability use cases: log management, metrics, APM, and distributed tracing. Every recommendation addresses the tradeoff triangle: **search performance**, **storage cost**, and **operational complexity**.

## How to Approach Tasks

When you receive a request:

1. **Classify** the request by area:
   - **Architecture** (components, Elastic Agent, Fleet, ILM, ingest pipelines) -- Load `references/architecture.md`
   - **Log management** (collection, KQL, data streams, ILM policies) -- Load `references/log-management.md`
   - **APM** (agents, distributed tracing, OTel integration) -- Load `references/apm.md`
   - **Diagnostics** (cluster health, shard allocation, slow queries) -- Load `references/diagnostics.md`

2. **Recommend Elastic Agent over legacy Beats** -- Elastic Agent is the unified collection agent replacing Filebeat, Metricbeat, Heartbeat, etc. Recommend Beats only for air-gapped or resource-constrained environments.

3. **Default to data streams** -- All observability data (logs, metrics, traces) should use data streams with the naming convention `<type>-<dataset>-<namespace>`.

4. **Design ILM policies from day one** -- Hot/warm/cold/frozen tiers with searchable snapshots dramatically reduce storage cost. Never store all data on hot tier indefinitely.

5. **Normalize to ECS** -- Elastic Common Schema is required for Kibana Observability features. Always include ECS normalization in ingest pipelines.

## Core Expertise

- **Architecture:** Elasticsearch cluster topology (hot/warm/cold/frozen nodes), Elastic Agent (Fleet-managed and standalone), Fleet Server, integration packages, ingest pipelines (grok, dissect, date, enrich, script), data streams, ILM phases and actions
- **Log management:** Filebeat/Elastic Agent log collection, Kubernetes autodiscover, index templates, KQL, Lucene, ES|QL (pipe-based analytics), Discover, Logs Explorer, data stream operations
- **Metrics:** Metricbeat modules, TSDB index mode (40-70% storage reduction), Logsdb index mode (65% reduction in 9.x), metric downsampling, Lens visualization, Prometheus remote write integration
- **APM:** Elastic APM agents (Java, .NET, Node.js, Python, Go, Ruby, PHP, Browser/RUM), distributed tracing (W3C Trace Context), service maps, correlations, OpenTelemetry OTLP integration, continuous profiling (eBPF)
- **Alerting:** Rule types (index threshold, metric threshold, log threshold, anomaly, uptime, SLO burn rate), connectors (Slack, PagerDuty, email, webhook, ServiceNow, OpsGenie), maintenance windows
- **9.x Features:** Logsdb default for logs, ES|QL expanded (LOOKUP JOIN, INLINESTATS, window functions), unified Observability app, hierarchical agent policies

## Elastic Stack Quick Reference

```
DATA SOURCES (Apps, OS, Containers, Network, Cloud)
        |
COLLECTION LAYER
  Elastic Agent (unified, Fleet-managed)
  └─ 300+ Integrations: logs, metrics, APM
        |
PROCESSING LAYER
  Ingest Node Pipelines (grok/dissect/date/enrich)
  Logstash (heavy ETL -- optional)
        |
STORAGE LAYER
  Elasticsearch Cluster
  ├─ Hot nodes (NVMe SSD, recent data)
  ├─ Warm nodes (SSD/HDD, 1-30 days, read-only)
  ├─ Cold nodes (searchable snapshots from S3/GCS)
  └─ Frozen tier (on-demand snapshot mounts)
        |
VISUALIZATION / MANAGEMENT
  Kibana: Discover, Logs Explorer, APM, Dashboards, Fleet, Alerting
```

## Top 10 Operational Rules

1. **Use data streams for all observability data** -- Convention: `<type>-<dataset>-<namespace>` (e.g., `logs-nginx.access-production`). Enables automatic rollover and ILM.

2. **Configure ILM with tiered storage** -- Hot (1 day) > Warm (2-30 days, forcemerge, shrink) > Cold (30-90 days, searchable snapshots) > Frozen (90-365 days) > Delete.

3. **Target 10-50 GB primary shard size for logs** -- Over-sharding wastes heap. Use `max_primary_shard_size: 50gb` + `max_age: 1d` for rollover.

4. **Normalize all data to ECS** -- Kibana Observability features (Service Map, Logs Explorer, APM correlations) require Elastic Common Schema fields.

5. **Use TSDB index mode for metrics** -- 40-70% storage reduction via synthetic source and dimension-based routing.

6. **Use Logsdb index mode for logs (9.x default)** -- Column-store format provides ~65% storage reduction over standard indexing.

7. **Set refresh_interval to 30s on hot indices** -- Default 1s burns I/O during high-ingest periods. Most dashboards refresh every 30-60s anyway.

8. **Deploy Elastic Agent via Fleet** -- Zero-touch rollout with centralized policy management. Reserve standalone mode for air-gapped environments.

9. **Use ES|QL for analytics** -- Pipe-based syntax is more intuitive than complex Elasticsearch DSL aggregations. GA since 8.11.

10. **Monitor cluster health proactively** -- Track `_cluster/health`, shard allocation, disk watermarks, and circuit breaker trips.

## Common Pitfalls

**1. Over-sharding**
Many small shards waste heap and degrade query performance. Target 10-50 GB per shard for logs, 1-5 GB for metrics. Use rollover policies instead of time-based index creation.

**2. No ILM policy**
Storing all data on hot tier indefinitely wastes expensive NVMe storage. Implement hot/warm/cold/frozen from day one.

**3. Mapping explosion**
Dynamic mapping with unlimited fields causes `TooManyBucketsException` and heap pressure. Use `dynamic: false` or `dynamic: runtime` to prevent unbounded field creation.

**4. Missing ECS normalization**
Without ECS field mapping, Kibana Observability features (Service Map, APM UI, Logs Explorer) do not work. Always include ECS transformation in ingest pipelines.

**5. Ignoring disk watermarks**
Default watermarks: 85% low (stop allocating), 90% high (relocate), 95% flood (read-only). Monitor proactively and scale storage before hitting thresholds.

**6. Using grok when dissect suffices**
Grok uses regex (CPU-intensive). Dissect uses simple tokenization and is much faster. Use dissect for structured log formats; reserve grok for unstructured text.

## Reference Files

Load these when you need deep knowledge for a specific area:

- `references/architecture.md` -- Elasticsearch cluster components, Elastic Agent and Fleet, APM Server, ILM phases and policies, ingest pipeline processors, data stream naming, hot/warm/cold/frozen node configuration, ECS key fields, performance tuning, Snapshot Lifecycle Management, Cross-Cluster Search. Read for architecture and setup.
- `references/log-management.md` -- Log collection (Filebeat, Elastic Agent, Kubernetes autodiscover), index templates, data stream operations, KQL, Lucene, ES|QL, Discover and Logs Explorer. Read for log ingestion and search questions.
- `references/apm.md` -- APM agent setup (Java, .NET, Node.js, Python, Go, Ruby, Browser/RUM), APM data model (transactions, spans, errors, metrics), distributed tracing, service maps, correlations, OpenTelemetry integration, continuous profiling. Read for APM and tracing questions.
- `references/diagnostics.md` -- Cluster health APIs, shard allocation diagnostics, disk watermarks, slow query diagnostics, ILM troubleshooting, APM data stream health, Fleet agent status. Read for troubleshooting.

## Diagnostic Scripts

Ready-made Elasticsearch API scripts (read-only; OpenSearch-compatible endpoints) in `scripts/`.

- `scripts/01-cluster-health.sh` -- Status, node disk/heap, unassigned-shard explanation
- `scripts/02-index-audit.sh` -- Biggest indices, shard-size sanity, ILM errors