Splunk

---
name: splunk
description: "Covers Splunk Enterprise and Splunk Cloud as a log/metrics observability platform: SPL (Search Processing Language), distributed architecture, data ingestion, dashboards, alerting, ITSI, and license cost optimization. Use for \"Splunk\", \"splunk\", \"SPL\", \"Search Processing Language\", \"Splunk Enterprise\", \"Splunk Cloud\", \"Splunk indexer\", \"Splunk forwarder\", \"HEC\", \"HTTP Event Collector\", \"SmartStore\", \"Splunk ITSI\", \"sourcetype\", \"Splunk dashboard\", \"Splunk alert\", \"savedsearches.conf\", \"props.conf\", \"transforms.conf\", \"Universal Forwarder\", \"Heavy Forwarder\". Do NOT use for SIEM/security detection use of Splunk (correlation searches, notable events, threat detection content) — that's the `splunk`/`splunk-es` skills in the `security` plugin."
license: MIT
---

# Splunk Technology Expert

This skill covers Splunk Enterprise 9.x and Splunk Cloud: SPL, distributed architecture, data ingestion, dashboards, alerting, ITSI, and license cost optimization. Every recommendation addresses the tradeoff triangle: **search performance**, **license cost**, and **operational complexity**.

Splunk is licensed by daily indexing volume (GB/day). Always remind users to verify current licensing at https://www.splunk.com/en_us/products/pricing.html.

## How to Approach Tasks

When you receive a request:

1. **Classify** the request by domain:
   - **Architecture / deployment** -- Load `references/architecture.md`
   - **SPL queries** -- Load `references/spl.md`
   - **Ingestion, dashboards, alerting, ITSI** -- Load `references/features.md`
   - **Cost optimization** -- Load `references/cost.md`

2. **Think SPL-first** -- Splunk's power lives in SPL. For any data question, provide the SPL query. Always start with the most efficient search (indexed fields, tstats) before falling back to raw search.

3. **Include license context** -- Every byte indexed counts against the daily license. Before recommending new data sources, estimate the daily volume impact.

4. **Prefer search-time over index-time extractions** -- Index-time field extractions increase tsidx size and cannot be changed retroactively. Search-time extractions are flexible and free.

5. **Recommend filtering at the forwarder** -- The cheapest data to process is data that never reaches the indexer. Always consider forwarder-level filtering first.

## Core Expertise

You have deep knowledge across these Splunk areas:

- **Architecture:** Search Heads, Indexers, Universal Forwarders, Heavy Forwarders, Deployment Server, Cluster Manager, License Server, Indexer Cluster (RF/SF), Search Head Cluster, SmartStore
- **SPL:** Pipeline-based query language -- streaming commands (where, eval, rex), transforming commands (stats, timechart, chart), generating commands (tstats, inputlookup), orchestrating commands (join, append, subsearch), macros
- **Data Ingestion:** Monitor inputs, HEC (HTTP Event Collector), syslog (SC4S), scripted inputs, modular inputs, props.conf, transforms.conf, index management
- **Dashboards:** SimpleXML (Classic), Dashboard Studio (JSON), tokens, drilldown patterns, Splunk Observability Cloud integration
- **Alerting:** Saved searches as alerts, scheduled vs real-time, throttling, alert actions (email, webhook, script, notable events), Enterprise Security
- **ITSI:** Service trees, KPIs, Glass Tables, Episode Review, Adaptive Thresholds, Predictive Analytics
- **Cost:** GB/day license model, forwarder filtering, summary indexing, data model acceleration, SmartStore, tiered retention

## Distributed Architecture

```
[Universal Forwarders] -> [Heavy Forwarder] -> [Indexer Cluster (3+ peers)]
                                                        |
                                              [Cluster Manager]
[Search Head Cluster (3 SH)] <-> [Deployer]
                  |
          [License Server]
```

### Core Components

| Component | Role |
|-----------|------|
| Search Head | User-facing; runs searches, dashboards, alerts |
| Indexer | Receives events, indexes to disk, responds to searches |
| Universal Forwarder | Lightweight agent on source hosts (~100 MB RAM) |
| Heavy Forwarder | Full instance that can parse, filter, mask, route data |
| Deployment Server | Centrally manages forwarder configuration |
| Cluster Manager | Manages Indexer Cluster replication and search factors |
| License Server | Enforces daily GB/day indexing limits |

### Index Bucket Lifecycle

```
Hot (actively written, fast SSD) -> Warm (closed, local) -> Cold (cheaper storage) -> Frozen (archived/deleted)
```

SmartStore moves warm/cold buckets to S3-compatible object storage, reducing local disk cost.

## SPL Quick Reference

SPL is pipeline-based. Commands chained with `|`. Processing flows left to right.

```spl
index=main sourcetype=access_combined | stats count by status
```

### Essential Commands

| Command | Type | Purpose |
|---------|------|---------|
| `search` / `where` | Streaming | Filter events |
| `eval` | Streaming | Create/modify fields |
| `rex` | Streaming | Regex field extraction |
| `stats` | Transforming | Aggregate into table |
| `timechart` | Transforming | Aggregate over time |
| `chart` | Transforming | Pivot-style aggregation |
| `table` | Streaming | Select and order fields |
| `top` / `rare` | Transforming | Most/least frequent values |
| `transaction` | Transforming | Group events by session/ID |
| `lookup` | Streaming | Enrich with lookup table |
| `tstats` | Generating | Fast search over data models |
| `eventstats` | Transforming | Add aggregates back to events |
| `streamstats` | Streaming | Running/cumulative statistics |
| `join` | Orchestrating | SQL-style join (use sparingly) |

### Key SPL Examples

**Error rate by service:**
```spl
index=app_logs
| eval is_error = if(level="ERROR", 1, 0)
| timechart span=1h sum(is_error) as errors, count as total by service
| eval error_rate = round(errors / total * 100, 2)
```

**Top 10 slowest endpoints:**
```spl
index=api_logs sourcetype=api_access
| stats perc95(response_ms) as p95, count by endpoint
| sort -p95 | head 10
```

**License volume by index and sourcetype:**
```spl
index=_internal sourcetype=splunkd source=*license_usage*
| stats sum(b) as bytes by idx, st
| eval GB = round(bytes / 1024 / 1024 / 1024, 3)
| sort -GB
```

## Top 10 Cost Rules

1. **Filter at the forwarder** -- Drop DEBUG logs, health checks, and noisy events in `transforms.conf` on the Heavy Forwarder using `nullQueue`. Data never reaches the indexer.
2. **Reduce verbosity at source** -- Set application log levels to INFO/WARN in production. Disable access logging for internal health endpoints.
3. **Use summary indexing** -- Pre-aggregate high-volume data into a summary index. Dashboards query the summary instead of raw data.
4. **Accelerate data models** -- CIM-compliant data models with acceleration enable `tstats` (10-100x faster than raw search) without additional license cost.
5. **Implement SmartStore** -- Move warm/cold buckets to object storage. Reduces local disk cost for long-retention data.
6. **Use tiered retention** -- Short retention for noisy indexes (7 days for debug), long retention for compliance (1-7 years on cold/SmartStore).
7. **Prefer search-time extractions** -- Index-time field extractions increase tsidx size. Search-time extractions add no index overhead.
8. **Monitor license usage weekly** -- Query `index=_internal source=*license_usage*` to identify growth trends by sourcetype.
9. **Use HEC with batching** -- Batch events in HEC requests to reduce overhead and control flow.
10. **Avoid real-time alerts on high-volume indexes** -- Use scheduled alerts with `| stats` aggregation instead.

## Common Pitfalls

**1. Five license warnings in 30 days**
After 5 daily volume violations in a rolling 30-day window, Splunk disables search for non-internal indexes. Monitor daily usage and set alerts at 80% of license capacity.

**2. Using `join` for everything**
SPL `join` is expensive and limited. Prefer `stats` with shared fields, `lookup`, or `append` + `stats`. Reserve `join` for genuinely different data sets.

**3. Real-time searches on high-volume indexes**
Real-time searches consume significant indexer CPU. Use scheduled searches with 5-minute windows instead.

**4. Index-time field extractions that cannot be undone**
Fields extracted at index time (`TRANSFORMS` in props.conf) increase tsidx size permanently. If requirements change, historical data retains the old extraction.

**5. Subsearch limits**
Subsearch default: 10,000 results, 60-second timeout. Results silently truncated. Use `| join` or `| lookup` for large result sets.

**6. Missing sourcetype configuration**
Without proper `props.conf` settings, Splunk auto-detects sourcetype, often incorrectly. Always define `SHOULD_LINEMERGE`, `TIME_PREFIX`, `TIME_FORMAT` for each sourcetype.

**7. Not using Deployment Server for forwarders**
Manual forwarder configuration does not scale. Use Deployment Server with server classes for centralized management of thousands of UFs.

## Splunk Cloud vs Enterprise

| Aspect | Splunk Cloud | Splunk Enterprise |
|--------|-------------|------------------|
| Infrastructure | Managed by Splunk | Customer-managed |
| Upgrades | Automatic | Manual |
| SmartStore | Default | Optional (S3-compatible) |
| Pricing | Per-GB ingest or workload | Per-GB/day license |
| Apps | Splunk-vetted only | Any app |

## Reference Files

Load these when you need deep knowledge for a specific area:

- `references/architecture.md` -- Core components (Search Head, Indexer, Forwarder), distributed architecture (Indexer Cluster, SHC), data pipeline (input/parsing/indexing/search), bucket lifecycle, SmartStore, Splunk Cloud vs Enterprise. Read for deployment and architecture questions.
- `references/spl.md` -- SPL command reference (streaming, transforming, generating, orchestrating), subsearch, macros, and 15+ real-world SPL examples covering error analysis, anomaly detection, log correlation, regex extraction, and SLA calculation. Read for query construction.
- `references/features.md` -- Data ingestion (monitor inputs, HEC, syslog/SC4S, scripted/modular inputs, props.conf, transforms.conf), dashboards (SimpleXML, Dashboard Studio, tokens, drilldown), alerting (saved searches, types, actions, throttling), ITSI (services, KPIs, Glass Tables, Episode Review, Adaptive Thresholds). Read for feature and configuration questions.
- `references/cost.md` -- License model (GB/day), forwarder filtering, summary indexing, data model acceleration, SmartStore, tiered retention, volume monitoring queries. Read for cost and license optimization.