Splunk

---
name: splunk
description: "Splunk as a SIEM/security platform across all versions (9.x-10.x): SPL query development for security detection, indexer/search head architecture, forwarder management, SmartStore, CIM normalization for security data models, knowledge objects, and deployment at scale. WHEN: \"Splunk\", \"SPL\", \"search head\", \"indexer\", \"forwarder\", \"SmartStore\", \"Splunkbase\", \"props.conf\", \"transforms.conf\", \"data model\", \"saved search\". Do NOT use for general log/metrics observability, IT operations dashboards, or ITSI — that's the `splunk` skill in the `monitoring` plugin. For Splunk's security content layer (correlation searches, notable events, RBA), use the `splunk-es` skill."
license: MIT
---

# Splunk

This skill covers Splunk across all supported versions (9.x through 10.x), with deep knowledge of:

- Search Processing Language (SPL) development and optimization
- Splunk architecture (indexers, search heads, forwarders, deployment server)
- Indexer clustering and search head clustering
- SmartStore (remote storage tiering)
- Knowledge objects (lookups, macros, saved searches, field extractions, tags, event types)
- Common Information Model (CIM) and data models
- Splunk apps and add-ons (Splunkbase ecosystem)
- Data onboarding (inputs.conf, props.conf, transforms.conf)
- Deployment and management at scale
- Splunk Cloud vs. Splunk Enterprise differences

This skill's coverage spans Splunk holistically. When a question is version-specific, read the matching file in `references/versions/`. When the version is unknown, apply general guidance and note where behavior differs across versions.

## How to Approach Tasks

When you receive a request:

1. **Classify** the request type:
   - **Troubleshooting** -- Load `references/diagnostics.md`
   - **SPL optimization** -- Load `references/best-practices.md`
   - **Architecture / Deployment** -- Load `references/architecture.md`
   - **Data onboarding** -- Apply parsing and normalization expertise below
   - **Search development** -- Apply SPL expertise directly
   - **Enterprise Security** -- Read the `splunk-es` skill

2. **Identify version** -- Determine which Splunk version the user is running. If unclear, ask. Version matters for SPL2 availability (10.0+), feature access, and best practices. Load the matching `references/versions/<v>.md` file.

3. **Load context** -- Read the relevant reference file for deep knowledge.

4. **Analyze** -- Apply Splunk-specific reasoning, not generic SIEM advice.

5. **Recommend** -- Provide actionable, specific guidance with SPL examples.

6. **Verify** -- Suggest validation steps (search commands, REST API checks, btool for config verification).

## Core Expertise

### SPL Fundamentals

SPL (Search Processing Language) is a pipe-delimited language. Every search starts with a data retrieval command and pipes through transforming commands.

```spl
index=main sourcetype=WinEventLog:Security EventCode=4625
| stats count by src_ip, user
| where count > 10
| sort -count
| lookup geoip src_ip OUTPUT country
| table src_ip, user, count, country
```

Key principles:
- **Filter early** -- Use `index`, `sourcetype`, and time range to limit data before piping
- **Use indexed fields first** -- `index`, `source`, `sourcetype`, `host` are indexed; custom fields are extracted at search time
- **Avoid wildcards at start** -- `sourcetype=Win*` is fast; `field=*something` forces full scan
- **Prefer `stats` over `transaction`** -- `stats` is map-reduce parallelizable; `transaction` is single-threaded
- **Use `tstats` for data model queries** -- Orders of magnitude faster than raw search when data models are accelerated

### SPL Command Categories

| Category | Commands | Purpose |
|---|---|---|
| **Searching** | `search`, `where`, `regex` | Filter events |
| **Aggregation** | `stats`, `chart`, `timechart`, `eventstats`, `streamstats` | Compute statistics |
| **Transformation** | `eval`, `rex`, `rename`, `replace`, `fillnull` | Modify fields |
| **Ordering** | `sort`, `head`, `tail`, `reverse`, `dedup` | Order and limit results |
| **Lookup** | `lookup`, `inputlookup`, `outputlookup` | Enrich from CSV/KV store |
| **Join** | `join`, `append`, `appendcols` | Combine result sets |
| **Subsearch** | `[search ...]` | Nested searches (use sparingly -- memory-limited) |
| **Reporting** | `table`, `fields`, `top`, `rare` | Format output |
| **Data Model** | `tstats`, `datamodel`, `from` | Accelerated data model queries |

### Advanced SPL Patterns

**Risk-based pattern (for Splunk ES):**
```spl
| tstats summariesonly=true count from datamodel=Risk.All_Risk
    where All_Risk.risk_object_type="user"
    by All_Risk.risk_object, All_Risk.risk_score, All_Risk.source
| stats sum(All_Risk.risk_score) as total_risk, dc(All_Risk.source) as source_count, values(All_Risk.source) as sources
    by All_Risk.risk_object
| where total_risk > 100 AND source_count > 3
```

**Transaction alternative using stats:**
```spl
index=web sourcetype=access_combined
| stats min(_time) as start, max(_time) as end, count, values(uri_path) as pages by session_id
| eval duration=end-start
| where duration > 300 AND count > 50
```

**Subsearch optimization (avoid when possible):**
```spl
| Bad: index=firewall [search index=threat_intel | fields ip | rename ip as src_ip]
| Better: index=firewall | lookup threat_intel ip as src_ip OUTPUT threat_score | where isnotnull(threat_score)
```

### Configuration File Hierarchy

Splunk uses a layered configuration system with precedence rules:

```
$SPLUNK_HOME/etc/system/default/          (lowest priority -- never edit)
$SPLUNK_HOME/etc/system/local/            (system-wide overrides)
$SPLUNK_HOME/etc/apps/<app>/default/      (app defaults)
$SPLUNK_HOME/etc/apps/<app>/local/        (app local overrides)
$SPLUNK_HOME/etc/users/<user>/<app>/local/ (user-level overrides -- highest priority)
```

Key configuration files:

| File | Purpose | Key Settings |
|---|---|---|
| `inputs.conf` | Data inputs (monitors, scripted, TCP/UDP, HTTP Event Collector) | `[monitor://path]`, `[http://token]`, `index`, `sourcetype` |
| `props.conf` | Parsing, timestamp extraction, field extraction, line breaking | `TIME_FORMAT`, `LINE_BREAKER`, `SHOULD_LINEMERGE`, `TRANSFORMS-*` |
| `transforms.conf` | Field extraction regex, lookup definitions, routing | `REGEX`, `FORMAT`, `DEST_KEY`, `filename` |
| `outputs.conf` | Forwarding destinations | `[tcpout:group]`, `server`, `sslCertPath` |
| `indexes.conf` | Index definitions, storage paths, retention | `homePath`, `coldPath`, `frozenTimePeriodInSecs`, `maxTotalDataSizeMB` |
| `server.conf` | Server-level settings, clustering, SSL | `[clustering]`, `[sslConfig]`, `serverName` |
| `savedsearches.conf` | Saved searches, alerts, scheduled reports | `search`, `cron_schedule`, `alert.severity` |
| `authorize.conf` | Role definitions and capabilities | `[role_*]`, `srchIndexesAllowed`, `importRoles` |

**Always validate configs with btool:**
```bash
$SPLUNK_HOME/bin/splunk btool props list --debug | grep -i sourcetype_name
$SPLUNK_HOME/bin/splunk btool inputs list --debug
```

### Data Onboarding Workflow

```
1. Identify log source format (syslog, JSON, CSV, custom)
        |
2. Create sourcetype (props.conf: line breaking, timestamp, field extraction)
        |
3. Define inputs (inputs.conf: monitor, TCP/UDP, HEC, scripted)
        |
4. Map to CIM (props.conf/transforms.conf: field aliases, lookups, tags)
        |
5. Validate (search for the new data, check field extraction, verify CIM compliance)
        |
6. Deploy (deployment server push to forwarders, or app package)
```

**Example: Onboarding a JSON log source:**

```ini
# props.conf
[custom:myapp_json]
KV_MODE = json
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%Z
TIME_PREFIX = \"timestamp\":\"
MAX_TIMESTAMP_LOOKAHEAD = 30
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
category = Custom
description = My Application JSON Logs
```

### Forwarder Types

| Type | Purpose | Capabilities |
|---|---|---|
| **Universal Forwarder (UF)** | Lightweight log shipping | Collects and forwards raw data. No parsing, no search. Minimal resource usage. |
| **Heavy Forwarder (HF)** | Intermediate processing | Full Splunk instance. Can parse, filter, route, mask data before forwarding. |
| **HTTP Event Collector (HEC)** | Token-based HTTP/HTTPS input | Receives JSON events over HTTP. Ideal for applications, containers, serverless. |
| **Syslog** | Network-based log reception | Splunk can receive syslog on TCP/UDP. Use HF or dedicated syslog server for scale. |

### Index Design

Best practices for index architecture:

- **Separate indexes by data source type** -- `index=windows`, `index=firewall`, `index=cloud_audit`. Enables granular retention, access control, and search efficiency.
- **Separate indexes by retention requirement** -- Different compliance needs = different indexes.
- **Size indexes appropriately** -- Each index has overhead. Don't create one index per host.
- **Use `lastChanceIndex`** -- Catch misconfigured inputs instead of losing data.
- **Volume-based retention** -- Use `maxTotalDataSizeMB` as the primary retention control; `frozenTimePeriodInSecs` as secondary.

### Search Optimization

Performance tuning for expensive searches:

1. **Narrow time range** -- The single most impactful optimization. Always specify the smallest time window.
2. **Use indexed fields** -- `index`, `source`, `sourcetype`, `host` skip the raw data scan.
3. **Accelerate data models** -- `tstats` on accelerated data models is 10-100x faster than raw search.
4. **Avoid `join`** -- Use `stats` with shared keys or `lookup` instead. `join` is memory-limited.
5. **Use `fields` early** -- `| fields src_ip, dest_ip, action` reduces data passed through the pipeline.
6. **Avoid real-time search** -- Real-time searches consume persistent search slots. Use indexed real-time or scheduled searches.
7. **Parallelize with `map`** -- For iterative searches, `map` can parallelize (but use carefully).

### Common Pitfalls

**1. License violations from misconfigured inputs**
Every event indexed counts toward your license. Duplicate inputs (e.g., forwarder + monitor on same file) double your license usage. Always check:
```spl
index=_internal source=*license_usage.log type=Usage | stats sum(b) as bytes by s, st | eval GB=round(bytes/1024/1024/1024,2) | sort -GB
```

**2. Search head memory exhaustion**
Searches that return millions of rows without aggregation consume search head memory. Always use `stats`, `timechart`, or `head` to limit results.

**3. Timestamp extraction failures**
Incorrect `TIME_FORMAT` or `TIME_PREFIX` causes events to cluster at parse time instead of event time. Verify with:
```spl
index=your_index sourcetype=your_st | eval _time_diff=abs(now()-_time) | where _time_diff > 86400
```

**4. Field extraction at index time vs. search time**
Index-time extraction increases indexing overhead and is irreversible. Use search-time extraction (default) unless you have a strong performance reason.

**5. Knowledge object conflicts**
Multiple apps defining the same field extraction, tag, or event type creates conflicts. Use app namespacing and check with:
```bash
$SPLUNK_HOME/bin/splunk btool props list --debug | grep EXTRACT
```

## Version-Specific Guidance

| Version | Reference | What's version-specific |
|---|---|---|
| Splunk 9.4.x | `references/versions/9.4.md` | Federated search improvements, Dashboard Studio maturity, ingest actions, security hardening defaults |
| Splunk 10.0 | `references/versions/10.0.md` | SPL2, Edge Processor, FIPS 140-3, dataset catalog, Ingest Processor, pipe-first syntax |

## Reference Files

Load these when you need deep knowledge for a specific area:

- `references/architecture.md` -- Indexer clustering, search head clustering, SmartStore, deployment server, data pipeline internals. Read for "how does Splunk work" or scaling questions.
- `references/diagnostics.md` -- License usage troubleshooting, search performance issues, forwarder connectivity, indexer bottlenecks, REST API diagnostics. Read when troubleshooting problems.
- `references/best-practices.md` -- SPL optimization patterns, CIM compliance, data model acceleration, notable event management, ES correlation search tuning. Read for optimization and detection engineering questions.