Analyzing Security Logs With Splunk

---
name: analyzing-security-logs-with-splunk
description: 'Leverages Splunk Enterprise Security and SPL (Search Processing Language) to investigate security incidents
  through log correlation, timeline reconstruction, and anomaly detection. Covers Windows event logs, firewall logs, proxy
  logs, and authentication data analysis. Activates for requests involving Splunk investigation, SPL queries, SIEM log analysis,
  security event correlation, or log-based incident investigation.

  '
domain: cybersecurity
subdomain: incident-response
tags:
- splunk
- SPL
- SIEM
- log-analysis
- security-monitoring
mitre_attack:
- T1070
- T1562
- T1059
version: 1.0.0
author: mahipal
license: Apache-2.0
atlas_techniques:
- AML.T0070
- AML.T0066
- AML.T0082
d3fend_techniques:
- Executable Denylisting
- Execution Isolation
- File Metadata Consistency Validation
- Content Format Conversion
- File Content Analysis
nist_ai_rmf:
- MEASURE-2.7
- MAP-5.1
- MANAGE-2.4
- MANAGE-3.1
- MEASURE-3.1
nist_csf:
- RS.MA-01
- RS.MA-02
- RS.AN-03
- RC.RP-01
---

# Analyzing Security Logs with Splunk

## When to Use

- Investigating a security incident that requires correlation across multiple log sources
- Hunting for adversary activity using known TTPs and IOCs
- Building detection rules for specific attack patterns
- Reconstructing an incident timeline from disparate log sources
- Analyzing authentication anomalies, lateral movement, or data exfiltration patterns

**Do not use** for real-time packet-level analysis; use Wireshark or Zeek for full packet capture analysis.

## Prerequisites

- Splunk Enterprise or Splunk Cloud with Enterprise Security (ES) app installed
- Log sources ingested: Windows Event Logs (via Splunk Universal Forwarder or WEF), firewall, proxy, DNS, EDR, email gateway
- Splunk CIM (Common Information Model) data models configured for normalized field names
- SPL proficiency at intermediate level or higher
- Role-based access with `search` and `accelerate_search` capabilities in Splunk

## Workflow

### Step 1: Scope the Investigation in Splunk

Define search parameters based on incident triage data:

```spl
| Set initial investigation scope
index=windows OR index=firewall OR index=proxy
  earliest="2025-11-14T00:00:00" latest="2025-11-16T00:00:00"
  (host="WKSTN-042" OR src_ip="10.1.5.42" OR user="jsmith")
| stats count by index, sourcetype, host
| sort -count
```

This query establishes which log sources contain relevant data for the investigation timeframe and affected assets.

### Step 2: Analyze Authentication Events

Investigate suspicious authentication patterns using Windows Security Event Logs:

```spl
| Detect brute force and credential stuffing
index=windows sourcetype="WinEventLog:Security" EventCode=4625
  earliest=-24h
| stats count as failed_attempts, values(src_ip) as source_ips,
  dc(src_ip) as unique_sources by TargetUserName
| where failed_attempts > 10
| sort -failed_attempts

| Detect pass-the-hash (Logon Type 9 - NewCredentials)
index=windows sourcetype="WinEventLog:Security" EventCode=4624
  Logon_Type=9
| table _time, host, TargetUserName, src_ip, LogonProcessName

| Detect lateral movement via RDP
index=windows sourcetype="WinEventLog:Security" EventCode=4624
  Logon_Type=10
| stats count, values(host) as targets by TargetUserName, src_ip
| where count > 3
| sort -count
```

### Step 3: Trace Process Execution

Use Sysmon logs to reconstruct process execution chains:

```spl
| Process creation with parent chain (Sysmon Event ID 1)
index=sysmon EventCode=1 host="WKSTN-042"
  earliest="2025-11-15T14:00:00" latest="2025-11-15T15:00:00"
| table _time, ParentImage, ParentCommandLine, Image, CommandLine, User, Hashes
| sort _time

| Detect suspicious PowerShell execution
index=sysmon EventCode=1 Image="*\\powershell.exe"
  (CommandLine="*-enc*" OR CommandLine="*-encodedcommand*"
   OR CommandLine="*downloadstring*" OR CommandLine="*iex*")
| table _time, host, User, ParentImage, CommandLine
| sort _time

| Detect LSASS credential dumping
index=sysmon EventCode=10 TargetImage="*\\lsass.exe"
  GrantedAccess=0x1010
| table _time, host, SourceImage, SourceUser, GrantedAccess
```

### Step 4: Analyze Network Activity

Correlate network logs with endpoint events:

```spl
| Detect C2 beaconing pattern
index=proxy OR index=firewall dest_ip="185.220.101.42"
| timechart span=1m count by src_ip
| where count > 0

| Detect DNS tunneling (high query volume to single domain)
index=dns
| rex field=query "(?<subdomain>[^\.]+)\.(?<domain>[^\.]+\.[^\.]+)$"
| stats count, avg(len(query)) as avg_query_len by domain, src_ip
| where count > 500 AND avg_query_len > 40
| sort -count

| Detect large data transfers (potential exfiltration)
index=proxy action=allowed
| stats sum(bytes_out) as total_bytes by src_ip, dest_ip, dest_host
| eval total_MB=round(total_bytes/1024/1024,2)
| where total_MB > 100
| sort -total_MB
```

### Step 5: Build the Incident Timeline

Reconstruct a unified timeline across all log sources:

```spl
| Unified incident timeline
index=windows OR index=sysmon OR index=proxy OR index=firewall
  (host="WKSTN-042" OR src_ip="10.1.5.42" OR user="jsmith")
  earliest="2025-11-15T14:00:00" latest="2025-11-15T16:00:00"
| eval event_summary=case(
    sourcetype=="WinEventLog:Security" AND EventCode==4624, "Logon: ".TargetUserName." from ".src_ip,
    sourcetype=="WinEventLog:Security" AND EventCode==4625, "Failed logon: ".TargetUserName,
    sourcetype=="XmlWinEventLog:Microsoft-Windows-Sysmon/Operational" AND EventCode==1,
      "Process: ".Image." by ".User,
    sourcetype=="proxy", "Web: ".http_method." ".url,
    1==1, sourcetype.": ".EventCode)
| table _time, sourcetype, host, event_summary
| sort _time
```

### Step 6: Create Detection Rules

Convert investigation findings into persistent Splunk correlation searches:

```spl
| Correlation search: PowerShell spawned by Office applications
index=sysmon EventCode=1
  Image="*\\powershell.exe"
  (ParentImage="*\\winword.exe" OR ParentImage="*\\excel.exe"
   OR ParentImage="*\\outlook.exe")
| eval severity="high"
| eval mitre_technique="T1059.001"
| collect index=notable_events
```

## Key Concepts

| Term | Definition |
|------|------------|
| **SPL (Search Processing Language)** | Splunk's query language for searching, filtering, transforming, and visualizing machine data |
| **CIM (Common Information Model)** | Splunk's field normalization standard that maps vendor-specific field names to common names for cross-source queries |
| **Notable Event** | An event in Splunk Enterprise Security flagged for analyst review based on a correlation search match |
| **Data Model** | Structured representation of indexed data in Splunk enabling accelerated searches and pivot-based analysis |
| **Sourcetype** | Classification label in Splunk that defines the format and parsing rules for a specific log type |
| **Correlation Search** | Scheduled Splunk search that runs continuously and generates notable events when conditions are met |
| **Timechart** | SPL command that creates time-series visualizations for identifying patterns, anomalies, and trends |

## Tools & Systems

- **Splunk Enterprise Security (ES)**: Premium SIEM application providing correlation searches, risk-based alerting, and investigation workbench
- **Splunk SOAR**: Orchestration platform integrated with Splunk ES for automated response playbooks
- **Sysmon**: Microsoft system monitoring tool providing detailed process, network, and file change telemetry ingested into Splunk
- **Splunk Attack Analyzer**: Automated threat analysis that detonates suspicious files and URLs, feeding results into Splunk
- **BOSS of the SOC (BOTS)**: SANS/Splunk training dataset for practicing incident investigation SPL queries

## Common Scenarios

### Scenario: Investigating Credential Stuffing Leading to Account Takeover

**Context**: Security operations receives an alert for multiple successful logins to a single account from geographically dispersed IP addresses within a 30-minute window.

**Approach**:
1. Query Event ID 4624 for the affected account to map all login sources and times
2. Correlate login IPs against threat intelligence feeds using a Splunk lookup table
3. Check proxy logs for suspicious activity from the authenticated sessions
4. Search for lateral movement from the compromised account (Event ID 4624 Type 3 to other hosts)
5. Build a timeline showing credential stuffing attempts, successful login, and post-compromise activity
6. Create a correlation search to detect similar patterns on other accounts

**Pitfalls**:
- Searching only the last 24 hours when the credential stuffing may have occurred over weeks
- Not checking for VPN logs that may show the same account authenticating from impossible travel distances
- Failing to normalize timestamps across log sources in different time zones

## Output Format

```
SPLUNK INVESTIGATION REPORT
============================
Incident:        INC-2025-1547
Analyst:         [Name]
Investigation Period: 2025-11-14 00:00 UTC - 2025-11-16 00:00 UTC

SEARCH SCOPE
Indexes:         windows, sysmon, proxy, firewall, dns
Hosts:           WKSTN-042, SRV-FILE01
Users:           jsmith, svc-backup
Source IPs:      10.1.5.42, 10.1.10.15

KEY FINDINGS
1. [timestamp] - Initial compromise via phishing (Sysmon Event 1)
2. [timestamp] - C2 established (proxy logs, beacon pattern detected)
3. [timestamp] - Credential theft (Sysmon Event 10, LSASS access)
4. [timestamp] - Lateral movement to SRV-FILE01 (Event 4624 Type 3)
5. [timestamp] - Data staging and exfiltration (proxy bytes_out anomaly)

SPL QUERIES USED
[numbered list of key queries with descriptions]

DETECTION GAPS IDENTIFIED
- No Sysmon deployed on SRV-FILE01 (blind spot)
- Proxy logs missing SSL inspection for C2 domain
- PowerShell ScriptBlock logging not enabled

RECOMMENDED DETECTIONS
1. Correlation search for Office-spawned PowerShell
2. Threshold alert for LSASS access patterns
3. Behavioral rule for beacon-interval network traffic
```