Analyzing Linux Audit Logs For Intrusion

---
name: analyzing-linux-audit-logs-for-intrusion
description: >
  Parse and analyze Linux auditd logs to detect intrusion indicators
  including unauthorized file access, privilege escalation, syscall
  anomalies, and suspicious process execution using ausearch and Python.
domain: cybersecurity
subdomain: log-analysis
tags: [auditd, linux-forensics, syscall-monitoring, intrusion-detection]
version: "1.0"
author: mahipal
license: Apache-2.0
---

# Analyzing Linux Audit Logs for Intrusion

## When to Use

- Investigating suspected unauthorized access or privilege escalation on Linux hosts
- Hunting for evidence of exploitation, backdoor installation, or persistence mechanisms
- Auditing compliance with security baselines (CIS, STIG, PCI-DSS) that require system call monitoring
- Reconstructing a timeline of attacker actions during incident response
- Detecting file tampering on critical system files such as `/etc/passwd`, `/etc/shadow`, or SSH keys

**Do not use** for network-level intrusion detection; use Suricata or Zeek for network traffic analysis. Auditd operates at the kernel level on individual hosts.

## Prerequisites

- Linux system with `auditd` package installed and the audit daemon running (`systemctl status auditd`)
- Root or sudo access to configure audit rules and query logs
- Audit rules deployed via `/etc/audit/rules.d/*.rules` or loaded with `auditctl`
- Recommended: Neo23x0/auditd ruleset from GitHub for comprehensive baseline coverage
- Familiarity with Linux syscalls (`execve`, `open`, `connect`, `ptrace`, etc.)
- Log storage with sufficient retention (default location: `/var/log/audit/audit.log`)

## Workflow

### Step 1: Verify Audit Daemon Status and Configuration

Confirm the audit system is running and check the current rule set:

```bash
# Check auditd service status
systemctl status auditd

# Show current audit rules loaded in the kernel
auditctl -l

# Show audit daemon configuration
cat /etc/audit/auditd.conf | grep -E "log_file|max_log_file|num_logs|space_left_action"

# Check if the audit backlog is being exceeded (dropped events)
auditctl -s
```

If the backlog limit is being reached, increase it:

```bash
auditctl -b 8192
```

### Step 2: Deploy Intrusion-Focused Audit Rules

Add rules that target common intrusion indicators. Place these in `/etc/audit/rules.d/intrusion.rules`:

```bash
# Monitor credential files for unauthorized reads or modifications
-w /etc/passwd -p wa -k credential_access
-w /etc/shadow -p rwa -k credential_access
-w /etc/gshadow -p rwa -k credential_access
-w /etc/sudoers -p wa -k privilege_escalation
-w /etc/sudoers.d/ -p wa -k privilege_escalation

# Monitor SSH configuration and authorized keys
-w /etc/ssh/sshd_config -p wa -k sshd_config_change
-w /root/.ssh/authorized_keys -p wa -k ssh_key_tampering

# Monitor user and group management commands
-w /usr/sbin/useradd -p x -k user_management
-w /usr/sbin/usermod -p x -k user_management
-w /usr/sbin/groupadd -p x -k user_management

# Detect process injection via ptrace
-a always,exit -F arch=b64 -S ptrace -F a0=0x4 -k process_injection
-a always,exit -F arch=b64 -S ptrace -F a0=0x5 -k process_injection
-a always,exit -F arch=b64 -S ptrace -F a0=0x6 -k process_injection

# Monitor execution of programs from unusual directories
-a always,exit -F arch=b64 -S execve -F exe=/tmp -k exec_from_tmp
-a always,exit -F arch=b64 -S execve -F exe=/dev/shm -k exec_from_shm

# Detect kernel module loading (rootkit installation)
-a always,exit -F arch=b64 -S init_module -S finit_module -k kernel_module_load
-a always,exit -F arch=b64 -S delete_module -k kernel_module_remove
-w /sbin/insmod -p x -k kernel_module_tool
-w /sbin/modprobe -p x -k kernel_module_tool

# Monitor network socket creation for reverse shells
-a always,exit -F arch=b64 -S socket -F a0=2 -k network_socket_created
-a always,exit -F arch=b64 -S connect -F a0=2 -k network_connection

# Detect cron job modifications (persistence)
-w /etc/crontab -p wa -k cron_persistence
-w /etc/cron.d/ -p wa -k cron_persistence
-w /var/spool/cron/ -p wa -k cron_persistence

# Monitor log deletion or tampering
-w /var/log/ -p wa -k log_tampering
```

Reload rules after editing:

```bash
augenrules --load
auditctl -l | wc -l   # Confirm rule count
```

### Step 3: Search for Intrusion Indicators with ausearch

Use `ausearch` to query the audit log for specific events:

```bash
# Search for all failed login attempts in the last 24 hours
ausearch -m USER_LOGIN --success no -ts recent

# Search for commands executed by a specific user
ausearch -ua 1001 -m EXECVE -ts today

# Search for all file access events on /etc/shadow
ausearch -f /etc/shadow -ts this-week

# Search for privilege escalation via sudo
ausearch -m USER_CMD -ts today

# Search for kernel module loading events
ausearch -k kernel_module_load -ts this-month

# Search for processes executed from /tmp (common attack staging)
ausearch -k exec_from_tmp -ts this-week

# Search for SSH key modifications
ausearch -k ssh_key_tampering -ts this-month

# Search events in a specific time range
ausearch -ts 03/15/2026 08:00:00 -te 03/15/2026 18:00:00

# Interpret syscall numbers and format output readably
ausearch -k credential_access -i -ts today
```

### Step 4: Generate Summary Reports with aureport

Use `aureport` to produce aggregate summaries for triage:

```bash
# Summary of all authentication events
aureport -au -ts this-week --summary

# Report of all failed events
aureport --failed --summary -ts today

# Report of executable runs
aureport -x --summary -ts today

# Report of all anomaly events
aureport --anomaly -ts this-week

# Report of file access events
aureport -f --summary -ts today

# Report of all events by key (maps to custom rule keys)
aureport -k --summary -ts this-month
```

### Step 5: Reconstruct the Attack Timeline

Combine ausearch queries to build a chronological narrative:

```bash
# Identify the initial access timestamp
ausearch -m USER_LOGIN -ua 0 --success yes -ts this-week -i | head -50

# Trace what the attacker did after gaining access
ausearch -ua <UID> -ts "03/15/2026 14:00:00" -te "03/15/2026 18:00:00" -i | aureport -f -i

# Extract all commands executed during the incident window
ausearch -m EXECVE -ts "03/15/2026 14:00:00" -te "03/15/2026 18:00:00" -i

# Check for persistence mechanisms installed
ausearch -k cron_persistence -ts "03/15/2026 14:00:00" -i
ausearch -k ssh_key_tampering -ts "03/15/2026 14:00:00" -i
```

### Step 6: Forward Audit Logs to SIEM

Configure `audisp-remote` or `auditbeat` to ship logs to a central SIEM:

```bash
# Using audisp-remote plugin (/etc/audit/plugins.d/au-remote.conf)
active = yes
direction = out
path = /sbin/audisp-remote
type = always

# Remote target (/etc/audit/audisp-remote.conf)
remote_server = siem.internal.corp
port = 6514
transport = tcp
```

## Key Concepts

| Term | Definition |
|------|------------|
| **auditd** | Linux Audit daemon that receives kernel audit events and writes to `/var/log/audit/audit.log` |
| **auditctl** | CLI to control the audit system: add/remove rules, check status, set backlog size |
| **ausearch** | Query tool for searching audit logs by message type, user, file, key, or time range |
| **aureport** | Reporting tool that generates aggregate summaries for triage and compliance |
| **audit rule key (-k)** | User-defined label on audit rules for fast filtering with ausearch and aureport |
| **augenrules** | Merges `/etc/audit/rules.d/` files into `audit.rules` and loads into the kernel |

## Verification

- [ ] auditd is running and rules are loaded (`auditctl -l` returns expected rule count)
- [ ] No audit backlog overflow (`auditctl -s` shows lost: 0)
- [ ] ausearch returns events for each custom key
- [ ] aureport generates non-empty summaries for authentication, executable, and file events
- [ ] Timeline reconstruction produces a coherent chronological sequence
- [ ] Critical file watches trigger alerts on test modifications
- [ ] Logs are forwarding to central SIEM
- [ ] Audit rules persist across reboot (rules in `/etc/audit/rules.d/`)