Security

---
name: log-analysis
description: Parsing structured logs, finding patterns in noise, and ELK/Splunk querying.
triggers: [log analysis]
tags: [security]
role: ops-monitor
context_cost: low
---
# log-analysis Skill

## Goal
This skill helps agents and engineers interpret logging data to find root causes of issues.

## Steps
## 1. Structured Logging (The Prerequisite)
- **Format**: JSON. Always.
- **Why**: Allows querying `level="ERROR" AND service="user-api"`. Text logs require fragile regex.
- **Context**: Every log must have `trace_id`, `user_id`, and `environment`.

## 2. Analysis Workflow

1.  **Scope the Timeframe**: "Last 15 minutes" or "Since deployment at 14:00".
2.  **Filter by Level**: Start with `ERROR` and `FATAL`. Ignore `INFO` initially.
3.  **Group by Message**:
    - **Bad**: Reading 1000 lines of `Connection timeout`.
    - **Good**: "Count of errors by message type: Timeout (800), Auth Fail (200)".
4.  **Trace Correlation**: take one `trace_id` from an error and filter ALL logs (inc INFO) for that ID to see the sequence of events leading to failure.

## 3. Query Cheatsheet

### Loki (LogQL)
```logql
{app="frontend"} |= "error" | json | latency > 500ms
```

### Elasticsearch / Kibana (Lucene)
```lucene
service:backend AND level:ERROR AND NOT message:"healthcheck"
```

### Splunk
```splunk
index=prod sourcetype=k8s | stats count by message | sort - count
```

### Command Line (jq)
```bash
cat logs.json | jq 'select(.level=="error") | .message' | sort | uniq -c
```

## 4. Red Flags ("Smells")
- **"Swallowed Exception"**: Logs that say "Error occurred" but print no stack trace.
- **"Noise"**: Periodic errors (e.g., every 5 mins) that everyone ignores. These mask real issues.
- **"Sensitive Data"**: Passwords or PII in logs. -> **CRITICAL**: Report immediately.

## Security & Guardrails

### 1. Skill Security (Log Analysis)
- **Mandatory PII Scrubbing**: The agent must utilize `Rebertha` or an equivalent dedicated regex engine to proactively redact SSNs, Credit Card numbers, Auth Tokens, and Passwords BEFORE it writes the log summary to a file or transmits it for analysis. A failure in the redaction layer must cause the log analysis to fail-closed.
- **Log Forgery Resilience**: The agent must trust the centralized log aggregation server (Splunk/ELK), but it must NOT implicitly trust the contents of the `message` payload. It must recognize the vulnerability of Log Injection (e.g., an attacker injecting `\n [INFO] User Admin Authenticated`) and prioritize `trace_id` and cryptographically signed application context.

### 2. System Integration Security
- **Access Control on Audit Logs**: The agent must verify that the systems generating high-stakes logs (auth, financial transactions) are forwarding them to an immutable, WORM (Write-Once-Read-Many) storage bucket. If the agent detects that the application layer has `DELETE` permissions on an audit index, it must raise a sev-1 alarm.
- **Alert Fatigue Exploitation**: An attacker may flood the system with low-severity `WARN` logs to mask a stealthy exfiltration attack occurring under the radar. The agent must implement rate-anomaly detection: if the volume of a specific log spikes by 1000%, it must condense the alert to prevent SOC burnout but escalate the anomaly.

### 3. LLM & Agent Guardrails
- **Hallucinated Root Cause**: In the presence of highly ambiguous logs, the LLM might confidently hallucinate a plausible but incorrect root cause (e.g., blaming a DNS failure when it was actually a subtle authorization bug). The agent must qualify its findings with confidence scores and cite the exact log line (via `trace_id`) that supports its hypothesis.
- **Malicious Query Injection**: An attacker with access to the agent's prompts might attempt to inject a destructive Splunk query or perform unconstrained searches (e.g., `* | delete` or searching across all indexes to exfiltrate data). The agent must execute LogQL/Lucene queries under a strictly scoped, read-only service account.