Security Audit Log Design

# Audit Log Design

> Log the who, what, when, where, and outcome of every security-relevant event in a
> structured, tamper-evident format that enables both real-time detection and forensic
> reconstruction

## When to Use

- Designing the logging infrastructure for a new application or service
- Adding security event logging to an existing system that lacks audit trails
- Preparing for SOC2, ISO 27001, HIPAA, or PCI-DSS compliance audits
- Investigating a security incident and discovering logging gaps
- Choosing between application-level and infrastructure-level audit logging
- Designing tamper-evident log storage for high-assurance environments

## Threat Context

Attackers who gain access to a system routinely attempt to cover their tracks by deleting or
modifying logs. Log tampering is not theoretical -- it is standard attacker tradecraft:

- **SolarWinds (2020)**: The attackers specifically targeted logging and monitoring
  infrastructure to evade detection, disabling security tools and modifying logs to remove
  evidence of their presence. They remained undetected for 9+ months.
- **Detection time without logging**: The IBM/Ponemon Cost of a Data Breach Report
  consistently finds that organizations without adequate security logging take 200+ days to
  detect a breach. Organizations with security AI and automation (which depends on structured
  log data) detect breaches 108 days faster.
- **Compliance requirements**: SOC2 Trust Services Criteria CC7.2 requires monitoring of
  system components for anomalies. PCI-DSS Requirement 10 mandates audit trails for all
  access to cardholder data. HIPAA requires audit controls for electronic protected health
  information. ISO 27001 A.12.4 requires event logging, protection of log information, and
  administrator/operator logs. Failure to maintain adequate logs results in audit findings,
  regulatory penalties, and inability to investigate incidents.
- **Legal and forensic value**: Audit logs are legal evidence. In breach litigation,
  regulators and courts expect timestamped, tamper-evident records of what happened. Without
  them, the organization cannot demonstrate due diligence or reconstruct the incident
  timeline.

Logging is not optional overhead -- it is the immune system of the application. A system
without audit logging is a system where breaches are invisible.

## Instructions

1. **Define the security-relevant events that must be logged.** At minimum, capture these
   event categories:
   - **Authentication events**: Login success, login failure (with username attempted),
     logout, MFA challenge issued, MFA challenge success/failure, password change, password
     reset request, account lockout
   - **Authorization events**: Access granted, access denied, permission change, role
     assignment, role revocation, privilege escalation attempt
   - **Data access events**: Read of sensitive data (PII, financial, health), bulk data
     export, bulk query execution, data download
   - **Administrative actions**: User creation, user deletion, user modification, role
     creation, configuration change, feature flag change, deployment, certificate rotation
   - **System events**: Application startup, application shutdown, dependency failure,
     database migration, backup execution
   - **Anomalous events**: Rate limit triggered, input validation rejection, WAF block,
     unexpected error, authentication from new device/location

2. **Use a structured event format for every audit entry.** Every audit event must include
   these fields:
   - `timestamp`: ISO 8601 with timezone, always UTC (e.g., `2024-01-15T14:30:00.123Z`)
   - `event_type`: Enumerated, dot-separated category (e.g., `auth.login.success`,
     `authz.permission.denied`, `data.export.initiated`)
   - `actor`: Who performed the action -- user ID, service account name, or `system` for
     automated processes
   - `action`: The verb -- `create`, `read`, `update`, `delete`, `login`, `logout`,
     `export`, `deny`
   - `resource`: What was acted upon -- resource type and identifier (e.g., `user:u-12345`,
     `order:o-67890`, `config:feature-flags`)
   - `outcome`: `success` or `failure`, with `reason` for failures (e.g.,
     `insufficient_permissions`, `invalid_credentials`, `rate_limited`)
   - `source_ip`: The IP address of the request origin
   - `session_id`: Session or request correlation ID for linking related events
   - `metadata`: Additional context -- user agent, request path, changed fields for update
     operations, previous and new values for configuration changes

3. **Log at the application layer, not just infrastructure.** Infrastructure logs (web
   server access logs, firewall logs, load balancer logs) capture network-level events but
   miss application-level semantics. The infrastructure sees
   `GET /api/customers?limit=99999 200 OK`. The application knows "User 123 exported all
   50,000 customer records to CSV." Both are needed; the application event is far more
   actionable for security investigation. Implement audit logging as a first-class
   application concern, not an afterthought bolted onto HTTP middleware.

4. **Make logs tamper-evident.** An attacker who compromises the application server will
   attempt to delete or modify logs to cover their tracks. Tamper evidence ensures
   modifications are detectable:
   - **Write-once storage**: S3 Object Lock (Governance or Compliance mode), Azure Immutable
     Blob Storage, GCP Bucket Lock, WORM (Write Once Read Many) storage. Once written, log
     entries cannot be modified or deleted for the retention period.
   - **Hash chaining**: Each log entry includes
     `entry_hash = SHA-256(entry_data + previous_entry_hash)`. The first entry uses a known
     sentinel value. Any modification, deletion, or insertion of entries breaks the hash
     chain from that point forward.
   - **Real-time forwarding**: Forward logs to an independent, append-only log aggregator
     (Splunk, Elasticsearch/OpenSearch, CloudWatch Logs, Datadog) in real time via syslog,
     Fluentd, or direct API integration. The application server's local logs are not the
     sole copy.
   - **Digital signatures**: Sign log batches with a key held in an HSM or KMS. Provides
     non-repudiation -- the logs can be verified as authentic even if the logging
     infrastructure is later compromised.

5. **Never log secrets or PII unnecessarily.** Audit logs must not contain passwords, API
   keys, session tokens, credit card numbers, social security numbers, or other sensitive
   data. Log the fact that an action occurred, not the sensitive content. Correct: "User
   u-12345 updated their password at 2024-01-15T14:30:00Z." Incorrect: "User u-12345
   changed password from 'oldpass123' to 'newpass456'." For PII, log only the minimum needed
   for investigation -- user ID rather than full name and email, unless specific compliance
   requirements mandate otherwise. Implement log scrubbing as a safety net: regex-based
   filters that detect and redact patterns matching known secret formats before log entries
   are written.

6. **Define retention and archival policies aligned with compliance requirements.** Common
   minimums: SOC2 requires 1 year of audit log retention. PCI-DSS requires 1 year with 3
   months immediately accessible for analysis. HIPAA requires 6 years. GDPR does not specify
   a retention period but requires that retention be justified and proportionate. Archive
   older logs to cold storage (S3 Glacier, Azure Cool/Archive Blob) to manage costs, but
   ensure archived logs remain searchable and verifiable. Test restoration from cold storage
   regularly -- logs that cannot be retrieved when needed are equivalent to no logs.

7. **Alert on high-severity events in real time.** Logs that are only read during
   post-incident investigation provide forensic value but no detection value. Configure
   real-time alerts for: multiple failed login attempts from a single IP or against a single
   account (brute force, credential stuffing), privilege escalation events, access to
   sensitive data outside normal patterns (time of day, volume, geographic location),
   administrative actions from unusual source IPs, any `outcome: failure` on critical
   operations (payment processing, data export), and new device/location for privileged
   accounts. Route alerts to the security team's incident response channel (PagerDuty,
   Slack, email) with enough context to triage without reading raw logs.

## Details

- **The OWASP Logging Cheat Sheet event categories**: Authentication (success, failure,
  lockout), Authorization (access granted, denied, permission changes), Session management
  (creation, destruction, timeout, concurrent session detection), Input validation failures
  (rejected input, WAF blocks, malformed requests), Application errors (unhandled
  exceptions, dependency failures, timeout events), High-value transactions (financial
  operations, data exports, bulk operations, configuration changes). Use this as a starting
  checklist and add domain-specific events for your application.

- **Structured logging format example**: A complete JSON audit event:

  ```json
  {
    "timestamp": "2024-01-15T14:30:00.123Z",
    "event_type": "authz.permission.denied",
    "actor": { "user_id": "u-123", "session_id": "s-456", "service": "order-api" },
    "action": "delete",
    "resource": { "type": "order", "id": "o-789" },
    "outcome": "failure",
    "reason": "insufficient_permissions",
    "source_ip": "198.51.100.42",
    "user_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)",
    "request_id": "req-abc-def",
    "metadata": {
      "required_permission": "order:delete",
      "actual_permissions": ["order:read", "order:update"]
    }
  }
  ```

  Use an enumerated `event_type` taxonomy defined in code so events can be queried
  consistently across services. Avoid free-text event descriptions that vary between
  developers.

- **Tamper evidence with hash chaining in detail**: Each log entry includes:
  `entry_hash = SHA-256(canonical_json(entry_data) + previous_entry_hash)`. The first entry
  uses a well-known sentinel (e.g., `SHA-256("GENESIS")`) as the "previous hash."
  Verification: recompute the chain from any starting point. If any entry has been modified,
  deleted, or inserted, all subsequent hashes fail verification. Store periodic "anchor
  hashes" in an independent system (a separate database, a blockchain, a signed timestamping
  service) so that the integrity of the chain can be verified even if the primary log store
  is compromised. This technique is used by AWS CloudTrail log file integrity validation.

- **Log levels vs audit events -- they are different concerns**: Application log levels
  (DEBUG, INFO, WARN, ERROR) serve operational troubleshooting. Audit events record
  security-relevant business events regardless of severity. A successful login is not an
  "error" or a "warning" -- it is a security event that must be recorded. Implement audit
  logging as a separate subsystem with its own transport, storage, and retention. Do not
  rely on application log level configuration to control audit event emission -- changing the
  log level to WARN to reduce noise must never suppress security audit events.

## Anti-Patterns

1. **No logging of failed authentication attempts.** Failed logins are the single most
   important signal for detecting brute force attacks, credential stuffing campaigns, and
   account takeover attempts. An application that only logs successful logins is blind to the
   10,000 failed attempts that preceded the one successful compromise.

2. **Logging everything at DEBUG level in production.** Produces such volume that security
   events are buried in noise. Debug logging also risks exposing sensitive data -- request
   bodies containing passwords, internal state containing encryption keys, SQL queries
   containing user data. Use structured audit events for security concerns, not verbose debug
   output. Keep DEBUG logging disabled in production or restricted to specific components
   during active troubleshooting.

3. **Logs stored only on the application server.** If the server is compromised, the
   attacker deletes the local logs and the evidence is gone. Always forward logs to an
   independent aggregator in real time. The local log file on the application server should
   be treated as a buffer, not as the permanent record.

4. **No correlation IDs across services.** In a distributed system, a single user action may
   span multiple services. Without a request or correlation ID propagated through all service
   calls and included in every audit event, reconstructing an attack timeline requires manual
   timestamp correlation across dozens of log sources -- slow, error-prone, and often
   impossible when clocks are skewed. Propagate a correlation ID (via HTTP header, message
   metadata, or distributed tracing context) and include it in every audit event.

5. **Audit logging as an afterthought, added post-launch.** The most vulnerable period for
   an application is immediately after launch -- configuration is fresh, security hardening
   is incomplete, and the team is still learning the system's behavior. Adding audit logging
   weeks or months after launch means this critical period has no audit trail. Design and
   implement audit logging alongside the feature it monitors, shipping them together.

6. **Logging PII in violation of privacy regulations.** Audit logs that contain full names,
   email addresses, phone numbers, IP addresses (considered PII under GDPR), or health
   information create a secondary data protection liability. The audit log itself becomes a
   sensitive data store requiring access controls, encryption, and data subject access
   request handling. Log the minimum identifiers needed (user IDs, resource IDs) and join
   with the primary data store only when investigation requires it.

7. **Alert fatigue from poorly tuned thresholds.** Alerting on every failed login produces
   thousands of alerts per day, all ignored. Tune alert thresholds to reduce false positives:
   alert on 10+ failed logins from a single IP in 5 minutes, not on individual failures. Use
   progressive escalation: first alert goes to a Slack channel, repeated alerts page the
   on-call engineer. Review and adjust thresholds monthly based on alert-to-incident ratio.