Security Secrets Lifecycle

# Secrets Lifecycle

> Secrets are born (generated), distributed (delivered to consumers), rotated (replaced on
> schedule), and die (revoked and destroyed) -- manage every phase or the secret manages you

## When to Use

- Designing how an application retrieves database credentials, API keys, or encryption keys
- Establishing a secret rotation policy for a production system
- Responding to a secret exposure (leaked in logs, committed to Git, visible in error
  messages)
- Choosing between static secrets, dynamic secrets, and short-lived tokens
- Auditing an existing system for secret management gaps
- Evaluating secrets managers (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager)

## Threat Context

Exposed secrets are the most common initial access vector in cloud breaches. The attack
surface is vast and the consequences are immediate:

- **GitHub secret exposure scale**: GitHub's 2023 secret scanning report found 12+ million
  secrets exposed in public repositories in a single year. Automated bots scan every public
  commit within seconds of push -- the median time from commit to attacker exploitation of
  an exposed AWS key is under 1 minute.
- **Uber breach (2022)**: Began with hardcoded credentials discovered in a PowerShell script
  on an internal network share. The attacker used these credentials to access Uber's AWS,
  GCP, and internal Slack environments.
- **CircleCI breach (2023)**: Attackers compromised CircleCI's secrets management
  infrastructure itself, exposing customer secrets stored in the platform. This demonstrates
  that even the secrets manager is a high-value target.
- **Codecov supply chain attack (2021)**: Attackers modified a bash uploader script to
  exfiltrate environment variables (containing secrets) from CI/CD environments to an
  attacker-controlled server.

Secrets have a lifecycle -- generation, distribution, usage, rotation, and revocation -- and
failure at any single phase is a breach waiting to happen. A secret that is generated
securely but never rotated accumulates exposure risk over time. A secret that is rotated
regularly but distributed through insecure channels is exposed at every rotation.

## Instructions

1. **Generate secrets with sufficient entropy.** API keys, tokens, and passwords must
   contain minimum 256 bits of randomness from a cryptographically secure pseudorandom
   number generator (CSPRNG). Use `crypto.randomBytes(32)` in Node.js,
   `secrets.token_hex(32)` in Python, `SecureRandom` in Java, or `/dev/urandom` on Unix.
   Database passwords should be 32+ random characters (letters, digits, symbols). Encryption
   keys must be generated using the crypto library's dedicated key generation function, not
   by hashing a passphrase (unless using a proper KDF like Argon2id with appropriate
   parameters -- see `security-credential-storage`).

2. **Distribute secrets through a secrets manager, never through code or config files.** Use
   HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, or Kubernetes
   Secrets (with envelope encryption via a KMS). Applications retrieve secrets at startup or
   on-demand via authenticated API calls to the secrets manager. The application
   authenticates to the secrets manager using platform identity (IAM roles, Kubernetes
   service accounts, Vault AppRole) -- not with another static secret. Secrets must never be
   committed to version control, embedded in Docker images, passed as build arguments, or
   included in deployment manifests.

3. **Use dynamic secrets when possible.** Instead of a static database password that lives
   forever, use Vault's database secrets engine (or equivalent) to generate short-lived
   credentials with a TTL of 1 hour per application instance. When the TTL expires, the
   credential is automatically revoked at the database level. This eliminates long-lived
   secrets entirely -- there is nothing to rotate because every credential is born with a
   death date. Dynamic secrets are available for databases (PostgreSQL, MySQL, MongoDB),
   cloud providers (AWS STS, GCP service account keys), and PKI certificates.

4. **Rotate static secrets on a mandatory schedule.** When dynamic secrets are not feasible,
   static secrets must be rotated periodically: 90 days maximum for high-sensitivity secrets
   (database root credentials, encryption keys, signing keys), 365 days for low-sensitivity
   secrets (monitoring API keys). Implement dual-read rotation to avoid downtime:
   (a) generate new secret B while secret A is still active, (b) configure all consumers to
   accept both A and B, (c) switch producers to emit B, (d) verify via audit logs that no
   traffic uses A, (e) revoke A. Automate this process -- manual rotation is forgotten
   rotation.

5. **Revoke secrets immediately upon any exposure.** If a secret appears in a log file,
   error message, Git commit, Slack message, support ticket, or any unintended location,
   treat it as compromised. Do not assess whether "anyone noticed" -- assume the worst.
   Revocation procedure: (a) revoke the secret at the source (delete the API key, change the
   password, rotate the certificate), (b) generate a new secret and distribute it through
   the secrets manager, (c) audit access logs from the moment of exposure to determine if
   the secret was exploited, (d) investigate and fix the root cause of the exposure, (e) add
   detection to prevent recurrence (pre-commit hooks, log scrubbing).

6. **Audit all secret access.** The secrets manager must log every access event: who
   requested the secret, when, from which IP/service, and whether the request was granted or
   denied. Configure alerts for anomalies: access from unknown IP ranges, access frequency
   spikes, access to secrets outside the caller's normal pattern, and any access to
   high-sensitivity secrets (root credentials, encryption keys). Audit logs from the secrets
   manager are themselves sensitive and must be stored in a tamper-evident,
   access-controlled location (see `security-audit-log-design`).

7. **Scope secrets to minimum blast radius.** Each service or application instance should
   have its own credentials. Do not share a single database password across 10
   microservices -- if one service is compromised, the attacker has access to the database
   with the privileges of all 10 services. Service-specific credentials enable targeted
   revocation: revoke the compromised service's credentials without disrupting the other 9.
   Apply the principle of least privilege to secret scope: a service that only reads from the
   database should have read-only credentials, not read-write.

## Details

- **The secret exposure cascade**: A leaked AWS access key committed to a public GitHub
  repository triggers a predictable cascade. Automated bots (scanning every public commit
  via GitHub's Events API) detect the key within seconds. The attacker uses the key to call
  `sts:GetCallerIdentity` to identify the account, then `iam:ListUsers`,
  `s3:ListBuckets`, and `ec2:DescribeInstances` to enumerate resources. Data is exfiltrated
  from S3 buckets. EC2 instances are launched for cryptomining. The entire cascade can
  complete within 10 minutes of the commit. Response must be: (1) revoke the key immediately
  via `aws iam delete-access-key`, (2) audit CloudTrail for all API calls made with the key
  since the commit timestamp, (3) rotate all credentials the leaked key could access,
  (4) check for persistence mechanisms the attacker may have installed (new IAM users,
  Lambda functions, EC2 instances), (5) add pre-commit hooks (truffleHog, git-secrets,
  gitleaks) and enable GitHub push protection to prevent recurrence.

- **Dual-read rotation pattern in practice**: Step 1: Generate new database password B.
  Step 2: Execute `ALTER USER app_user SET PASSWORD = B` but also keep A valid (PostgreSQL
  allows this via connection pooler configuration or multiple valid passwords in some
  setups; alternatively, create a new database user with B and grant identical permissions).
  Step 3: Update the secrets manager to serve B. Step 4: Applications pick up B on their
  next secret refresh (within the refresh interval, typically 5-15 minutes). Step 5: Monitor
  audit logs to confirm no connections use A. Step 6: Revoke A (drop the old user or change
  the password). The critical principle: there must never be a moment when neither A nor B
  is valid.

- **Short-lived tokens vs static secrets -- a hierarchy of preference**: Prefer (from best
  to worst): (1) Dynamic secrets with automatic TTL and revocation (Vault database secrets
  engine, AWS STS temporary credentials), (2) OAuth2 access tokens with 1-hour TTL and
  refresh tokens, (3) Platform-managed rotating credentials (AWS RDS IAM authentication,
  GCP Cloud SQL IAM authentication), (4) Static secrets with automated rotation on a 90-day
  schedule, (5) Static secrets with manual rotation (worst case, requires process
  discipline). Each level down the hierarchy increases the window of opportunity for a
  compromised credential.

- **Pre-commit secret scanning**: Tools: truffleHog (entropy-based and regex-based
  detection), git-secrets (AWS-focused patterns), gitleaks (broad pattern library), GitHub's
  built-in push protection (blocks pushes containing known secret formats for supported
  providers). Configure as both a local pre-commit hook (catches secrets before they reach
  the remote) and a CI check (catches secrets that bypass local hooks). Block commits
  containing high-entropy strings matching known secret patterns: AWS keys (prefix `AKIA`),
  GitHub tokens (`ghp_`, `gho_`, `ghs_`), Slack tokens (`xoxb-`, `xoxp-`), private keys
  (`-----BEGIN RSA PRIVATE KEY-----`). Maintain an allowlist for false positives (test
  fixtures, documentation examples with dummy values).

## Anti-Patterns

1. **Secrets in source code or version control.** Once committed, a secret persists in Git
   history even after deletion from HEAD. Every clone of the repository contains the secret
   in its history. Removing it requires `git filter-repo` or BFG Repo Cleaner to rewrite
   history, followed by force-pushing -- and even then, any existing clone or fork retains
   the secret. Always treat a committed secret as compromised regardless of how quickly it
   was removed.

2. **Shared credentials across services and environments.** Production and staging sharing
   the same database password. Multiple microservices using the same API key. A single
   "admin" password for all team members. Compromise of any one consumer compromises all
   consumers. Incident response requires rotating the shared credential and redeploying
   every consumer simultaneously -- an operational nightmare that delays response.

3. **Never-rotated secrets.** A database password created in 2019 and never rotated has had
   5+ years of potential exposure windows: developer laptops, CI/CD logs, backup tapes,
   departed employees' password managers. The longer a secret lives, the more opportunities
   for undetected exposure. Establish rotation schedules, automate them, and alert on
   overdue rotations.

4. **Secrets in environment variables without a secrets manager.** Environment variables are
   visible in process listings (`/proc/PID/environ` on Linux), crash dumps, debugging tools
   (`docker inspect`), container orchestration dashboards, and CI/CD build logs. They are
   better than hardcoding in source but far worse than a proper secrets manager. Use a
   secrets manager to inject secrets directly into the application's memory at startup,
   bypassing the environment entirely.

5. **Logging secrets accidentally.** Application logs that include full HTTP request/response
   bodies (containing Authorization headers or API keys), error messages that dump
   configuration objects (containing database connection strings), and debug logging that
   prints environment variables. Implement structured logging with explicit field selection
   (log only what you intend), add log scrubbing for known secret patterns, and never log
   request bodies containing authentication credentials.

6. **Secrets in CI/CD pipeline definitions.** Storing secrets as plaintext variables in
   GitHub Actions workflow files, Jenkins pipeline scripts, or GitLab CI YAML. Use the CI
   platform's encrypted secrets feature (GitHub Actions secrets, GitLab CI/CD variables
   marked as "masked" and "protected") and ensure secrets are not echoed in build logs.

7. **Using the same secret for authentication and encryption.** An API key used for both
   authenticating to a service and encrypting data at rest. If the authentication use case
   exposes the key (via logs, network capture), the encryption is also compromised. Use
   separate secrets for separate purposes -- key separation is a fundamental cryptographic
   principle.