Sealed Secrets

---
name: sealed-secrets
description: Sealing, rotating, and troubleshooting SealedSecrets in the K3s cluster — file naming rules, placeholder base64 pitfalls, seal-argocd-dex subcommands, pod restart requirements after reseal.
---

# Sealed Secrets

Bitnami sealed-secrets is the only commit-safe way to ship Kubernetes
Secrets through this repo. `kubeseal` reads the live cluster's public
key and encrypts secret data into a committed `*-secret.yaml` file that
the in-cluster controller decrypts into a real `Secret`.

`kubeseal` is one of the few commands allowed to touch the live cluster
directly (see CLAUDE.md hard rules).

## File naming

- SealedSecret files **must be named `*-secret.yaml` (singular)**.
  `*-secrets.yaml` is not in the `.gitleaks.toml` allowlist — the
  pre-commit hook will block the commit with no obvious reason.

## The placeholder base64 foot-gun

**Never commit a `*-secret.yaml` with placeholder text like `REPLACE_ME`
in `encryptedData`.** The chain of silent failures is brutal:

1. Controller fails to decrypt: `illegal base64 data`
2. The real `Secret` is never created
3. Any Prometheus operator CR (Alertmanager / Prometheus) referencing
   the missing Secret via `secrets:` silently **omits the volume mount**
4. You get runtime errors that don't obviously point at the placeholder

Always seal a real value immediately, or **omit the file entirely**
until you have one. If you must stub it out, use a short real string
and reseal later — never a literal `REPLACE_ME`.

## `scripts/seal-argocd-dex` — per-secret subcommands

Running `seal-argocd-dex` bare (or with `all`) re-seals everything and
prompts for GitHub creds + Slack webhook. To rotate just one secret
without re-entering unrelated values, use a subcommand:

```
seal-argocd-dex [github|argocd|monitor|grafana|open-webui|slack]
```

Subcommands read existing keys from the running `argocd-dex-secret` to
preserve values they don't touch, so `all` must have been run at least
once before a subcommand will work.

## Pod restarts after reseal

Pods that read secret values via `envFrom` or `secretKeyRef` **cache
values at pod startup**. After re-sealing, running pods keep stale
values and OAuth/service login fails with errors like
`invalid client_secret` — even though the Kubernetes `Secret` object
itself is correct.

`seal-argocd-dex` now restarts affected pods automatically. If you
re-seal through any other path (manual `kubeseal`, rebuild playbook,
etc.), you must restart pods in these namespaces:

- `argo-cd` / `argocd-monitor` (Dex: `just restart-dex`)
- `monitoring` (`kubectl rollout restart sts grafana-prometheus -n monitoring`)
- `open-webui`
- `headlamp`

## `cookie-secret` generation

oauth2-proxy's cookie secret must be **exactly 16, 24, or 32 bytes**
for the AES cipher. Regressions have happened twice:

- `base64.b64encode(token_bytes(32))` → 44 chars → oauth2-proxy crash
- Use `secrets.token_hex(16)` (32 hex chars = 32 bytes) — this is what
  `scripts/seal-argocd-dex` does. Do **not** change it.

## Non-deterministic re-encryption

`kubeseal` adds random padding, so re-sealing the same plaintext
produces a different ciphertext each run. This is expected. Don't try
to diff-match sealed ciphertexts — diff the underlying plaintext via
the generator script or compare decrypted `Secret` contents on-cluster.

## Key files

- `scripts/seal-argocd-dex` — main Dex/OAuth sealing entry point
- `scripts/seal-from-json` — batch sealing from `/tmp/cluster-secrets/`
- `scripts/create-prometheus-admission-secret` — one-shot webhook TLS
- `kubernetes-services/additions/*/`-secret.yaml` — committed sealed secrets