Secrets architecture reference for node-template — when to use OpenBao+ESO vs GitHub env secrets, which operation pattern fits which write/rotate/add flow, the load-bearing invariants from the spec, the YAML catalog + Zod-loader the script consumes, and where to find the canonical implementation. Use when adding/rotating a secret, designing a service that consumes secrets, debugging an ExternalSecret or writer-role login, deciding between substrate and Compose-infra routing, touching `pnpm se...
Scanned 9/9/2026
Install to Claude Code
Skills Directorynpx -y skills add cogni-dao/cogni --skill cicd-secrets-expert --agent claude-codeInstalls into .claude/skills of the current project.
Are you the author of Cicd Secrets Expert?
Add the live security badge to your README — it updates automatically with every re-scan.
[](https://www.skillsdirectory.com/skills/cogni-dao-cicd-secrets-expert)More formats (shields.io, HTML) on the badges page.
---
name: cicd-secrets-expert
description: "Secrets architecture reference for node-template — when to use OpenBao+ESO vs GitHub env secrets, which operation pattern fits which write/rotate/add flow, the load-bearing invariants from the spec, the YAML catalog + Zod-loader the script consumes, and where to find the canonical implementation. Use when adding/rotating a secret, designing a service that consumes secrets, debugging an ExternalSecret or writer-role login, deciding between substrate and Compose-infra routing, touching `pnpm secrets:set` / `scripts/secrets/` / `scripts/lib/secrets-catalog-loader.ts` / `nodes/<node>/.cogni/secrets-catalog.yaml` / `infra/secrets-catalog.yaml` / `infra/k8s/argocd/{openbao,external-secrets}/` / per-node ExternalSecret manifests, or evaluating any new workflow that touches secret values. Triggers: 'add a secret', 'add a node secret', 'rotate a key', 'OpenBao', 'ESO', 'ExternalSecret', 'secrets-catalog', 'catalog tier', 'A1', 'A2', 'B-tier', 'writer role', 'bao login', 'vault-action', 'vault-config-operator', 'secrets-manage', 'secret in GH env vs OpenBao', 'where do I put this credential', 'per-node catalog'."
---
# CI/CD Secrets Expert
One-page reference for anyone touching secrets in node-template. Read this BEFORE the spec; this points at what to actually read.
## Node self-serve secrets — the value-write path
A node-owner sets a secret with only an API key — `POST <env-host>/api/v1/nodes/<id>/secrets {env, key, value}` → `200 written, version`, written to `cogni/<env>/<node>/<KEY>`, then ESO + Reloader carry it into the pod.
**The node-dev contract + the LIVE per-env status are single-sourced in the hub guide `node-self-serve-secrets`** (`GET /api/v1/knowledge/node-self-serve-secrets`) — not snapshotted in git. Design canon: [`docs/design/node-self-serve-secrets.md`](../../../docs/design/node-self-serve-secrets.md). The durable, non-obvious facts (why this skill exists):
- **Per-env sealed operators — NO cross-env write.** Each env runs its own operator pod; the OpenBao policy (`cogni/<env>/*`, `_system`/`_shared` denied) makes a candidate-a token physically unable to write prod. The auth chain: operator pod's **projected SA token** (`audience: cogni-openbao`) → `auth/kubernetes/login` (role `<env>-node-secrets-writer`, SA `operator-secrets-writer@cogni-<env>`) → KV put/patch. No kube/SSH; caller holds only an API key.
- **RBAC is granular: `can_manage_secrets ← secrets_manager`, NOT `developer`.** `developer → can_flight` (logs/flight); `secrets_manager` is a **distinct** grant for secret writes (both ∪ `admin`). Requesting `developer` and expecting secret-write is the trap — 403 `authz_denied` is the right answer to the wrong role.
- **`env` is an explicit validated body param (D1, #1754)**, == the operator's own `DEPLOY_ENVIRONMENT` else `409 wrong_operator_env`. Node resolved via `resolveNodeRef` (status-agnostic) through the shared `withNodeRbac` seam (`src/app/_lib/node-rbac.ts`, #1771).
- **Per-env provisioning + the three 5xx (diagnose by status, not by guess):** the write needs, per env, the `<env>-node-secrets-writer` OpenBao role + the operator overlay (`OPENBAO_NODE_SECRETS_WRITER_ROLE` via `envFrom` + a projected SA token, `audience: cogni-openbao`) + Reloader present + the OpenFGA model carrying the checked relation. Failure modes: **`503 secrets_plane_config_missing`** = `OPENBAO_NODE_SECRETS_WRITER_ROLE` unset on the pod; **`502` at write** = the OpenBao role/login is missing (vs the env var); **`503 authz_unavailable`** = the env's OpenFGA **model lacks the checked relation**. That last one is a recurring **silent-drift class** (`task.5049`): `bootstrap-openfga.sh` only runs inside `deploy-infra` (`skip_infra=false`), so normal promotes never re-bootstrap — a new RBAC relation (e.g. `can_manage_secrets`, #1627) silently never reaches an env, and secrets `503` while older actions `403`. Fix: POST the current `rbac-model.json` to that env's OpenFGA store + repoint `cogni/<env>/operator` `OPENFGA_AUTHORIZATION_MODEL_ID` + let Reloader roll the operator. (Live per-env readiness: recall the hub guide, above.)
- **`source: agent` keys are off-limits** (denylist `node-secrets-reserved.data.ts`, #1753) — they're substrate-minted per node; clobbering one breaks the node.
- **Control-plane framing:** `OperatorSecretsPlanePort` is the secrets row of the typed operator control plane ([`cicd-platform-boundary.md`](../../../docs/spec/cicd-platform-boundary.md)) — sibling of `OperatorDeployPlanePort`; one port per substrate (Argo/OpenBao/Cherry-Akash), unified by the `(node_id, env)+OpenFGA` contract. A **node is a bundle of services** (`node → services → deployments`, [`node-baas-architecture.md`](../../../docs/spec/node-baas-architecture.md)), not one deployable.
## North star
[`proj.agentic-fork-bootstrap`](../../../work/projects/proj.agentic-fork-bootstrap.md) — easy-start guide for a forking dev that uses OpenBao. Every PR is measured against the **forker's manual-command count**. If your change adds a manual step to `fork-quickstart.md`, that's debt — try a workflow first.
## Load-bearing invariants — gate every secrets decision
Load-bearing subset — canonical numbering is [`docs/spec/secrets-management.md`](../../../docs/spec/secrets-management.md) Invariants 1–16; the rows below are the ones that gate day-to-day secrets work (7, 10–12, 14 omitted — read the spec for the full set):
| # | Rule | Where it bites |
| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- |
| 1 | PATH = `cogni/<env>/<service>/<KEY>`; `<service>` = catalog name | New service → new ExternalSecret dir |
| 2 | ONE ExternalSecret per (service, env) with `dataFrom: extract`; target `<service>-env-secrets` | Adding keys = NO YAML edit |
| 3 | Pod `envFrom: secretRef: name: <service>-env-secrets` once per container | Pod spec set ONCE at service creation |
| 4 | NO secret value in git — ever | Base64-in-YAML = immediate rotate + audit |
| 5 | OpenBao is runtime SSOT; VM `.env` files are rendered views, not authorities | Don't seed runtime values in two places |
| 6 | RBAC via path policy (`eso-reader`, `<env>-writer`) bound to k8s SAs | Phase 5b.3 + 5b.4 of `provision-env-vm.sh` |
| 8 | Every access audited via OpenBao audit device → Loki | Pipeline not built yet — bug.0445 follow-up |
| 9 | Three entry points only: CLI / workflow_dispatch / operator API (`POST /api/v1/nodes/<id>/secrets`, OpenFGA `can_manage_secrets` — shipped #1627). Never raw `bao kv put` | See decision tree below |
| 13 | NO_OPERATOR_ROOT_TOKEN_ON_LAPTOP — bootstrap window only; day-2 uses writer-role JWT | `.local/<env>-openbao-root-token` is never read post-Phase-5b |
| 15 | Pod-facing DB role material is OpenBao-owned, even when Compose renders a copy | No DB password authority in GitHub env or VM `.env` |
| 16 | New-node secret materialization precedes substrate reconcile/assert | Generate safe agent values before first flight |
## Authority model — classify by three axes
Do not use `tier` as the authority model. Every secret has:
| Axis | Values | Question |
| ----------- | ---------------------------------------- | ------------------------------ |
| `origin` | `agent` · `human` · `derived` | Who can produce the bytes? |
| `custody` | `openbao` · `github-env` · `repo-config` | Which system is authoritative? |
| `consumers` | `pod` · `compose` · `ci` · `external` | Where does the value get used? |
Hard rule: if a value is consumed by a pod, provisions a pod-facing role, or
must agree with a pod-facing value, custody is OpenBao. GitHub Environment
Secrets may bootstrap access and carry CI-only credentials. VM `.env` files are
rendered views for Compose, never authorities.
Examples:
- `POSTGRES_ROOT_PASSWORD`: Compose/bootstrap-only for now, because pods should
not consume it.
- `APP_DB_PASSWORD`, `APP_DB_SERVICE_PASSWORD`,
`APP_DB_READONLY_PASSWORD`, `DOLTGRES_PASSWORD`,
`DOLTGRES_READER_PASSWORD`, `DOLTGRES_WRITER_PASSWORD`: OpenBao custody;
Compose renders copies to create roles. (`DOLTGRES_PASSWORD` is the env Doltgres
superuser the pod itself authenticates as — Doltgres RBAC is table-DML-only, so no
per-node role; why + value-shape in [databases.md §5.2](../../../docs/spec/databases.md).)
- `OPENFGA_DB_PASSWORD`: OpenBao custody. **OpenFGA is the first shared-infra DB given a dedicated, non-root login role** (its own `openfga` role, OpenBao-sourced — Phase A dropped root from the datastore DSN). That's the **hardening direction, not a snowflake**: `litellm` / `postgres` are still the un-hardened root-owned legacy. Consequence — **shared-infra DBs (openfga, litellm) are provisioned in the `INFRA_ONLY` pass only**; the per-node provision pass must not touch them (it holds no shared-infra password). `litellm` survives the per-node pass only by being root-owned (no password to demand); openfga's dedicated role _exposed_ that the per-node pass should never provision shared infra. `provision.sh` now gates openfga to `INFRA_ONLY`; gating `litellm` the same way is the fuller fix. **Shared-infra DB-cred rule (learned the hard way, fleet reprovision 2026-08-04 — applies to any NEW one):** (1) a shared-infra DB-role password (`OPENFGA_DB_PASSWORD`, `TEMPORAL_DB_PASSWORD`) is NOT a catalog entry (infra ≠ node; the loader allowlist is node-domain) — `provision-env-vm.sh` Phase 5c **generates it (`randHex 32`) + seeds it set-once** (reuse the existing OpenBao value on re-provision, never regenerate → the created role can't diverge, the 2026-06-11 temporal-drift class). (2) `deploy-infra` MUST read it via the **ungated `openbao_get_field` (`${env}-db-reader` seam)** — NOT `source_openbao_runtime_key`, which is gated on `operator-env-secrets` (`OPENBAO_RUNTIME_SSOT=true` = "established ESO env"). That gate is **false on a fresh provision** (no namespace/secret yet), so a `required` read there silently skips and the `.env` render trips `set -u` → `<VAR>: unbound variable` → deploy-infra aborts before the app layer, leaving a 0-byte runtime `.env` (a red-herring symptom). Wiring a _consumer_ (compose DSN + `provision.sh` role) without also wiring the _producer_ (Phase 5c seed) + the _ungated read_ leaves the env unprovisionable-from-clean but green on established re-deploys — latent until the next from-scratch provision. (3) **The superuser password MUST be minted set-once (`randHex`), NEVER _derived_ from another secret.** `DOLTGRES_PASSWORD` was the one left out of the 2026-08-04 set-once fix: `deploy-infra` fell back to `derive_secret doltgres-root = sha256(POSTGRES_ROOT_PASSWORD)`, so a root rotation (or genesis-derive ≠ a later value) **drifted the rendered password from the volume's superuser, which is baked at first-init and IMMUTABLE** (Doltgres can't `ALTER` its superuser; postgres/temporal can, via the local-trust socket — that's why _they_ have an in-place Step-6.5 reconcile and doltgres cannot). Fix = seed set-once at `cogni/<env>/operator/DOLTGRES_PASSWORD`, mirroring OPENFGA/TEMPORAL → the superuser can never drift. (4) **NEVER "heal" a credential drift by recreating/wiping the DB volume** — destroying data to fix a password is a footgun (a transient auth-probe miss then nukes a live store). Prevent drift with (3); an already-drifted volume is recovered by an **explicit operator action** (wipe only if provably empty; otherwise re-pin the SSOT to the volume's value), never an automatic deploy-path behavior.
- Public URLs / owner slugs / feature modes: repo-config, not OpenBao.
## Biggest systemic shortcoming — the `.env`/`_shared` split-brain
One logical `shared` secret becomes **three physical copies** — `_shared` SSOT, per-node inherited copy (ESO→pod), VM `.env` render (Compose) — written by three writers with **no lockstep**. They drift: the recurring root of secrets outages (litellm / temporal / doltgres / billing `401`·`28P01`). The tell: **pods have no `.env` and rarely drift; every `.env`-bound Compose service is the surface.** Never heal toward `.env` (Invariant 15); the endgame is to **eliminate the copies** (purge the `_shared` bucket + purge the server `.env` → Bao Agent / k8s). Mechanism, evidence, three-layer fix, falsification gate → [`docs/spec/secrets-split-brain.md`](../../../docs/spec/secrets-split-brain.md).
## Routing tree — where does the value render?
| Tier | Consumed by | Render path | Custody |
| ---- | ------------------------------------------------------------------------ | -------------------------------------------------------------- | -------------------------------- |
| A1 | k8s pod baseline (anything under `nodes/<n>/app/`, every fork) | OpenBao `cogni/<env>/<service>/*` → ESO → k8s Secret → envFrom | OpenBao |
| A2 | k8s pod node-specific (downstream node like `poly`) | OpenBao `cogni/<env>/<node>/*` → ESO → k8s Secret → envFrom | OpenBao |
| B | Compose-infra service (postgres, litellm, temporal, redis, alloy, caddy) | Rendered to VM `.env` or future Bao Agent | OpenBao unless CI/bootstrap-only |
| D | CI-only (workflow consumption, never runtime) | GH Env Secret → workflow `env:` block | GH Environment Secrets |
| E | Repo-level CI (cross-env, one value per repo) | GH Repo Secret | GH Repo Secrets |
| F | Local dev only | `.env.local` (gitignored) | Operator's laptop |
| G | Derived from repo state at provision time (e.g. `nodes/*` listing) | Computed by loader; written alongside other catalog values | Auto-generated |
Full tier definitions + invariants: [`docs/spec/secrets-classification.md`](../../../docs/spec/secrets-classification.md).
Layer-cake framing (Identity → AuthN → AuthZ → Secrets → DAO → Operator): [`docs/spec/access-control-charter.md`](../../../docs/spec/access-control-charter.md).
Routing checklist (file-by-file propagation): [`.claude/commands/env-update.md`](../../commands/env-update.md) §0.5.
## Runtime env triage
`serverEnv()` validates process env; it does not decide the source of truth.
Classify first:
- **Secret:** leaking it requires rotation or incident response. Route through
OpenBao/ESO or the proper GH secret tier.
- **Plain config:** owner slugs, repo names, public URLs, feature modes, and
routing values. Route through GitOps ConfigMaps or repo config.
For either path, k8s object presence is not process proof. Pods read env only at
startup, so prove: source object -> Deployment `envFrom` -> restarted pod env ->
public health. Treat old Argo/workflow failures as leads until live cluster
checks agree.
## App flight substrate assertions are read-only
`candidate-flight.yml` uses `scripts/ci/assert-target-substrate.sh` as a
preflight for selected app rollouts. That gate may verify a Deployment-consumed
k8s Secret exists and its matching ExternalSecret is Ready, but it must not seed
OpenBao, patch GitHub secrets, run `deploy-infra.sh`, or repair Compose/env
state. A missing secret is a substrate failure. For a wizard-created ordinary
node, there should be **zero per-node human secret values**. The environment
must already have the substrate/runtime inputs classified in
`docs/spec/secrets-classification.md#node-wizard-formation-contract`.
If an environment-bank value is missing, repair that bank; do not pass values
through candidate-flight inputs or store them in the wizard.
**"Read-only" applies to `assert`, not the whole flight.** Distinct from the
read-only assertion, `candidate-flight.yml` / `promote-and-deploy.yml` run a
**`materialize-substrate`** job _first_ — the **sole OpenBao writer in the flight**
(`scripts/ci/secret-materialize.sh`, `<env>-writer` token) — which generates the
node's `source: agent` secrets at `cogni/<env>/<node>/*` idempotently before
reconcile/assert, **including the per-node DB creds** (`app_<node>`/`service_<node>`
passwords) and the composed Postgres DSNs (`DATABASE_URL`/`DATABASE_SERVICE_URL`;
only `DOLTGRES_URL` is still deferred) since **#1584** (canonical:
[`ci-cd.md` Axiom 22](../../../docs/spec/ci-cd.md),
`SUBSTRATE_IS_RECONCILED_BEFORE_PROMOTION`). Since #1584 `reconcile-substrate` is
**read-only on OpenBao** (`<env>-db-reader` token, zero `bao kv put/patch`) and
`assert-target-substrate.sh` is read-only too — so only `materialize-substrate`
writes. **bug.5007 status (corrected 2026-06-24):** the `openbao-operator`
ServiceAccount that `materialize-substrate` uses to mint `<env>-writer` **does
exist on prod** (`default` ns, 21d), and `operator-secrets-writer@cogni-production`
exists too (6d) — the "SA absent on prod" framing is **stale**. The self-serve write
path is proven (operator returns `200`). The original bug.5007 promote-pipeline e2e
(`materialize-substrate` green on a real prod promote) is the part still worth a direct
confirm; the SA/role prerequisite it blocked on is in place.
The target shape matters. Today the implemented branch is `type=node`; a future
`type=service` branch should assert the service's declared Secret /
ExternalSecret / ConfigMap contract without inheriting node DNS, Caddy, NodePort,
or node-DB assumptions.
## Decision tree — how do I write / rotate the value?
| Operation | Right pattern | Today's reality |
| ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Add new secret SHAPE (service X consumes key A) | PR → `vault-config-operator` CRD → Argo reconciles | Not built; tracked in `proj.agentic-fork-bootstrap` Walk |
| Materialize a new wizard node's runtime secrets | Generate/derive node-local values and inherit only explicit org/env grants; no per-node human values for ordinary nodes | Shipped: `materialize-substrate` is its own flight job (`scripts/ci/secret-materialize.sh`, `<env>-writer` token), idempotent read-once→diff→write-missing (#1582/#1585; ci-cd.md Axiom 22). The shared-bank / explicit `inheritFrom`-grant model is the follow-up |
| Rotate AUTO-GENERATED value (e.g., `AUTH_SECRET`) | `rotate-secret.yml` workflow with env-protection; auto-generates value; **human approves event, never sees value** | Not built; do manual `openssl rand` + `pnpm secrets:set` per [`secrets-rotate.md`](../../../docs/guides/secrets-rotate.md) |
| Rotate VENDOR-MINTED value (OpenAI key, Cherry token) | Operator API route `POST /api/v1/nodes/[id]/secrets`, OpenFGA `can_manage_secrets` — caller holds only an API key | **Shipped + live on candidate-a AND production** (operator returns `200`). `preview` still `503 authz_unavailable` (OpenFGA store not resolvable by the pod). For preview/cross-env: CLI + admin/kube custody. See "Who holds secret-write authority" above |
| Candidate-a experimentation | `pnpm secrets:set <env> <service> <KEY>` via port-forward + writer-role JWT | Shipped — see [`secrets-add-new.md`](../../../docs/guides/secrets-add-new.md) |
| Dynamic DB credentials | OpenBao DB engine, no human in loop | Future (Crawl row 3 of `proj.security-hardening`) |
The killer rule: **no human types a secret VALUE into a UI in production.** Auto-generated, vendor-minted via operator-app, or dynamic. Form-input is the anti-pattern.
## Who holds secret-write authority — TODAY vs TARGET (READ THIS)
The honest answer to "can a node add its own secret without Derek's GitHub/kube creds?" is **somewhere in between, closer to Derek/GitHub-admin-controlled today.** Do not claim "operator-app RBAC" where it isn't yet shipped. Three distinct authorities — keep them separate:
| Authority | What it decides | TODAY (custody) | TARGET |
| ------------------------------------------------------------------ | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- |
| **Secret declaration** (shape) | "node Y is allowed key KEY_X" | GitHub repo merge authority — whoever can merge a PR to the catalog. Guarded by required CI + Cogni Git PR Review + branch protections. **No value in the PR.** | Same (PR-gated). This rung is fine. |
| **Generated value** (`source: agent`) | random bytes minted for node Y | GitHub Actions `materialize-substrate` job: the flight workflow SSHes the env VM and mints an `<env>-writer` OpenBao token via the in-cluster `openbao-operator` SA. **Real and shipped** for candidate-a/preview; the `openbao-operator` SA **also exists on prod** (21d) — the old "prod lacks the writer SA (bug.5007)" line is stale. "Who can cause it" = whoever can merge/dispatch the env's deploy workflow → **GitHub merge authority again.** | In-cluster Job (SA-bound `<env>-writer`), zero SSH (design §North-star Stage 2). |
| **Human/vendor value** (`source: human`: OpenAI key, Cherry token) | the byte value a vendor minted | **Operator-controlled CLI/kube path** (`pnpm secrets:set` + short-lived writer JWT via port-forward). In this workspace that effectively means **Derek/admin-held repo + kube custody.** | Operator app route + OpenFGA RBAC (see below) — **prototyped, not merged.** |
**The gap that keeps us on Derek's creds:** there is **no shipped operator-app secrets route** and **no `.github/workflows/secret-set.yml`**. Every value-write today bottoms out in GitHub-merge authority or operator/admin kube custody — i.e. Derek.
**The precise PR/design that closes it** — the value-write complement to the flight triangle (`developer → can_flight → operator-held GitHub App creds`), one rung over with its **own least-privilege role** (`secrets_manager → can_manage_secrets → operator-held OpenBao writer`; distinct from `developer`, mirroring `production_promoter`):
- Design: [`docs/design/node-self-serve-secrets.md`](../../../docs/design/node-self-serve-secrets.md) (operator-mediated, OpenFGA-authorized). Sibling: [`node-wizard-secret-setting.md`](../../../docs/design/node-wizard-secret-setting.md) (shape, shipped).
- PR: **#1627** (prototype) — `POST /api/v1/nodes/[id]/secrets` + `OperatorSecretsPlanePort` + `OpenBaoSecretsAdapter` (operator pod self-logins with its OWN projected SA token, `audience: cogni-openbao`) + `can_manage_secrets` OpenFGA relation + build-time A2 allowlist. Caller holds **only an API key — no kubeconfig, no vault token.**
- **Per-env readiness is live state — recall the hub guide `node-self-serve-secrets`, don't trust a git snapshot.** The provisioning prereqs + the `503 authz_unavailable` OpenFGA-model-drift class are above; the durable cross-env fix is `task.5049`.
Three defense-in-depth gates in #1627: OpenFGA `can_manage_secrets` (fail-closed) → catalog A2 allowlist (build-time typed module) → OpenBao policy explicit-`deny` on `_system`/`_shared`.
**Validation status + the road to a proper e2e:** [`docs/design/node-self-serve-secrets-validation-roadmap.md`](../../../docs/design/node-self-serve-secrets-validation-roadmap.md) — the layered proof (L1 candidate-a authz **done**; L2 full write needs the allowlist codegen + the OpenBao writer-role DRY fix; L3 human-in-loop deployed e2e). **Load-bearing gotcha:** the write env is operator-stamped (`deployEnv = env.DEPLOY_ENVIRONMENT`), so `cogni/<operator-env>/<node>/*` is the only path it can write — a prod operator **cannot** write a candidate-a node's secret. Any cross-env "set my candidate-a secret from the prod operator" needs an explicit authz-scoped `targetEnv` param (not in #1627).
## Placement is not provisioning — the empty-node-path class (READ THIS)
**Symptom:** a node deploys fine in one environment and its ExternalSecret fails in another with
`could not get secret data from provider` / `cannot find secret data for key: "AUTH_SECRET"` — failing at
`spec.data[0]`, i.e. the node's OpenBao path is **empty**, not one key short.
**Cause:** the node was moved to external compute (`deployment_provider.<env>: akash`) and thereby dropped
out of the job that provisions it. `node-substrate` — which runs `secret-materialize` — is matrixed on
`k3s_node_targets_json` in both `candidate-flight.yml` and `promote-and-deploy.yml`. Flip a node's provider
and it silently leaves the population that gets served. Nothing errors; nobody creates its secrets.
**Why it hides:** the first external nodes were k3s nodes _first_. They kept the secrets they already had,
so test and preview looked healthy while inheriting from a k3s past. The defect only surfaces in an
environment where that node never ran on k3s. Do not treat a green lower environment as evidence.
**Two rules:**
1. **Branch the steps, never the population.** Every `type: node` target gets materialized regardless of
provider; only provider-specific work (k3s Caddy/NodePort edge reconcile vs external DNS reconcile) is
selected inside the job. The typed planner already emits the correct wider node list — prefer it over the
k3s-only one.
2. **A provider that needs a different secret needs a catalog entry, not a rename.** External compute must
not receive fleet-shared credentials (they would reach a rented third-party host), so it consumes a
node-scoped equivalent — e.g. `LITELLM_VIRTUAL_KEY` in place of the shared `LITELLM_MASTER_KEY`. That is
correct, but it is only real once the catalog declares it with a generator. A key that has a consumer and
no declaration gets hand-seeded once and silently carried forever (`CATALOG_IS_THE_ONE_READER`).
**Fast triage:** read key NAMES only, never values —
`bao kv get -format=json cogni/<env>/<node>` and compare the environment that works against the one that
does not. If the failing path is empty rather than short, this is your class.
Spec: `docs/spec/secrets-management.md` invariants 17-18; `docs/spec/node-baas-architecture.md`
§ Placement vs Provisioning.
## Dual-plane secrets — the silent-webhook-fail class (READ THIS)
Some secrets must **byte-equal a value held by an external system**, not merely exist. The operator's `GH_WEBHOOK_SECRET` must equal the **GitHub App's webhook secret**; an OAuth `*_CLIENT_SECRET` must equal the provider's app config. These are **dual-plane**: one copy in our pod, one on the external plane — they only work if identical.
**The trap (live bug, preview 2026-06-03):** `GH_WEBHOOK_SECRET` was `source: agent` with **no `syncTo`**, generated `randHex 32` **every provision** → it could never match the App's webhook secret → **every webhook failed HMAC verification**, silently (a `level:40` warn `component:webhook-route msg:"webhook verification failed"`, no alert, no 5xx). The App just looks dead — no PR reviews, no node-wizard. `deploy-infra` re-applying the Secret was the **breaking** path, not the healing one — a generated value never equals an externally-held one **unless something pushes it there**. That push is what `syncTo` declares.
**`syncTo` is an external mirror, not custody.** `origin` says who can produce
the bytes, `custody` says where the value is authoritative, and `syncTo` says
whether a copy must also be pushed to an external system.
| axis | values | answers |
| ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| `source:` | `agent` (we generate, `generate:` required) · `human` (un-generatable; vendor-minted, human supplies once, e.g. `GH_REVIEW_APP_PRIVATE_KEY_BASE64`) | where the value **originates** |
| `syncTo:` _(opt)_ | `github-app-webhook` · _(unset)_ | does it **also** live in an external system we must keep in lockstep |
`GH_WEBHOOK_SECRET` is `source: agent` + `syncTo: github-app-webhook` — **we generate it** (origin is internal; calling it `source: external` lied about that), and it ALSO must byte-match the App. The materializer/provisioner generates it into OpenBao; the infra lane then pushes it (`scripts/secrets/sync-app-webhook-secret.sh`: App JWT → `PATCH /app/hook/config`):
```
APPID + GH_REVIEW_APP_PRIVATE_KEY_BASE64 → RS256 JWT (iss=APPID, exp≤10m)
PATCH /app/hook/config -d '{"secret":"<generated GH_WEBHOOK_SECRET>"}' # endpoint EXISTS
```
No-human-secret done right: agent generates, agent pushes, **zero human, self-healing** (provisioning owns both copies → every infra-lever deploy re-converges). Do NOT make it `source: human`/carry — that drags a human into the App's "Change secret" field for a value that's ours to generate.
⚠️ **Sync only fires on the infra lever** (`deploy-infra` via `candidate-flight-infra` / `provision-env`), NOT on app-lever promotes (`candidate-flight`/`flight-preview`/`promote-and-deploy` = Argo image bump, never touches the Secret). `assert-target-substrate.sh` is also read-only: it can fail a flight when the Deployment-consumed Secret / ExternalSecret is absent or not Ready, but it does not heal the value. In an established ESO environment, `deploy-infra` must source `GH_WEBHOOK_SECRET` from OpenBao before invoking the sync and must fail closed if the App PATCH fails; otherwise the pod plane and GitHub App plane can silently diverge again. Fresh/plain-Secret bootstrap may still use workflow env input until the operator ESO target exists.
**Heal-proof test** = redeploy twice; a PR on the test repo must still post a `cogni-git-review` review.
## OpenBao availability — sealed = DOWN, not secure (READ THIS)
Counterintuitive but load-bearing: a **sealed** OpenBao serves **nothing** — it 503s every ESO sync, every `materialize`/`reconcile-substrate` `auth/kubernetes/login`, and openfga's config load. OpenBao must run **UNSEALED ~100% of the time**; unsealed-and-running is the _correct_ operational state. The seal protects **data at rest** (stolen storage = encrypted/useless), **not** runtime access — that's auth methods + path policy (Invariant 6) + in-cluster-only (no Ingress). So "keep it sealed / minimize unsealed time" is backwards: an _unexpected_ seal is an **outage**, not a safe state.
**The fragility (`bug.5011`, prod outage 2026-06-11 02:25):** prod OpenBao is **Shamir 1-of-1, no auto-unseal**. It OOMKilled on the 6 GB box → resealed → secret plane down (openfga 503, `node-substrate` promote failures) until a human `kubectl exec … bao operator unseal`. **Any** restart (OOM, reboot, chart upgrade) reseals it.
**Recorded decision (`bug.5011`) — do not re-litigate without the trigger:**
1. **Memory headroom** — OpenBao limit 512Mi→1Gi (#1617) so it isn't container-OOMKilled; #1616 lean-prod cut node pressure too. Mitigates the OOM _trigger_ only.
2. **Sealed-state → Loki alert** — the no-silent-outage control: detect an unexpected seal, page, unseal in minutes. Free, no vendor. This is the durable fix, because you cannot prevent every reseal.
3. **Auto-unseal (KMS) DEFERRED** behind a written trigger: _reseals recur post-#1617, OR a SOC2 engagement starts_. KMS is the audit-correct control but it is our first AWS/GCP dependency on a Cherry+OSS stack — premature at 0 users. If a real SOC2 deadline lands, flip it to now.
4. **REJECTED: in-cluster stored-key unseal sidecar** — key-next-to-lock, fails SOC2.
## Anti-patterns — instant reject
- Human typing a secret VALUE into a production UI or GitHub workflow input. See killer rule.
- Treating `tier: B`, GitHub Environment Secrets, or VM `.env` as authority for
a runtime secret. If it feeds a pod or a pod-facing role, OpenBao owns it.
- Candidate-flight accepting secret values as inputs. Flight may invoke
materialization for safe generated/derived node values and fail if the
environment's required DAO/org bank is missing; it must not carry values.
- A **dual-plane** secret (must byte-match an external system — GitHub App webhook secret, OAuth client secret) declared with **no `syncTo:`** — it silently fails verification forever and `deploy-infra` re-breaks it every run. Add `syncTo:` (keep `source: agent` if we generate the value). See "Dual-plane secrets" above.
- Generic catch-all workflow (`secrets-manage.yml`-shaped). Per-operation only.
- `ssh root@vm kubectl ...` or `ssh root@vm bao ...`. Use local kubectl + port-forward + writer-role JWT.
- Treating k8s Secret or ConfigMap presence as proof that a running pod has the value. Prove the process after rollout.
- Treating a failed app-flight substrate assertion as permission to run
`deploy-infra` from the app flight. Heal secrets/substrate through the owning
secrets or infra lane; keep app flight read-only until image promotion.
- Re-exporting `.local/<env>-openbao-root-token` after Phase 5b — violates Invariant 13.
- `bao kv put` instead of `bao kv patch` (replaces sibling keys).
- `bao login -method=kubernetes` in OpenBao CLI 2.5.x — that subcommand doesn't exist; use raw API: `bao write auth/kubernetes/login role=X jwt=Y`.
- Per-secret ExternalSecret YAML — violates Invariant 2.
- `valueFrom: secretKeyRef` per env var in pod spec — violates Invariant 3.
- Base64-in-git "encryption" — violates Invariant 4.
- Sealed Secrets / SOPS+ksops — explicitly rejected per `proj.security-hardening` Design Notes.
- Editing `scripts/setup-secrets.ts` to add/remove a SECRETS array entry — there isn't one. The script loads YAML via `scripts/lib/secrets-catalog-loader.ts`. Edit the appropriate `secrets-catalog.yaml` instead.
- ExternalSecret manifests under `infra/k8s/secrets/external-secrets/<env>/<node>/` — that pre-wizard tree is PURGED. Remote-source/wizard nodes carry their leaf in the node repo at `k8s/external-secrets/<env>/`, mounted by the operator as `nodes/<node>/k8s/external-secrets/<env>/` when pinned. Only the cluster-scoped `ClusterSecretStore` remains under the old dir.
- Treat the operator overlay-local ExternalSecret as a legacy operator exception, not the node standard. `infra/k8s/overlays/<env>/operator/external-secret.yaml` is Argo-owned because the operator Deployment patch lives in that overlay; source name `env-secrets` renders through `namePrefix: operator-` to object `operator-env-secrets`, while `spec.target.name` stays `operator-env-secrets`.
## The catalog (per-node YAML + Zod loader)
`scripts/setup-secrets.ts` does NOT hold a hardcoded SECRETS array. It calls a Zod-validated loader that walks YAML catalogs:
| File | Domain | Holds |
| ------------------------------------------ | --------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `infra/secrets-catalog.yaml` | **operator-domain** | `_shared`, `_system`, B/D/E/G entries, A2 placeholders, **and (today) all node A1/A2 entries** — node-template's catalog was migrated here by task.5094 |
| `nodes/<node>/.cogni/secrets-catalog.yaml` | **node-domain** (single-node-scope: a node engineer adds their node's secret in ONE PR) | A1/A2 entries the node owns; `service:` auto-fills from parent dir. **Loader-supported but currently empty** — no node populates it yet (post-task.5094 consolidation) |
| `scripts/lib/secrets-catalog-loader.ts` | **operator-domain (substrate)** | Zod schema + walker (walks both paths above) + uniqueness + service-allowlist assertions |
**To add a node secret today:** edit `infra/secrets-catalog.yaml` — node entries are consolidated there post-task.5094 (per-node `.cogni/secrets-catalog.yaml` is loader-supported and is the sovereignty target, but no node uses it yet).
**To add an operator-domain secret (B/D/E/G, or `_shared` cross-cutting):** edit `infra/secrets-catalog.yaml`. Operator-domain PR.
**The loader rejects at module load:** missing `tier`, name collision across files, per-node `service:` mismatch with parent dir, unknown `service:` value not in the allowlist (`_shared`/`_system` + present nodes + canonical-future-domain names from `node-ci-cd-contract.md`).
## Files to read by topic
| If you're doing… | Read |
| ------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Understanding the tier system + invariants | [`docs/spec/secrets-classification.md`](../../../docs/spec/secrets-classification.md) |
| Designing node-wizard secret behavior | [`docs/design/node-wizard-secret-setting.md`](../../../docs/design/node-wizard-secret-setting.md) |
| The layered authority model (Identity → DAO) | [`docs/spec/access-control-charter.md`](../../../docs/spec/access-control-charter.md) |
| Adding a new secret to your node | Edit `nodes/<node>/.cogni/secrets-catalog.yaml` + read [`docs/guides/secrets-add-new.md`](../../../docs/guides/secrets-add-new.md) |
| Adding a `_shared` / B / D / E / G secret | Edit `infra/secrets-catalog.yaml` |
| Rotating an existing secret | [`docs/guides/secrets-rotate.md`](../../../docs/guides/secrets-rotate.md) |
| Following the bootstrap flow | [`docs/runbooks/fork-quickstart.md`](../../../docs/runbooks/fork-quickstart.md) |
| **Provisioning a new env** (candidate-\*, fork) | Dispatch [`.github/workflows/provision-env.yml`](../../../.github/workflows/provision-env.yml) (`-f env=<env> -f encryption_passphrase=…`). Full walkthrough — 7 minting secrets + init-artifact custody — in [`docs/runbooks/fork-quickstart.md`](../../../docs/runbooks/fork-quickstart.md) §6. The runner owns the tofu+bao+kubectl session; the operator's laptop never holds substrate creds. |
| Adding a new service (new k8s Deployment) | [`docs/guides/node-formation-guide.md`](../../../docs/guides/node-formation-guide.md) + add ExternalSecret under `nodes/<node>/k8s/external-secrets/<env>/` |
| Touching substrate provisioning | [`scripts/setup/provision-env-vm.sh`](../../../scripts/setup/provision-env-vm.sh) Phases 5b.1–5b.5 |
| Touching the CLI | [`scripts/secrets/set-secret.sh`](../../../scripts/secrets/set-secret.sh) + test [`scripts/ci/tests/set-secret.test.sh`](../../../scripts/ci/tests/set-secret.test.sh) |
| Touching the loader / catalog schema | [`scripts/lib/secrets-catalog-loader.ts`](../../../scripts/lib/secrets-catalog-loader.ts) (Zod schema + walker) |
| Touching any node's ExternalSecret | Remote-source/wizard nodes: `nodes/<node>/k8s/external-secrets/<env>/` (per-node, node-domain) — the single repo-wide convention. Operator legacy overlay: keep `infra/k8s/overlays/<env>/operator/external-secret.yaml` aligned until operator moves to the same node-domain leaf. No aggregator; only the cluster-scoped `ClusterSecretStore` lives under `infra/k8s/secrets/external-secrets/`. |
| Touching substrate Argo Applications | `infra/k8s/argocd/{openbao,external-secrets}-application.yaml` |
| Touching the env-var classification routing | [`.claude/commands/env-update.md`](../../commands/env-update.md) — k8s app vs Compose-infra split |
| Granting Grafana/Loki **query** access (dev, validator, or a gate) | [`docs/spec/grafana-observability-access.md`](../../../docs/spec/grafana-observability-access.md) — one admin root → scoped consumers. **Dev node-read = operator query-PROXY pinned to `{node="<id>"}`; the dev holds NO token** (a returned token's reach escapes the per-node OpenFGA check → dormant env-wide leak; never hand a dev a Grafana token). Real blocker: node Loki streams have no `node` label yet (`app/env/service` only). assertLive gates on the PUBLIC `run-carries` rung; the operator's gate holds no query token. Out of MVP scope: label-scoped `glc_` / per-dev SA minting. Cross-substrate plane + health scorecard: [`docs/spec/substrate-access-grant.md`](../../../docs/spec/substrate-access-grant.md). |
| Designing a new workflow that handles secrets | This file + `proj.agentic-fork-bootstrap` anti-patterns. Run it past the killer rule. |
## When to escalate
Surface to operator before writing code if:
- Adding a NEW entry point that isn't already CLI / workflow_dispatch / operator-MCP — Invariant 9 lists the only three sanctioned shapes.
- Changing `eso-reader` policy or `<env>-writer` role binding — affects every consumer.
- Bumping OpenBao or ESO chart version — rotation drill required (see `secrets-rotate.md` §Upgrade discipline).
- Anything that smells like Invariant 4 (NO_VALUE_IN_GIT) — finding a value in YAML / commit message / PR diff / chat is always a rotate-now event.
- Designing a workflow where humans type values into a form — recheck against the killer rule before building.
Is this your skill, or is something wrong with this listing? Request removal or report an issue. Author removals are honored within 72 hours.
No comments yet. Be the first to comment!
