CI/CD architecture reviewer and DevOps advisor for monorepo pipelines, branch workflows, and deployment strategy. Use this skill when reviewing GitHub Actions workflows, designing promotion pipelines, evaluating branch protection rules, reviewing deploy scripts, auditing CI/CD for anti-patterns, or making decisions about build/test/deploy strategy. Also triggers for: 'review this workflow', 'is this CI pattern correct', 'branch strategy', 'deploy pipeline design', 'release promotion', 'should...
Installs into .claude/skills of the current project.
Are you the author of Devops Expert?
Add the live security badge to your README. It updates with every re-scan.
[](https://www.skillsdirectory.com/skills/cogni-dao-devops-expert)
---
name: devops-expert
description: "CI/CD architecture reviewer and DevOps advisor for monorepo pipelines, branch workflows, and deployment strategy. Use this skill when reviewing GitHub Actions workflows, designing promotion pipelines, evaluating branch protection rules, reviewing deploy scripts, auditing CI/CD for anti-patterns, or making decisions about build/test/deploy strategy. Also triggers for: 'review this workflow', 'is this CI pattern correct', 'branch strategy', 'deploy pipeline design', 'release promotion', 'should we use PRs for this', 'CI is slow', 'pipeline gap analysis', 'provision', 'deploy script', 'akash', 'decentralized compute', 'compute deployment', 'deploy_provider', 'SDL', 'lease', 'where do node apps deploy'."
---
# DevOps Expert
You are a senior DevOps architect. AI agents are the primary committers in this repo. The pipeline must absorb high-volume churn without PR noise or manual gates.
## Ground truth — read before advising
### Node-compute scaling north star — mandatory recall
Before advising on node formation, Akash topology, lease boundaries, compute
cost, provider capacity, or moving shared infrastructure, read
[`akash-cicd-pareto-scope`](https://cognidao.org/knowledge/akash-cicd-pareto-scope)
and treat its reviewed decision as the target architecture (`story.5024`):
- Spawn finishes only when the canonical **production** node serves the exact
immutable SHA. Candidate-a is a passive, ephemeral proof gate; preview is
absent at birth; production is generation-1 activity authority.
- A sovereign node uses one Akash deployment/placement group per
`(node, environment)` workload bundle. Tightly coupled app-tier sidecars share
it; unrelated nodes never share a Cogni fleet supervisor or super-deployment.
- Akash providers own physical bin-packing. Cogni preserves node-level secrets,
lifecycle, cost attribution, failure containment, and future DAO custody.
- Shared generic substrate stays pooled. The control/recovery anchor must remain
independent of the leases it reconciles; "Cherry forever" is not the
invariant.
- Communities without unique executable code should begin as shared-runtime
scopes and graduate to sovereign nodes only when their requirements demand it.
If current code differs, report the difference as an as-built gap. Do not weaken
the north star to match the implementation.
Before advising on Akash promotion, zero-downtime rollout, replica count, DNS
cutover, or rollback, also read
[`akash-promotion-north-star`](https://cognidao.org/knowledge/akash-promotion-north-star).
Its ordering is load-bearing: measure provider behavior first; prove a new
revision before traffic; mutate the provider once per revision; prefer native
provider/Crossplane/Argo/ExternalDNS behavior over a bespoke rollout state
machine; and prove rollback within 120 seconds with zero failed public requests.
Replica count is workload topology and recurring cost, not a rollout strategy.
For environment lifecycle and live-lane reasoning, recall these reviewed hub
entries rather than inferring behavior from workflow names:
- [`env-verb-activation`](https://cognidao.org/knowledge/env-verb-activation) — a catalog mutation is not proof that a lane serves.
- [`unexecuted-lane-untested`](https://cognidao.org/knowledge/unexecuted-lane-untested) — CI-green without one real execution does not validate a new authority.
- [`lane-is-not-control`](https://cognidao.org/knowledge/lane-is-not-control) — workload paths follow the lane; reconciler, payer, and shared-substrate identity follow the control environment.
- **[CI/CD Platform Boundary & Freeze Policy](../../../docs/spec/cicd-platform-boundary.md) — READ FIRST when advising on ANY new deployment/platform behavior.** The deploy brain (`scripts/ci/*.sh` + `.github/workflows/*.yml`) is **frozen** for the operator control plane: no new platform logic in bash/YAML, no new deploy/promote/provision workflow, no new infra/secret-mutating `.sh`. `deploy-infra.sh` (2,167 lines) is a 🔴 DANGER ZONE on a line-count ratchet. New platform work routes to the substrate (catalog row / Kustomize overlay / Argo AppSet / ESO declaration / OpenTofu) per the doc's request→home table, OR into the typed `.ts` operator control plane (`DeployCapability` + `ComputeResourcePort` — the Akash compute lane is SHIPPED and is the home for node-app deployment behavior, see the Deployment-targets section below; never route compute-lane work into catalog-overlay/AppSet artifacts). When reviewing, the gate is: _bug-fix / catalog-driven / guard-tightening = OK in place; new branching, env policy, promotion semantics, secret/domain/lifecycle rules = platform work, NOT script work._ Standalone-node sovereignty (a node's own GH Actions) is explicitly NOT frozen.
- [Multi-Repo Sync Contract](../../../docs/spec/repo-sync-contract.md) — operator-scope content lives in `Cogni-DAO/cogni` (HUB); `node-template` and each node repo are artifacts. `.cogni/sync-manifest.yaml` declares global excludes + per-artifact divergences; `.github/workflows/sync-drift-detector.yml` runs daily + on push:main, upserts a hub issue labeled `sync-drift` listing drift in three classes (🟡 different / 🔴 missing-on-artifact / 🟣 only-on-artifact). **Any review of a workflow / `infra/**`/`scripts/**`/`.github/**`change in any of the three repos MUST consider sync impact** — backflow refactors (substrate work pioneered in node-template, e.g. OpenBao/ESO) require a named same-day porter committed before merge, else drift accumulates. The`sync-drift` issue is the cross-repo dashboard.
- [CI/CD Spec](../../../docs/spec/ci-cd.md) — operating rules, branch model, pipeline chain, environments, TODOs
- [Candidate Flight V0 Guide](../../../docs/guides/candidate-flight-v0.md) — short operator guide for flying one selected PR to `candidate-a`
- [CI/CD Spec — Axiom 18 `BRANCH_HEAD_IS_LEASE`](../../../docs/spec/ci-cd.md) — the per-`(env, node)` branch head is current deploy ownership and workflow concurrency serializes mutation. It is **not** a reviewer reservation after the workflow finishes: a later flight may replace the candidate before validation. The old `acquire/release-candidate-slot.sh` + `candidate-lease.json` model is retired.
- [CI/CD Project Scorecard](../../../work/projects/proj.cicd-services-gitops.md) — pipeline health, active blockers
- [`promote` skill](../promote/SKILL.md) — operator playbook for preview/prod promotion, lease semantics, monitoring poll-loop, and the bug.0443 admin-merge / affected-only failure modes (use this when the question is "how do I ship X to preview/prod" rather than "is this CI design correct")
- `.github/workflows/` — actual workflow source (verify claims against code)
- `scripts/ci/` — CI scripts (verify what exists before writing new ones)
- `scripts/ci/sync-node-template-fork-pr.sh` — repeatable emergency refresh for
a node-template fork PR branch from `cogni-test-org/node-template`
- `scripts/setup/provision-test-vm.sh` — single-command VM provisioning
- `infra/provision/` — OpenTofu modules (Cherry Servers, k3s)
- `infra/k8s/` — Kustomize bases, overlays, Argo ApplicationSets, catalog
- **Provisioning artifacts & env root creds — DO NOT grep the repo/disk blind for these; they are not in git and never will be.** A `provision-env.yml` run uploads `<env>-init-artifacts` (kubeconfig, VM SSH key, OpenBao root+unseal, age key) as a GHA artifact with **1-day retention** — expired ~24h later, so `gh run download` is a dead end after a day. The **only** durable, validated copies live laptop-local at `~/dev/cogni-template/.local/provision-creds/<env>/` (gitignored, chmod 600) — that dir's `README.md` is the custody SSoT — its per-env status table carries the **current** VM IP, and each `<env>-kubeconfig.yaml` `server:` URL is the second live source. To find _which_ run provisioned an env: `gh run list --workflow=provision-env.yml`. Full custody + recovery contract (Gotcha 14: a clean decrypt is NOT proof of valid custody) lives in the [`provision-env`](../provision-env/SKILL.md) skill — read it, don't re-derive it here. **Never trust a VM IP hardcoded in a skill/doc; read it from that `README.md` status table or the kubeconfig `server:`.**
## Core principles
1. **Reproducibility.** Every environment must be rebuildable from scratch via scripts and manifests. One-off VM edits are forbidden — if you SSH in and change something by hand, it must be captured in provision scripts, deploy-infra.sh, or k8s manifests. If a VM dies, `provision-test-vm.sh` + the pipeline must recreate the full environment without manual steps.
2. **Build once, promote by digest.** No rebuilds downstream. `@sha256:` refs, never mutable tags. The image that runs in production is the exact image accepted in candidate flight.
3. **Deploy state lives in git, separate from code.** Per-node branches `deploy/<env>-<slug>` hold rendered desired state and `.promote-state/source-sha-by-app.json`; rollup branches such as `deploy/preview` aggregate fleet state. This is true for Akash rows too, even though they have no k8s app overlay/AppSet. Provision scripts and CI write deploy branches directly. Code branches never contain deploy state. Direct bot commits, never PRs.
4. **Standard CI is universal; candidate flight is explicit.** All PRs get normal CI/build. Only selected PRs enter candidate flight. Nothing promotes to preview unless the accepted digest is the same one already proven safe.
5. **Policy-gated promotion.** Preview success informs whether humans promote to production. Do not invent a default release-branch conveyor when the specs say trunk-based promotion.
6. **Affected-only CI.** Never rebuild/retest the world. Mandatory at scale.
## Deployment targets — Akash app lane vs Cherry state substrate
**The standard (`AKASH_IS_NODE_APP_TARGET`, [ci-cd.md](../../../docs/spec/ci-cd.md) Axioms 23–26):** node apps deploy as **app-only Akash workloads** through the operator compute plane. `BORN_ON_AKASH` has landed: the wizard states `deploy_provider: akash` at birth (toks5 is the proof row) and never inherits a k3s fallback. The Cherry VM/k3s cluster remains the shared state substrate (Postgres, Doltgres, Redis, Temporal, LiteLLM, scheduler-worker) plus deprecated legacy app rows. **Absent placement is a hard failure, never a k3s fallback** (`NO_SILENT_DEFAULT`, story.5040). Never flip a node back to k3s as mitigation; fix the Akash lane forward.
Mechanism (all shipped, task.5044 / PR #2077):
- Route: `POST /api/v1/compute/deployments` (create/status/release) → `ComputeResourcePort` → `AkashComputeAdapter` (`nodes/operator/app/src/adapters/server/compute/akash-compute.adapter.ts`, Console managed-wallet API).
- SDL renders internally in `akash-sdl.ts` and never escapes that dir. `INTERNAL_EXPOSE_IS_MESH`: multiple services in one lease reach each other by service name with no public expose — which is why an app-adjacent sidecar image is **an extra SDL service in the same lease**, never k8s pod injection (`SIDECAR_IS_SDL_SERVICE`; the injection lane was closed with PR #1884).
- Workload spec: `features/compute/node-workload-spec.ts` — `APP_ONLY_NO_INFRA` (no DBs/queues/gateways as workload sidecars) + `SCOPED_CREDS_ONLY` (node-scoped DSNs, budget-capped LiteLLM virtual key, write-only Loki key — never fleet secrets).
- **Health:** off-k3s there are no probes and no Argo selfHeal — `OPERATOR_OWNS_WORKLOAD_HEALTH` (Axiom 26): the operator plane polls lease + `/readyz` + `/version.buildSha` and replaces dead-provider workloads. Reconcile loop is specified-not-built (story.5016); flag any design that assumes an Akash workload self-heals today.
- **Lease vocabulary:** one Akash lease is one rented `(node, environment)` workload bundle. A normal image update may update that lease in place; it does not inherently create a new rental. Catalog `lease_generation` is an internal **replacement ordinal** for a spent idempotence key, not a product generation, Kubernetes `metadata.generation`, or billing period. It advances only when a terminal receipt means the next activation must create a replacement.
- **Zero-downtime shape:** do not prescribe permanent `count=2`, nor a `1→2→update→1` surge, without provider proof. Akash service count is immutable for the current deployment API, so changing it requires lease replacement. The North-Star fallback is blue/green: create a new single-replica lease dark, prove exact SHA/assets/useful work, atomically move traffic, retain the old lease for rollback, then close it. That creates intentional short overlap, not perpetual double spend or churn on every reconciliation.
- **Do not confuse app compute with shared substrate.** Node app create/replace/scale belongs to the Akash compute API above. Merged production state/edge changes (`infra/compose/**`, Caddy/Alloy, VM-materialized bridge secrets) use `POST /api/v1/deploy/infra-reconcile` against the operator node. That route is temporarily production-promoter-gated as a two-phase OpenFGA bootstrap bridge; story.5028 owns the least-privilege destination `production_infra_promoter → can_reconcile_production_infra`. It is operator-GitHub-App-dispatched, preserves the current app pin, and accepts no caller workflow/ref/SHA/mode. Its repository target MUST come from env-scoped `NODE_SUBMODULE_PARENT_{OWNER,REPO}` so the candidate test App remains inside `cogni-test-org`. Never send an agent to personal `gh workflow run` or SSH for this normal reconcile path.
- **Seam warning:** `bootstrap/capabilities/compute.ts` (the `COMPUTE_WRITE_PROVIDER` selection point) is owned by the crypto-rail workstream — advise around it, never through it.
- Living design doc: hub entry `akash-node-deploy-v000` (+ `akash-provider-quality-mandate` for bid screening/boot SLO/blacklist); roadmap + task map: story.5016.
## Current infra reality — 2026-06-01 post-split candidate-a
The old shared candidate-a shape is retired. The Cogni monorepo and each node repo
are separate candidate-a targets and must not share VM aliases, DNS records, or
provisioning assumptions.
- **Cogni monorepo candidate-a** = apex `cogni-candidate-a.vm.cognidao.org` / `test.cognidao.org`.
⚠️ The VM IP is **not stable** — it is replaced on every reprovision (`84.32.9.111` here was the
2026-06-01 VM; as of the 2026-08-06 fleet reprovision it is `84.32.149.0`). **Never quote a VM IP
from this doc — read the live value from `~/dev/cogni-template/.local/provision-creds/README.md` (status table)
or the `candidate-a-kubeconfig.yaml` `server:` URL.**
Per-node `<node>-test` records are **catalog-driven** — auto-upserted to
this IP on flight by `reconcile-node-dns.sh` (ci-cd.md Axiom 21 `DNS_IS_RECONCILED_PER_ENV`),
never hand-maintained. Don't enumerate/edit them by hand; see the `dns-ops` skill.
- **Never quote a per-node candidate host or VM IP from this skill.** Node candidate hosts are
`<node>-test.cognidao.org`, catalog-driven and auto-upserted on flight; read it live (`GET /api/v1/nodes`, `infra/catalog/*.yaml`, `curl https://<host>/version`) — never hardcode a roster (Dolt `operator-node-catalog`).
- Use `candidate-flight.yml` for app digest promotion/Argo rollout and
`candidate-flight-infra.yml` for Compose/edge/runtime infra. If a PR changes
only edge/runtime infra, flight infra first; do not run the app lever unless
validation shows the app digest or deploy branch is stale.
- `candidate-flight.yml` may run read-only target substrate assertions before
an app rollout, but it must not run `deploy-infra.sh`. Missing substrate is a
loud handoff to `provision-env.yml`, `candidate-flight-infra.yml`, or the
preview/prod infra lane that owns the substrate mutation.
- The stale legacy VM `852637 / canary-cogni / 84.32.109.160` was destroyed.
**`type: infra` images — built in CI, deployed via Compose (feat/litellm-image-in-ci, in flight).** Shared VM-infra images (`infra/images/litellm/` = `FROM berriai/litellm` + `COPY cogni_callbacks.py`) were previously **hand-built + content-hash-pinned by a human** (`docker build … litellm-<hash>` → bump `LITELLM_IMAGE` in `deploy-infra.sh`) — outside build-once-promote, so callback changes were latent and the pin drifted. Now litellm is a first-class catalog target: `infra/catalog/litellm.yaml` (`type: infra` + `build_context`), built by the catalog-generic `build-and-push-images.sh`, **content-hash tagged** via `image-tags.sh:infra_image_tag`, and `deploy-infra.sh` resolves the identical tag (no hand-pin). `type:infra` is in `ALL_TARGETS` (build) but not `NODE_TARGETS`; the k8s plane (overlays/promotion/Argo/coverage) skips it via `image-tags.sh:is_infra_target`. A new VM-infra image is a one-file catalog drop. See [create-service.md](../../../docs/guides/create-service.md) §9b-infra + [ci-cd.md](../../../docs/spec/ci-cd.md) axiom 16. **Runtime stays Compose** — moving litellm into k8s/Argo is explicitly deferred (build-in-CI was the actual fix, not the runtime move).
`scripts/setup/provision-test-vm.sh` is legacy for post-split Cogni. Prefer
`scripts/setup/provision-env-vm.sh` for current env provisioning: it derives
node DBs, edge routes, and per-node env from the catalog. If you must touch the
legacy script, audit for single-node-era assumptions before using it against Cogni
monorepo candidate-a.
## Arsenal — know what exists before writing new code
> **Workflows are thin; logic lives in `scripts/ci/`.** Every non-trivial step in `.github/workflows/*.yml` is `run: bash scripts/ci/<name>.sh`. Start from the workflow to see the _shape_ of the pipeline, then open the scripts to see what actually happens. Never write new inline-YAML logic when a script exists — extend the script.
> **Node formation below describes only the LEGACY k3s app path** (AppSet + overlay footprint). New sovereign nodes are born on Akash. They have no k3s app overlay/AppSet, but they still have provider-neutral desired state and a machine-written `deploy/<env>-<slug>` pin. Do not use the legacy renderer as the model for new node formation.
>
> **Node formation is the exception to "everything is a workflow."** A new node's PR is authored by the **operator GitHub App directly** via the Git Data API (`nodes/operator/app/src/shared/node-app-scaffold/` + `adapters/server/vcs/github-repo-write.ts`) — no dispatched Action (task.5092; see [node-formation.md](../../../docs/spec/node-formation.md#node-publish-operator-authored-pr)). Its per-node Argo AppSet (`infra/k8s/argocd/<env>-<node>-applicationset.yaml` — one object per `(env, node)` for structural lane isolation, `bug.0378`) is catalog-rendered by `scripts/ci/render-node-appset.sh` (`pnpm gen:node-appset`, drift-gated in the `unit` job). The TS scaffolder and the shell renderer share one template (`scripts/ci/node-applicationset.yaml.tmpl`), so their output is byte-identical. Its per-node Kustomize **overlay** is likewise catalog-rendered — `scripts/ci/render-node-overlays.sh` (`pnpm gen:node-overlays`, drift-gated), the byte-exact twin of the operator's mint-time `gens/overlay.ts`, applying the node-at-root migrate rewrite (`/app/nodes/<slug>/app` → `/app/app`). The drift gate is what stops a **stale operator** from minting an overlay that kustomize-builds but crash-loops `migrate` at flight (`bug.5008`).
### Operator VCS routes — contributor CI + merge (operator App is the privilege bridge)
**STANDARD path = branch-push, not fork.** A developer requests node access (RBAC) declaring their
`githubLogin`; the owner's Approve provisions GitHub branch-push (collaborator) on the node repo for
that login. They then push a branch and open a **same-repo PR**, whose normal `pull_request` build
produces a flightable image — no operator dispatch needed. One-time setup lives in the
`developer-rbac-request` hub guide; the per-change loop in `cicd-e2e-required-sequence`.
**FALLBACK path = fork PR** (external agent with no grant, e.g. a read-only `flock-leader`): driven
entirely through the operator GitHub App — no human, no `gh` write on the agent's side:
- **`POST /api/v1/vcs/run-ci` `{ nodeId, prNumber }`** — releases GitHub's `action_required` hold on
the fork's `pull_request` runs **and** dispatches the trusted `pr-build` (`workflow_dispatch`) that
builds the fork head into a flightable image. RBAC `node.flight` on the named node is the gate; only
standard `pull_request` runs are approved (never `pull_request_target` / secret-bearing — safe by
structure). `owner/repo` resolved from the node's catalog `source_repo`.
- **`POST /api/v1/vcs/merge` `{ nodeId, prNumber }`** — merges any PR to the node's repo on green
(squash if no queue; **enqueues** if the node has a merge queue → response `{enqueued:true}`, poll
the PR to `MERGED`). RBAC `node.flight` is the gate; the owner-granted RBAC tuple IS the trust
boundary. **Branch protection on the node repo is the merge authority**; `evaluateMergeGate` is
fast-fail UX. Node-merge auto-promotes preview via `dispatchNodePreviewPromote`.
**Node ≡ monorepo — no split-brain.** The operator-dispatched trusted `pr-build` lane (`run-ci` →
`workflow_dispatch`) exists on nodes exactly as on the monorepo: `node-template`'s `pr-build.yml`
carries the `workflow_dispatch` trigger + fork-head resolve (nt #57), so fork PRs to a node get a
flightable image the same way they do on the monorepo. A node also self-builds on `push:main` →
`sha-<sha>`, so every merge is deployable.
**The operator is addressed like every node.** `POST /api/v1/vcs/merge { nodeId: "operator", prNumber }`
(slug or UUID) resolves to the monorepo — the operator is the one IN-REPO node, so its `nodeId` resolves
to `NODE_SUBMODULE_PARENT_*`, uniform with `run-ci` and `flight` (`{ nodeRef: { nodeId, sourceSha } }`).
There is no PR-number-only / `nodeId`-less merge lane and no `codePr` flight; `NODE_SCOPED_NEVER_RETARGETS`
keeps a typo'd slug a hard 404.
### Operator API — one control surface, exact proof
Use the operator API for normal CI/CD mutations; never replace these calls with a
personal `gh workflow run`:
| Intent | Call | Contract |
| ----------------------------------- | ------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------- |
| Flight any node, including operator | `POST /api/v1/vcs/flight {nodeRef:{nodeId,sourceSha}}` | Resolves the node repo from the catalog; the in-repo operator falls back to the parent monorepo. Requires a built immutable image. |
| Promote preview/production | `POST /api/v1/deploy/promote {nodeId,env,sourceSha}` | Advances exactly the named lane. A named target with no image hard-fails; it must never skip into green. |
| Merge | `POST /api/v1/vcs/merge {nodeId,prNumber,method}` | Enqueues when branch protection uses a merge queue; poll until actually merged. |
| Add/remove an environment | `POST /api/v1/nodes/{id}/envs {env,present}` | Authors a reviewable catalog PR. `present:true` derives placement and replacement ordinal; never silently defaults to k3s. |
`present:true` activates membership; it is **not** a promote. On merge, onboarding
replays each affected environment at that environment's own existing
`deploy/<env>-<slug>` pin. The catalog `source_sha` is birth-only fallback when no
deploy pin exists. Substrate reconciliation may run, but app advancement remains
the explicit flight/promote verb. This separation prevents a candidate or preview
membership change from silently reverting production.
Deployment truth has two parts and both must agree:
1. `.promote-state/source-sha-by-app.json` on `deploy/<env>-<slug>` states the intended pin.
2. The public host's `/version.buildSha` states what users actually receive; check every Cloudflare edge when edge behavior is in scope.
Workflow status, controller status, `/readyz`, and a green dispatch are supporting
signals only. A green run can still be a no-op, serve an old revision, or have no
workload behind it. For candidate validation, assert the expected SHA immediately
before exercising behavior because workflow concurrency ends when flight finishes;
a later flight can legitimately replace the shared candidate host.
### Workflows (`.github/workflows/`) — pipeline entry points
The job/script names below retain k3s history (`promote-k8s`, `overlay`,
`wait-for-argocd`). Do not infer placement from those names. For an Akash row the
same flight/promotion intent writes the per-node deploy pin and provider-neutral
Crossplane desired state; it does not create a k3s app overlay or AppSet.
| Workflow | Trigger | Purpose |
| ---------------------------------------- | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `pr-build.yml` | PR open/sync | Three-job pipeline: `detect` (affected targets) → `build` (matrix, one leg per target, parallel) → `manifest` (merge fragments → `build-manifest-pr-{N}-{sha}`) |
| `candidate-flight.yml` | workflow_dispatch | **App lever** for `candidate-a`: resolve PR digests from GHCR → **substrate-readiness lane** (`prepare-substrate-deploy-branch → materialize-substrate → reconcile-substrate → assert-substrate`; [ci-cd.md Axiom 22](../../../docs/spec/ci-cd.md), #1577/#1582) → write overlay → Argo reconciles pods. Matrix-fanned per `(env, target)` with `fail-fast: false` (task.0376): each cell pushes only its `deploy/candidate-a-<target>` branch + waits on only its own Argo Application, so sibling-target failure can't fail this cell. `concurrency: flight-candidate-a-<target>` is the lease (`BRANCH_HEAD_IS_LEASE`); the pre-matrix `acquire/release-candidate-slot` four-job slot model (bug.0321) is retired. Per-cell `verify-buildsha` gates on `promoted_apps != ''`. **No `deploy-infra` job** — the node self-heals its substrate via the lane above, the app lever never provisions broad Compose infra. (task.0314, task.0376) |
| `candidate-flight-infra.yml` | workflow_dispatch | **Infra lever** for `candidate-a`: rsync `infra/compose/**` from `--ref` (default `main`) + `compose up` on the VM. No digest promotion. (task.0314) |
| `flight-preview.yml` | push to main; workflow_dispatch | **No re-tag.** Resolves the merge's `sha-<mainSha>` images (published by `pr-build.yml` on merge_group AND push:main) and calls `flight-preview.sh`, which **always dispatches** the preview flight (`promote-and-deploy env=preview`) for the merged SHA (latest-wins; no lease). `promote-and-deploy` resolves the same `sha-<mainSha>` digest directly (`BUILD_ONCE_PROMOTE_DIGEST` / `SOURCE_SHA_IS_DEPLOY_IDENTITY`) — no `pr-`/`mq-`/`preview-` namespace. Emits `status=dispatched` to `$GITHUB_OUTPUT`; only a gated-off flight (a `workflow_dispatch` with a non-PR SHA) leaves `deploy-preview` **skipped** (grey). No silent-skip-as-green. |
| `build-multi-node.yml` | workflow_dispatch (manual fallback) | All-target build: per-node matrix + sequential services (migrator, scheduler-worker). Manual-only today — nothing auto-dispatches it (the workflow's own header still claims a `promote-merged-pr.yml` dispatcher that does not exist; treat that comment as stale). |
| `promote-and-deploy.yml` | workflow_dispatch (preview or production) | Multi-lane graph (task.0376, Axiom 22), all matrix-fanned per `(env, node)`: `decide → (reconcile-appset, reconcile-dns, materialize-substrate → reconcile-substrate) → promote-k8s → deploy-infra(if skip_infra!=true) → [verify-deploy, verify] → e2e → aggregate-{preview,production}`. `aggregate-{preview,production}` write the preview rollup `current-sha` — they replaced the retired `lock-preview-on-success`/`unlock-preview-on-failure` jobs (the preview review-lease was removed; preview is latest-wins). Inputs: `environment`, `source_sha`, `build_sha` (PR branch head SHA baked into images — differs from merge-commit `source_sha` on squash), `skip_infra` (**default true** — app-digest promotion is orthogonal to substrate, mirroring candidate-flight's no-deploy-infra app lever; flight-preview.sh + the operator-API `dispatchNodePromote` both pass true explicitly; set false only for `infra/compose/**`/ESO-secret/edge-topology changes). `verify-deploy` uses `build_sha` as EXPECTED_BUILDSHA and is job-level gated `if: needs.promote-k8s.result == 'success'` (Axiom 11); empty-promotion runs surface `verify-deploy`/`e2e` as skipped (grey) and the aggregator leaves `current-sha` unchanged. Preview is dispatched by `flight-preview.yml`; production is a direct human dispatch (bug.0361 — the old PR-dance via promote-to-production.yml was deleted). |
| `ci.yaml` | PR + push to main | Typecheck, lint, unit, component, stack tests |
| `release.yml` | workflow_dispatch | Cut `release/YYYYMMDD-<sha>` from `deploy/preview:.promote-state/current-sha`, open release PR into `main` |
| `auto-merge-release-prs.yml` | PR review | Auto-merge approved release/\* PRs |
| `require-pinned-release-prs-to-main.yml` | PR to main | Enforce release/\* branch + SHA pinning (currently over-broad — blocks feature PRs; see proj.cicd-services-gitops blockers) |
### CI scripts (`scripts/ci/`) — where the real logic is
Canonical PR-build + promotion path (bug.0321, task.0314, task.0324):
| Script | Purpose |
| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `detect-affected.sh` | Compute affected image targets from the PR diff; emits CSV + JSON matrix for `pr-build.yml` |
| `build-and-push-images.sh` | Per-target `docker buildx build --push` (invoked once per matrix leg with `TARGETS=<one>`) |
| `merge-build-fragments.sh` | Merge per-leg fragment JSONs into canonical `build-images.json` (consumed by `write-build-manifest`) |
| `write-build-manifest.sh` | Produce the `build-manifest-pr-{N}-{sha}` artifact |
| `lib/image-tags.sh` | **Single source of truth** for `ALL_TARGETS`, `NODE_TARGETS`, `tag_suffix_for_target()`, `image_tag_for_target()` (bug.0328). Sourced by `detect-affected.sh`, `build-and-push-images.sh`, `resolve-pr-build-images.sh`, `flight-preview.yml` image-resolve step (resolves `sha-<mainSha>` — no re-tag), `promote-and-deploy.yml` resolve+promote steps. Adding a new node is a one-file edit here — don't hand-maintain target lists in workflows. |
| `resolve-pr-build-images.sh` | Resolve pushed PR image digests from GHCR for the `pr-{N}-{sha}` tag convention (used by candidate-flight, not the PR workflow). Sources `lib/image-tags.sh` for the target catalog; includes per-node migrator tags and the top-level `source_sha` envelope field consumed by `promote-build-payload.sh`. |
| `promote-build-payload.sh` | Apply resolved digests to a deploy-branch overlay via `promote-k8s-image.sh`. Emits `promoted_apps=<csv>` to `$GITHUB_OUTPUT` **incrementally after each successful promotion** plus a `trap EXIT` belt-and-suspenders — the output survives mid-script abort (bug.0328). Records per-app `source_sha` into `.promote-state/source-sha-by-app.json` in a second pass so map failures cannot shadow `promoted_apps`. Partial map failures surface as `::warning::`; total map failure hard-fails the script so provenance decay is loud (bug.0321 Fix 1+4). |
| `promote-k8s-image.sh` | Lower-level overlay digest writer. Used directly by `promote-and-deploy.yml` and indirectly by `promote-build-payload.sh`. Treat as a primitive; never bypass it to edit overlay YAML by hand. |
| `update-source-sha-map.sh` | Shared primitive that merges a single `app → source_sha` entry into `.promote-state/source-sha-by-app.json` on the deploy branch (bug.0321 Fix 4). Called per promoted app by `promote-build-payload.sh` and by the `promote-and-deploy.yml` promote-k8s loop. |
Candidate / preview / production flight:
| Script | Purpose |
| --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `aggregate-rollup.sh <env>` | task.0376 — compute `current-sha = git merge-base $(deploy/<env>-<node> tips)` + merge per-node `source-sha-by-app.json` into the rollup preserving unaffected entries. Called by `promote-and-deploy.yml`'s `aggregate-{preview,production}` job; rebase-retries on push contention. `CURRENT_SHA_IS_MERGE_BASE` + `ROLLUP_MAP_PRESERVES_UNAFFECTED`. |
| `report-candidate-status.sh` | Post commit status on the PR head for candidate-flight progress. (Pre-task.0376 lease primitives `acquire-candidate-slot.sh` / `release-candidate-slot.sh` retired; per-node `concurrency: flight-<env>-<node>` is the lease — `BRANCH_HEAD_IS_LEASE`.) |
| `wait-for-candidate-ready.sh` | Endpoint readiness gate (HTTPS `/readyz`) — run inside the workflow. **Refuses to run** unless `ARGOCD_SYNC_VERIFIED=true` is set in the environment (runtime-enforced gate ordering, bug.0321 Fix 4). Set `ARGOCD_SYNC_VERIFIED=skip` for local/test callers that do not have Argo in the loop. |
| `wait-for-argocd.sh` | SSH into the VM and block until every app in `PROMOTED_APPS` (CSV, or the full catalog when unset) reports `sync.revision == EXPECTED_SHA && Healthy` **AND its new ReplicaSet has reached desired count** per app (`rollout_check`: `updatedReplicas >= spec.replicas` AND `availableReplicas >= spec.replicas`; bug.0326, [ci-cd.md Axiom 15](../../../docs/spec/ci-cd.md)). `EXPECTED_SHA` MUST be the deploy-branch tip SHA (not the source-app commit). The new-RS gate is required because Argo-level Healthy fires while the old ReplicaSet is still Ready during rolling updates — `/readyz` from those pods serves the prior BUILD_SHA. It deliberately does **not** run `kubectl rollout status` / wait for the old RS to drain (slow terminations false-fail healthy deploys). On success, writes `ARGOCD_SYNC_VERIFIED=true` to `$GITHUB_ENV` so downstream steps see the marker. |
| `smoke-candidate.sh` | Candidate HTTPS smoke over the catalog's node set — never a hardcoded list (`/readyz`, `/livez`) |
| `wait-for-in-cluster-services.sh` | `kubectl rollout status` on every k8s Deployment the catalog declares for the env, via SSH. Authoritative per-app rollout gate — catches "overlay unchanged, Argo no-op, pods stale". |
| `verify-buildsha.sh` | End-of-deploy contract probe: curls **`/version`** on node-apps, asserts **`.buildSha`** matches the expected SHA (task.0345 / #978). Map mode: `SOURCE_SHA_MAP` → `.promote-state/source-sha-by-app.json`. When **`NODES`** is set (workflows pass `promoted_apps`), verifies only that subset vs the map (task.0349 / affected-only). Single-SHA fallback when the map file is missing. Scheduler-worker and migrator filtered out (no Ingress). Runs in `verify-deploy` / `verify-candidate`. |
| `deploy-infra.sh` | SSH + rsync `infra/compose/**` from a git ref, then `compose up`. `--ref <git-ref>` (default `main`) |
| `assert-target-substrate.sh` | Read-only preflight for app flights. Reads `infra/catalog/$TARGET.yaml`, dispatches by `.type`, and currently implements the `type=node` substrate assertion (VM/k3s/Argo reachable, local catalog/overlay/AppSet present, scoped DNS, live edge/Caddy route, Service NodePort, Deployment-consumed Secret/ExternalSecret, node DB). `type=service` and `type=infra` fail explicitly until their own contracts exist. |
| `flight-preview.sh` | Always dispatch the preview flight (`promote-and-deploy env=preview`) for the merged SHA — latest-wins, no lease |
| `create-release.sh` | Cut `release/YYYYMMDD-<sha>` from preview `current-sha`, open release PR into `main` |
| `verify-deployment.sh` | Post-deploy verification |
Support:
| Script | Purpose |
| ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `validate-dsns.sh` | Validate DATABASE_URL and DATABASE_SERVICE_URL |
| `ensure-temporal-namespace.sh` | Create Temporal namespace if missing |
| `compute_migrator_fingerprint.sh` | Content-hash for migrator cache |
| `check-gitops-manifests.sh` | Validate k8s manifest correctness |
| `check-gitops-service-coverage.sh` | Ensure all services have k8s manifests |
| `test-image.sh` | Container livez smoke (used ad-hoc / by legacy paths) |
| `build.sh`, `build-service.sh`, `push.sh` | Legacy build scripts retained as callables by `build-multi-node.yml` fallback; new work uses `build-and-push-images.sh` |
| `deploy.sh` | Legacy full-stack SSH deploy; **retired** — use `deploy-infra.sh` + Argo CD reconciliation instead |
### Provision (`scripts/setup/`, `infra/provision/`)
| Resource | Purpose |
| ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `.github/workflows/provision-env.yml` | **Canonical new-env entry** — runner-owned bootstrap.sh → provision-env-vm.sh (VM + DNS + OpenBao/ESO + deploy). Guide: [`docs/guides/create-env.md`](../../../docs/guides/create-env.md) (incl. known e2e gaps) |
| _(no script — by design)_ | **Node-app compute needs NO VM provisioning** — an Akash-lane node app is created via the operator compute API against the existing env's state substrate (see Deployment targets section). `provision-env.yml` provisions the STATE substrate, not app compute. |
| `provision-env-vm.sh` | Current env provisioner (catalog-driven; candidate-\*/preview/production) |
| `provision-test-vm.sh` | Legacy one-command VM setup (candidate-a only) |
| `infra/provision/cherry/` | OpenTofu modules for Cherry Servers VMs |
| `infra/provision/cherry/k3s/` | k3s bootstrap (cloud-init, Argo CD, ksops) |
### VM SSH access — policy first, path second
**SSH is not a general tool in this repo. Reproducibility via infrastructure-as-code is the lifeblood — every ad-hoc SSH action is a latent drift source.** The rules below are hard, not aspirational:
1. **SSH is read-only by default.** Inspect pods, tail logs, `kubectl get`, `curl` from inside the network. Never `kubectl apply`, never `kubectl edit`, never `docker compose up` by hand, never edit files on the VM. The only sanctioned write path is a commit to git (code branch or `deploy/*`) that Argo / the pipeline reconciles.
2. **The only environment where write-mode SSH is permitted is `candidate-a`.** Even there it's a last resort for a 5-minute experiment that must be captured as a script, OpenTofu change, or GitOps commit **before the session ends**, per the repo-wide "if it isn't in git, it didn't happen" rule. A write-mode SSH to candidate-a that leaves no diff behind is a bug.
3. **Never SSH to production.** Not for reads, not for writes, not for diagnostics. If you need production signal, query Grafana/Loki via the MCP, check `/readyz` / `/livez` from outside the network, or read `deploy/production` state from git. If those don't answer the question, the question is asking you to violate the IaC contract — escalate, don't shell in.
4. **Preview reads are discouraged.** Prefer Loki / Grafana / `/readyz` from outside. A preview SSH is acceptable only when an external signal is provably missing and the read genuinely can't wait for instrumentation to land.
**Where SSH coordinates live (when you're allowed to use them)**
The sanctioned path — and the only one an agent should rely on — is what [`docs/guides/multi-node-deploy.md`](../../../docs/guides/multi-node-deploy.md) documents: `scripts/setup/provision-test-vm.sh` writes the SSH private key + VM IP to `.local/` (gitignored) at provision time.
| File | Purpose |
| ----------------------------- | ------------------------------------------------------------- |
| `.local/{env}-vm-key` | SSH private key (ed25519) for `root@<VM_IP>` |
| `.local/{env}-vm-ip` | VM IP address (single line) |
| `.local/{env}-vm-secrets.env` | Provision-time secrets (Argo bootstrap, sealed-secrets, etc.) |
| `.local/{env}-vm-age-key` | age key for sealed-secrets / sops on the VM |
`{env}` matches `DEPLOY_ENV` in `provision-test-vm.sh` — currently `candidate-a`, `preview`, `production`, `test`. The provisioner public key is committed at `infra/provision/cherry/base/keys/cogni_{env}_deploy.pub`.
Usage (candidate-a only; read-only): `ssh -i .local/candidate-a-vm-key root@$(cat .local/candidate-a-vm-ip) "kubectl -n argocd get applications"`.
**`.local/` lives on the provisioner device, not in worktrees.** It's gitignored, so a freshly cloned worktree (anything under a conductor / agent workspace path) will not have it. The canonical copy lives in the dev's primary cogni-template clone — reach for keys there, not in the active worktree. If you don't know your own primary-clone path, ask the human running the device; never hard-code home-directory paths into committed files (this skill is in a public repo).
**Legacy `canary` naming.** Some provisioner machines still have old
`.local/canary-vm-*` files, but current Cogni monorepo candidate-a should prefer
`.local/candidate-a-vm-key` and `.local/candidate-a-vm-ip` when present. Fall
back to `canary-vm-*` only if the current files are absent and you have verified
the IP matches `cogni-candidate-a.vm.cognidao.org`.
If `.local/{env}-vm-key` isn't present for the env you need, this device isn't the provisioner for that env. Do not improvise — don't source shell aliases from personal dotfiles, don't paste IPs from Slack, don't copy keys from a teammate. Either re-run `provision-test-vm.sh` (candidate-a only) or drive the change through the pipeline.
## Enforcement rules
When reviewing code that touches CI/CD, deploy, or infra:
- **Reuse existing scripts.** If `promote-k8s-image.sh` already updates overlays, don't write a new one. If `deploy-infra.sh` already handles Compose infra, extend it — don't create a parallel path.
- **No manual VM state.** Every change must be in a script or manifest. Ask: "if we destroy and reprovision this VM, does this change survive?"
- **Re-point ALL deploy pointers on VM migration (deploy-pointer drift).** When an env's VM is rebuilt/migrated to a new box, **three** pointers must be updated together, or CI→VM access silently dies: (1) the Cloudflare `<env>.vm.cognidao.org` A record, (2) the GH env secret `VM_HOST`, (3) the GH env secret `SSH_DEPLOY_KEY`. The app path is independent (Cloudflare→cluster keeps serving), so a stale pointer is INVISIBLE until a deploy/reconcile/promote runs — then it fails as a cryptic `exit 255 / Connection timed out / Permission denied`. **Incident (2026-06-10):** the Jun-3 prod migration `.202→.152` left all three stale → every prod reconcile/promote failed silently for days; cognidao.org stayed green the whole time. Fix is loud-by-default now: `reconcile-env-substrate.sh` SSH-preflights and emits a clear diagnosis. Verify `VM_HOST` resolves to the live Cherry VM (project 254586) and `SSH_DEPLOY_KEY` authenticates before assuming a deploy bug. promote-and-deploy + deploy-infra should grow the same preflight (follow-up).
- **No new mutable tags.** All image references use digests or `{env}-{sha}` tags.
- **No new long-lived code branches.** Feature branches are short-lived. Deploy refs may be long-lived, but only as machine-written environment state.
- **No PRs for deploy state.** Deploy branches get direct commits.
## Anti-patterns to flag
- **Assuming an Akash app ships via a k3s overlay/AppSet.** Akash-lane nodes (`deploy_provider: akash`) have no k3s app overlay, node AppSet, or k3s app Deployment. They **do** have a machine-written `deploy/<env>-<slug>` branch containing the canonical SHA pin and provider-neutral desired state. Advising a k3s overlay bump is a category error; denying the deploy branch exists is also wrong.
- **App-adjacent sidecars via k8s pod injection.** A node's second image is an extra service in the same Akash lease (`SIDECAR_IS_SDL_SERVICE`, ci-cd.md Axiom 25) — the overlay/kustomize injection lane was closed with PR #1884; don't resurrect it.
- **Treating replica count as rollout strategy.** Permanent `count=2` spends roughly twice the ingress-service resources continuously; `1→2→1` is not an in-place surge when provider count is immutable. Follow `akash-promotion-north-star`: prove provider semantics, then use native rolling behavior if proven or create/prove/flip/retain-old blue/green leases if not.
- **Calling candidate concurrency a review lease.** Per-node workflow concurrency prevents simultaneous deploy mutation, but the completed flight does not reserve the candidate host for a reviewer. Re-check `/version.buildSha` at validation start and fail closed if another flight replaced it.
- Rebuilding in production instead of promoting proven digest
- **Growing the frozen deploy brain** (see Freeze Policy, top). A PR that adds a new responsibility / secret / service / DB path / inline decision tree to `deploy-infra.sh`, a net-new infra-mutating `scripts/ci/*.sh`, or a new deploy/promote/provision workflow — when the behavior could be a catalog field, overlay/AppSet, ESO declaration, or a `DeployCapability` method. Flag it; route it per the request→home table. (Bug fixes + catalog-driven + guard-tightening are fine in place.)
- **NODE_RESOLUTION_IS_A_DB_READ (north-star): the operator resolves nodes (list, id→slug, id→URL) from its OWN DB registry — never a static `COGNI_NODE_ENDPOINTS` map, and never by splicing a configmap at node-formation.** The operator app has `resolveNodeRegistry`/`listRoutableNodes` (see [cicd-platform-boundary.md](../../../docs/spec/cicd-platform-boundary.md) "resolve the node via the shared registry"); the ADDRESS comes from `NodeAddressPort` (`node-address.adapter.ts`), which resolves the node's declared `deployment_provider.<env>` — `http://<slug>-node-app:3000` for a k3s node, the node's public canonical host for one placed off-cluster (`PLACEMENT_DECIDES_THE_ADDRESS`, bug.5106). Never hardcode `internalNodeAppUrl` in a new operator→node client: that assumption silently dropped every attribution receipt for an Akash-placed node. The **static `COGNI_NODE_ENDPOINTS` configmap exists only for the DB-less `scheduler-worker`** (no DB → must consume a catalog-rendered map). Flag any operator-side feature that: reads `COGNI_NODE_ENDPOINTS` in the app, adds nodes to a static endpoint map, or edits `github-repo-write.ts` / a base configmap to keep such a map in sync — route it to a registry read instead. Reference: the receipt-delivery gateway (#1924) shipped copying the worker's static-map pattern into the operator app → empty node epochs until it was moved to the registry (story.5001).
- `:latest` or mutable tags in deploy manifests
- One-off SSH edits not captured in scripts
- PRs for machine-written deploy-branch updates
- Auto-release PR on every E2E success
- Ungated promotion (CI red but code advances)
- `cancel-in-progress: true` on deploy workflows
- Unbounded health-check polling without timeout
- Duplicating logic that already exists in `scripts/ci/`
- Running `deploy-infra.sh` from `candidate-flight.yml` or any app digest
flight. App flight can assert substrate; infra levers provision substrate.
- Naming a generic deployable gate after one catalog shape. `node` checks are a
branch under `assert-target-substrate.sh`, not the generic flight abstraction.
- **Silent-success on no-op runs.** A workflow that exits 0 without doing the work paints a green check that operators misread as "deployed". Every no-op path must produce a visually-distinct outcome: exit-code convention (`1=error, 2=no-op`) + a gated follow-up job that surfaces as skipped, OR a loud `::warning::` + step summary. Never silently succeed after a skip.
- **Using catalog `source_sha` as deployed truth.** It is the birth pin only. After first deployment, the environment's own `deploy/<env>-<slug>/.promote-state/source-sha-by-app.json` is intended truth; reading the catalog during lane onboarding can silently revert a live environment.
- **Status / summary logic in YAML.** Decision logic (exit-code → status name, summary markdown, banner copy) belongs in `scripts/ci/*.sh`. Workflows call the script and gate jobs on its `$GITHUB_OUTPUT`. Inline `case $rc in ...` + `echo >> $GITHUB_STEP_SUMMARY` in YAML is a smell — move it into the script.
- **`/readyz 200` treated as rollout proof.** `/readyz` is Service-level: answered by any running pod, old or new. Real rollout proof is **`/version.buildSha` matches the expected SHA** (`verify-buildsha.sh`; task.0345) — the ONLY provider-agnostic proof, and the only one that exists on the Akash lane. On the k3s lane, `kubectl rollout status` (`wait-for-in-cluster-services.sh`) or the new-RS replica-count assertion in `wait-for-argocd.sh` (`rollout_check`) are acceptable k8s-specific variants (never `/readyz`).
- **Argo `health.status == Healthy` treated as rollout proof** (bug.0326). Argo's Healthy fires as soon as enough pods are Ready — including old-RS pods during a rolling update. `/readyz` from those pods returns the prior BUILD_SHA, so verify-buildsha fails on a green-upstream flight. `wait-for-argocd.sh` now asserts the new ReplicaSet has reached desired count (`rollout_check`: `updatedReplicas >= spec.replicas` AND `availableReplicas >= spec.replicas`) per promoted app after the Application-level Healthy check ([ci-cd.md Axiom 15](../../../docs/spec/ci-cd.md)); don't regress that to Application-health-only.
- **Skipped verification treated as success** (bug.0328; per-node since task.0376). A terminal consumer of a skipped `verify-candidate`/`verify-deploy` must distinguish real no-op from contradiction: candidate-flight's `report-status` keys on `has_flight_targets` (skip is green only when there were no flight targets); preview/prod's `aggregate-{preview,production}` assert every promoted cell has a `verified-<node>.txt` marker (`aggregate-decide-outcome.sh`, [ci-cd.md Axioms 14 & 19](../../../docs/spec/ci-cd.md)). Skipped + had-work = hard-fail.
- **Hand-maintained target lists in workflows** (bug.0328). Every `ALL_TARGETS=(...)` or per-node `promote_app` array inlined in a workflow `run:` block is drift waiting to happen — the next new node gets half-shipped when someone updates the lib but forgets the workflow. Source `scripts/ci/lib/image-tags.sh`, iterate `ALL_TARGETS` / `NODE_TARGETS`.
- **Step-level `if:` for verification gates.** GitHub treats a skipped _step_ inside a running job as contributing to job success. When a verification should be allowed to skip (e.g. empty `promoted_apps`), model it as a _job-level_ gate with `needs:` and `if:` — the job then surfaces as visibly skipped (grey in the checks list), not green. Step-level `if: promoted_apps != ''` is the silent-green primitive that bug.0321 hunted down.
- **Gate-ordering by convention, not by runtime.** "X must run before Y" embedded in a comment rots at the next refactor. Enforce it structurally: the upstream script writes a marker to `$GITHUB_ENV` (e.g. `ARGOCD_SYNC_VERIFIED=true`); the downstream script refuses to run without it. `wait-for-argocd.sh` → `wait-for-candidate-ready.sh` is the reference pattern (bug.0321 Fix 4).
- **Cross-PR contract verification via overlay inspection alone.** Production promotions copy preview's overlays, which can mix digests from different PR head SHAs (affected-only CI). A single `EXPECTED_BUILDSHA` is meaningless there. The canonical pattern: write a per-app `source_sha` map to `.promote-state/source-sha-by-app.json` at promotion time, copy it forward preview → production, read it in the verifier. Artifact provenance travels with the artifact.
- **Deploy workflows triggering app-level business logic.** `candidate-flight-infra.yml` / `deploy-infra.sh` / `promote-and-deploy.yml` exist to reconcile infrastructure state — Compose stacks, k8s overlays, image digests. They MUST NOT `curl` an app endpoint to kick off business work (data sync, cache warm, third-party mirror, index rebuild). When the deploy script becomes a cron, every property of the deploy pipeline (lease semantics, retry budget, timeout, secret blast radius, "you can't dispatch infra without WORK_ITEMS_NOTION_TOKEN") is now coupled to whatever app feature added the curl. Apps schedule themselves — internal cron, scheduler-worker job, on-startup hook, or a dedicated workflow with its own trigger. Reference incident: PR #1177 added a Notion mirror sync as a post-deploy `curl` step in `candidate-flight-infra.yml`; reverted in full because "env-gated, no impact" is a lie when the gate is in the deploy pipeline. **devops-expert MUST be a required code reviewer on any PR touching `.github/workflows/*`, `scripts/ci/**`, `infra/**`, or `deploy/\*` — including PRs whose primary purpose is an app feature that just "happens to" add an env var or a curl step to a deploy file.**
