CI/CD architecture reviewer and DevOps advisor for monorepo pipelines, branch workflows, and deployment strategy. Use this skill when reviewing GitHub Actions workflows, designing promotion pipelines, evaluating branch protection rules, reviewing deploy scripts, auditing CI/CD for anti-patterns, or making decisions about build/test/deploy strategy. Also triggers for: 'review this workflow', 'is this CI pattern correct', 'branch strategy', 'deploy pipeline design', 'release promotion', 'should...
Scanned 9/9/2026
Install to Claude Code
Skills Directorynpx -y skills add cogni-dao/cogni --skill devops-expert --agent claude-codeInstalls into .claude/skills of the current project.
Are you the author of Devops Expert?
Add the live security badge to your README — it updates automatically with every re-scan.
[](https://www.skillsdirectory.com/skills/cogni-dao-devops-expert)More formats (shields.io, HTML) on the badges page.
---
name: devops-expert
description: "CI/CD architecture reviewer and DevOps advisor for monorepo pipelines, branch workflows, and deployment strategy. Use this skill when reviewing GitHub Actions workflows, designing promotion pipelines, evaluating branch protection rules, reviewing deploy scripts, auditing CI/CD for anti-patterns, or making decisions about build/test/deploy strategy. Also triggers for: 'review this workflow', 'is this CI pattern correct', 'branch strategy', 'deploy pipeline design', 'release promotion', 'should we use PRs for this', 'CI is slow', 'pipeline gap analysis', 'provision', 'deploy script', 'akash', 'decentralized compute', 'compute deployment', 'deploy_provider', 'SDL', 'lease', 'where do node apps deploy'."
---
# DevOps Expert
You are a senior DevOps architect. AI agents are the primary committers in this repo. The pipeline must absorb high-volume churn without PR noise or manual gates.
## Ground truth — read before advising
- **[CI/CD Platform Boundary & Freeze Policy](../../../docs/spec/cicd-platform-boundary.md) — READ FIRST when advising on ANY new deployment/platform behavior.** The deploy brain (`scripts/ci/*.sh` + `.github/workflows/*.yml`) is **frozen** for the operator control plane: no new platform logic in bash/YAML, no new deploy/promote/provision workflow, no new infra/secret-mutating `.sh`. `deploy-infra.sh` (2,167 lines) is a 🔴 DANGER ZONE on a line-count ratchet. New platform work routes to the substrate (catalog row / Kustomize overlay / Argo AppSet / ESO declaration / OpenTofu) per the doc's request→home table, OR into the typed `.ts` operator control plane (`DeployCapability` + `ComputeResourcePort` — the Akash compute lane is SHIPPED and is the home for node-app deployment behavior, see the Deployment-targets section below; never route compute-lane work into catalog-overlay/AppSet artifacts). When reviewing, the gate is: _bug-fix / catalog-driven / guard-tightening = OK in place; new branching, env policy, promotion semantics, secret/domain/lifecycle rules = platform work, NOT script work._ Standalone-node sovereignty (a node's own GH Actions) is explicitly NOT frozen.
- [Multi-Repo Sync Contract](../../../docs/spec/repo-sync-contract.md) — operator-scope content lives in `Cogni-DAO/cogni` (HUB); `node-template` and `cogni-poly` are artifacts. `.cogni/sync-manifest.yaml` declares global excludes + per-artifact divergences; `.github/workflows/sync-drift-detector.yml` runs daily + on push:main, upserts a hub issue labeled `sync-drift` listing drift in three classes (🟡 different / 🔴 missing-on-artifact / 🟣 only-on-artifact). **Any review of a workflow / `infra/**`/`scripts/**`/`.github/**`change in any of the three repos MUST consider sync impact** — backflow refactors (substrate work pioneered in node-template, e.g. OpenBao/ESO) require a named same-day porter committed before merge, else drift accumulates. The`sync-drift` issue is the cross-repo dashboard.
- [CI/CD Spec](../../../docs/spec/ci-cd.md) — operating rules, branch model, pipeline chain, environments, TODOs
- [Candidate Flight V0 Guide](../../../docs/guides/candidate-flight-v0.md) — short operator guide for flying one selected PR to `candidate-a`
- [CI/CD Spec — Axiom 18 `BRANCH_HEAD_IS_LEASE`](../../../docs/spec/ci-cd.md) — slot lease semantics: the per-`(env, node)` branch head is the lease (the old `acquire/release-candidate-slot.sh` + `candidate-lease.json` model is retired)
- [CI/CD Project Scorecard](../../../work/projects/proj.cicd-services-gitops.md) — pipeline health, active blockers
- [`promote` skill](../promote/SKILL.md) — operator playbook for preview/prod promotion, lease semantics, monitoring poll-loop, and the bug.0443 admin-merge / affected-only failure modes (use this when the question is "how do I ship X to preview/prod" rather than "is this CI design correct")
- `.github/workflows/` — actual workflow source (verify claims against code)
- `scripts/ci/` — CI scripts (verify what exists before writing new ones)
- `scripts/ci/sync-node-template-fork-pr.sh` — repeatable emergency refresh for
a node-template fork PR branch from `cogni-test-org/node-template`
- `scripts/setup/provision-test-vm.sh` — single-command VM provisioning
- `infra/provision/` — OpenTofu modules (Cherry Servers, k3s)
- `infra/k8s/` — Kustomize bases, overlays, Argo ApplicationSets, catalog
- **Provisioning artifacts & env root creds — DO NOT grep the repo/disk blind for these; they are not in git and never will be.** A `provision-env.yml` run uploads `<env>-init-artifacts` (kubeconfig, VM SSH key, OpenBao root+unseal, age key) as a GHA artifact with **1-day retention** — expired ~24h later, so `gh run download` is a dead end after a day. The **only** durable, validated copies live laptop-local at `~/dev/cogni-template/.local/provision-creds/<env>/` (gitignored, chmod 600) — that dir's `README.md` is the custody SSoT — its per-env status table carries the **current** VM IP, and each `<env>-kubeconfig.yaml` `server:` URL is the second live source. To find _which_ run provisioned an env: `gh run list --workflow=provision-env.yml`. Full custody + recovery contract (Gotcha 14: a clean decrypt is NOT proof of valid custody) lives in the [`provision-env`](../provision-env/SKILL.md) skill — read it, don't re-derive it here. **Never trust a VM IP hardcoded in a skill/doc; read it from that `README.md` status table or the kubeconfig `server:`.**
## Core principles
1. **Reproducibility.** Every environment must be rebuildable from scratch via scripts and manifests. One-off VM edits are forbidden — if you SSH in and change something by hand, it must be captured in provision scripts, deploy-infra.sh, or k8s manifests. If a VM dies, `provision-test-vm.sh` + the pipeline must recreate the full environment without manual steps.
2. **Build once, promote by digest.** No rebuilds downstream. `@sha256:` refs, never mutable tags. The image that runs in production is the exact image accepted in candidate flight.
3. **Deploy state lives in git, separate from code.** Deploy branches like `deploy/candidate-a`, `deploy/preview`, and `deploy/production` hold rendered overlay state. Provision scripts and CI write directly to deploy branches. Code branches never contain deploy state. Direct bot commits, never PRs.
4. **Standard CI is universal; candidate flight is explicit.** All PRs get normal CI/build. Only selected PRs enter candidate flight. Nothing promotes to preview unless the accepted digest is the same one already proven safe.
5. **Policy-gated promotion.** Preview success informs whether humans promote to production. Do not invent a default release-branch conveyor when the specs say trunk-based promotion.
6. **Affected-only CI.** Never rebuild/retest the world. Mandatory at scale.
## Deployment targets — Akash app lane vs Cherry state substrate
**The standard (gated north star — `AKASH_IS_NODE_APP_TARGET`, [ci-cd.md](../../../docs/spec/ci-cd.md) Axioms 23–26):** node apps deploy as **app-only Akash workloads** through the operator compute API; the node wizard's default is `deploy_provider: akash` (born-on-Akash). The Cherry VM/k3s cluster is the **state substrate** (postgres/doltgres/redis/temporal/LiteLLM/scheduler-worker) plus the legacy node-app lane. Gate: present-tense once story.5016's S0–S3 land; until then new nodes still birth on k3s — but never advise extending the k3s app lane, and route every new deploy-plane behavior toward the Akash lane.
Mechanism (all shipped, task.5044 / PR #2077):
- Route: `POST /api/v1/compute/deployments` (create/status/release) → `ComputeResourcePort` → `AkashComputeAdapter` (`nodes/operator/app/src/adapters/server/compute/akash-compute.adapter.ts`, Console managed-wallet API).
- SDL renders internally in `akash-sdl.ts` and never escapes that dir. `INTERNAL_EXPOSE_IS_MESH`: multiple services in one lease reach each other by service name with no public expose — which is why an app-adjacent sidecar image is **an extra SDL service in the same lease**, never k8s pod injection (`SIDECAR_IS_SDL_SERVICE`; the injection lane was closed with PR #1884).
- Workload spec: `features/compute/node-workload-spec.ts` — `APP_ONLY_NO_INFRA` (no DBs/queues/gateways as workload sidecars) + `SCOPED_CREDS_ONLY` (node-scoped DSNs, budget-capped LiteLLM virtual key, write-only Loki key — never fleet secrets).
- **Health:** off-k3s there are no probes and no Argo selfHeal — `OPERATOR_OWNS_WORKLOAD_HEALTH` (Axiom 26): the operator plane polls lease + `/readyz` + `/version.buildSha` and replaces dead-provider workloads. Reconcile loop is specified-not-built (story.5016); flag any design that assumes an Akash workload self-heals today.
- **Seam warning:** `bootstrap/capabilities/compute.ts` (the `COMPUTE_WRITE_PROVIDER` selection point) is owned by the crypto-rail workstream — advise around it, never through it.
- Living design doc: hub entry `akash-node-deploy-v000` (+ `akash-provider-quality-mandate` for bid screening/boot SLO/blacklist); roadmap + task map: story.5016.
## Current infra reality — 2026-06-01 post-split candidate-a
The old shared candidate-a shape is retired. Cogni monorepo and `cogni-poly`
are separate candidate-a targets and must not share VM aliases, DNS records, or
provisioning assumptions.
- **Cogni monorepo candidate-a** = apex `cogni-candidate-a.vm.cognidao.org` / `test.cognidao.org`.
⚠️ The VM IP is **not stable** — it is replaced on every reprovision (`84.32.9.111` here was the
2026-06-01 VM; as of the 2026-08-06 fleet reprovision it is `84.32.149.0`). **Never quote a VM IP
from this doc — read the live value from `~/dev/cogni-template/.local/provision-creds/README.md` (status table)
or the `candidate-a-kubeconfig.yaml` `server:` URL.**
Per-node `<node>-test` records are **catalog-driven** — auto-upserted to
this IP on flight by `reconcile-node-dns.sh` (ci-cd.md Axiom 21 `DNS_IS_RECONCILED_PER_ENV`),
never hand-maintained. Don't enumerate/edit them by hand; see the `dns-ops` skill.
- `5.199.173.155` is **cogni-poly candidate-a only**. Cloudflare records
observed on 2026-06-01: `candidate-a.vm.cognidao.org` and
`poly-test.cognidao.org`.
- Use `candidate-flight.yml` for app digest promotion/Argo rollout and
`candidate-flight-infra.yml` for Compose/edge/runtime infra. If a PR changes
only edge/runtime infra, flight infra first; do not run the app lever unless
validation shows the app digest or deploy branch is stale.
- `candidate-flight.yml` may run read-only target substrate assertions before
an app rollout, but it must not run `deploy-infra.sh`. Missing substrate is a
loud handoff to `provision-env.yml`, `candidate-flight-infra.yml`, or the
preview/prod infra lane that owns the substrate mutation.
- The stale legacy VM `852637 / canary-cogni / 84.32.109.160` was destroyed.
**`type: infra` images — built in CI, deployed via Compose (feat/litellm-image-in-ci, in flight).** Shared VM-infra images (`infra/images/litellm/` = `FROM berriai/litellm` + `COPY cogni_callbacks.py`) were previously **hand-built + content-hash-pinned by a human** (`docker build … litellm-<hash>` → bump `LITELLM_IMAGE` in `deploy-infra.sh`) — outside build-once-promote, so callback changes were latent and the pin drifted. Now litellm is a first-class catalog target: `infra/catalog/litellm.yaml` (`type: infra` + `build_context`), built by the catalog-generic `build-and-push-images.sh`, **content-hash tagged** via `image-tags.sh:infra_image_tag`, and `deploy-infra.sh` resolves the identical tag (no hand-pin). `type:infra` is in `ALL_TARGETS` (build) but not `NODE_TARGETS`; the k8s plane (overlays/promotion/Argo/coverage) skips it via `image-tags.sh:is_infra_target`. A new VM-infra image is a one-file catalog drop. See [create-service.md](../../../docs/guides/create-service.md) §9b-infra + [ci-cd.md](../../../docs/spec/ci-cd.md) axiom 16. **Runtime stays Compose** — moving litellm into k8s/Argo is explicitly deferred (build-in-CI was the actual fix, not the runtime move).
`scripts/setup/provision-test-vm.sh` is legacy for post-split Cogni. Prefer
`scripts/setup/provision-env-vm.sh` for current env provisioning: it derives
node DBs, edge routes, and per-node env from the catalog. If you must touch the
legacy script, audit for poly-era assumptions before using it against Cogni
monorepo candidate-a.
## Arsenal — know what exists before writing new code
> **Workflows are thin; logic lives in `scripts/ci/`.** Every non-trivial step in `.github/workflows/*.yml` is `run: bash scripts/ci/<name>.sh`. Start from the workflow to see the _shape_ of the pipeline, then open the scripts to see what actually happens. Never write new inline-YAML logic when a script exists — extend the script.
> **Node formation below describes the LEGACY k3s birth path** (AppSet + overlay footprint). The standard is born-on-Akash — wizard `deploy_provider: akash`, no AppSet/overlay footprint — gated on story.5016 (ci-cd.md Axiom 23); until the wizard flips, this is still what runs.
>
> **Node formation is the exception to "everything is a workflow."** A new node's PR is authored by the **operator GitHub App directly** via the Git Data API (`nodes/operator/app/src/shared/node-app-scaffold/` + `adapters/server/vcs/github-repo-write.ts`) — no dispatched Action (task.5092; see [node-formation.md](../../../docs/spec/node-formation.md#node-publish-operator-authored-pr)). Its per-node Argo AppSet (`infra/k8s/argocd/<env>-<node>-applicationset.yaml` — one object per `(env, node)` for structural lane isolation, `bug.0378`) is catalog-rendered by `scripts/ci/render-node-appset.sh` (`pnpm gen:node-appset`, drift-gated in the `unit` job). The TS scaffolder and the shell renderer share one template (`scripts/ci/node-applicationset.yaml.tmpl`), so their output is byte-identical. Its per-node Kustomize **overlay** is likewise catalog-rendered — `scripts/ci/render-node-overlays.sh` (`pnpm gen:node-overlays`, drift-gated), the byte-exact twin of the operator's mint-time `gens/overlay.ts`, applying the node-at-root migrate rewrite (`/app/nodes/<slug>/app` → `/app/app`). The drift gate is what stops a **stale operator** from minting an overlay that kustomize-builds but crash-loops `migrate` at flight (`bug.5008`).
### Operator VCS routes — contributor CI + merge (operator App is the privilege bridge)
**STANDARD path = branch-push, not fork.** A developer requests node access (RBAC) declaring their
`githubLogin`; the owner's Approve provisions GitHub branch-push (collaborator) on the node repo for
that login. They then push a branch and open a **same-repo PR**, whose normal `pull_request` build
produces a flightable image — no operator dispatch needed. One-time setup lives in the
`developer-rbac-request` hub guide; the per-change loop in `cicd-e2e-required-sequence`.
**FALLBACK path = fork PR** (external agent with no grant, e.g. a read-only `flock-leader`): driven
entirely through the operator GitHub App — no human, no `gh` write on the agent's side:
- **`POST /api/v1/vcs/run-ci` `{ nodeId, prNumber }`** — releases GitHub's `action_required` hold on
the fork's `pull_request` runs **and** dispatches the trusted `pr-build` (`workflow_dispatch`) that
builds the fork head into a flightable image. RBAC `node.flight` on the named node is the gate; only
standard `pull_request` runs are approved (never `pull_request_target` / secret-bearing — safe by
structure). `owner/repo` resolved from the node's catalog `source_repo`.
- **`POST /api/v1/vcs/merge` `{ nodeId, prNumber }`** — merges any PR to the node's repo on green
(squash if no queue; **enqueues** if the node has a merge queue → response `{enqueued:true}`, poll
the PR to `MERGED`). RBAC `node.flight` is the gate; the owner-granted RBAC tuple IS the trust
boundary. **Branch protection on the node repo is the merge authority**; `evaluateMergeGate` is
fast-fail UX. Node-merge auto-promotes preview via `dispatchNodePreviewPromote`.
**Node ≡ monorepo — no split-brain.** The operator-dispatched trusted `pr-build` lane (`run-ci` →
`workflow_dispatch`) exists on nodes exactly as on the monorepo: `node-template`'s `pr-build.yml`
carries the `workflow_dispatch` trigger + fork-head resolve (nt #57), so fork PRs to a node get a
flightable image the same way they do on the monorepo. A node also self-builds on `push:main` →
`sha-<sha>`, so every merge is deployable.
**The operator is addressed like every node.** `POST /api/v1/vcs/merge { nodeId: "operator", prNumber }`
(slug or UUID) resolves to the monorepo — the operator is the one IN-REPO node, so its `nodeId` resolves
to `NODE_SUBMODULE_PARENT_*`, uniform with `run-ci` and `flight` (`{ nodeRef: { nodeId, sourceSha } }`).
There is no PR-number-only / `nodeId`-less merge lane and no `codePr` flight; `NODE_SCOPED_NEVER_RETARGETS`
keeps a typo'd slug a hard 404.
### Workflows (`.github/workflows/`) — pipeline entry points
| Workflow | Trigger | Purpose |
| ---------------------------------------- | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `pr-build.yml` | PR open/sync | Three-job pipeline: `detect` (affected targets) → `build` (matrix, one leg per target, parallel) → `manifest` (merge fragments → `build-manifest-pr-{N}-{sha}`) |
| `candidate-flight.yml` | workflow_dispatch | **App lever** for `candidate-a`: resolve PR digests from GHCR → **substrate-readiness lane** (`prepare-substrate-deploy-branch → materialize-substrate → reconcile-substrate → assert-substrate`; [ci-cd.md Axiom 22](../../../docs/spec/ci-cd.md), #1577/#1582) → write overlay → Argo reconciles pods. Matrix-fanned per `(env, target)` with `fail-fast: false` (task.0376): each cell pushes only its `deploy/candidate-a-<target>` branch + waits on only its own Argo Application, so sibling-target failure can't fail this cell. `concurrency: flight-candidate-a-<target>` is the lease (`BRANCH_HEAD_IS_LEASE`); the pre-matrix `acquire/release-candidate-slot` four-job slot model (bug.0321) is retired. Per-cell `verify-buildsha` gates on `promoted_apps != ''`. **No `deploy-infra` job** — the node self-heals its substrate via the lane above, the app lever never provisions broad Compose infra. (task.0314, task.0376) |
| `candidate-flight-infra.yml` | workflow_dispatch | **Infra lever** for `candidate-a`: rsync `infra/compose/**` from `--ref` (default `main`) + `compose up` on the VM. No digest promotion. (task.0314) |
| `flight-preview.yml` | push to main; workflow_dispatch | **No re-tag.** Resolves the merge's `sha-<mainSha>` images (published by `pr-build.yml` on merge_group AND push:main) and calls `flight-preview.sh`, which **always dispatches** the preview flight (`promote-and-deploy env=preview`) for the merged SHA (latest-wins; no lease). `promote-and-deploy` resolves the same `sha-<mainSha>` digest directly (`BUILD_ONCE_PROMOTE_DIGEST` / `SOURCE_SHA_IS_DEPLOY_IDENTITY`) — no `pr-`/`mq-`/`preview-` namespace. Emits `status=dispatched` to `$GITHUB_OUTPUT`; only a gated-off flight (a `workflow_dispatch` with a non-PR SHA) leaves `deploy-preview` **skipped** (grey). No silent-skip-as-green. |
| `build-multi-node.yml` | workflow_dispatch (manual fallback) | All-target build: per-node matrix + sequential services (migrator, scheduler-worker). Manual-only today — nothing auto-dispatches it (the workflow's own header still claims a `promote-merged-pr.yml` dispatcher that does not exist; treat that comment as stale). |
| `promote-and-deploy.yml` | workflow_dispatch (preview or production) | Multi-lane graph (task.0376, Axiom 22), all matrix-fanned per `(env, node)`: `decide → (reconcile-appset, reconcile-dns, materialize-substrate → reconcile-substrate) → promote-k8s → deploy-infra(if skip_infra!=true) → [verify-deploy, verify] → e2e → aggregate-{preview,production}`. `aggregate-{preview,production}` write the preview rollup `current-sha` — they replaced the retired `lock-preview-on-success`/`unlock-preview-on-failure` jobs (the preview review-lease was removed; preview is latest-wins). Inputs: `environment`, `source_sha`, `build_sha` (PR branch head SHA baked into images — differs from merge-commit `source_sha` on squash), `skip_infra` (**default true** — app-digest promotion is orthogonal to substrate, mirroring candidate-flight's no-deploy-infra app lever; flight-preview.sh + the operator-API `dispatchNodePromote` both pass true explicitly; set false only for `infra/compose/**`/ESO-secret/edge-topology changes). `verify-deploy` uses `build_sha` as EXPECTED_BUILDSHA and is job-level gated `if: needs.promote-k8s.result == 'success'` (Axiom 11); empty-promotion runs surface `verify-deploy`/`e2e` as skipped (grey) and the aggregator leaves `current-sha` unchanged. Preview is dispatched by `flight-preview.yml`; production is a direct human dispatch (bug.0361 — the old PR-dance via promote-to-production.yml was deleted). |
| `ci.yaml` | PR + push to main | Typecheck, lint, unit, component, stack tests |
| `release.yml` | workflow_dispatch | Cut `release/YYYYMMDD-<sha>` from `deploy/preview:.promote-state/current-sha`, open release PR into `main` |
| `auto-merge-release-prs.yml` | PR review | Auto-merge approved release/\* PRs |
| `require-pinned-release-prs-to-main.yml` | PR to main | Enforce release/\* branch + SHA pinning (currently over-broad — blocks feature PRs; see proj.cicd-services-gitops blockers) |
### CI scripts (`scripts/ci/`) — where the real logic is
Canonical PR-build + promotion path (bug.0321, task.0314, task.0324):
| Script | Purpose |
| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `detect-affected.sh` | Compute affected image targets from the PR diff; emits CSV + JSON matrix for `pr-build.yml` |
| `build-and-push-images.sh` | Per-target `docker buildx build --push` (invoked once per matrix leg with `TARGETS=<one>`) |
| `merge-build-fragments.sh` | Merge per-leg fragment JSONs into canonical `build-images.json` (consumed by `write-build-manifest`) |
| `write-build-manifest.sh` | Produce the `build-manifest-pr-{N}-{sha}` artifact |
| `lib/image-tags.sh` | **Single source of truth** for `ALL_TARGETS`, `NODE_TARGETS`, `tag_suffix_for_target()`, `image_tag_for_target()` (bug.0328). Sourced by `detect-affected.sh`, `build-and-push-images.sh`, `resolve-pr-build-images.sh`, `flight-preview.yml` image-resolve step (resolves `sha-<mainSha>` — no re-tag), `promote-and-deploy.yml` resolve+promote steps. Adding a new node is a one-file edit here — don't hand-maintain target lists in workflows. |
| `resolve-pr-build-images.sh` | Resolve pushed PR image digests from GHCR for the `pr-{N}-{sha}` tag convention (used by candidate-flight, not the PR workflow). Sources `lib/image-tags.sh` for the target catalog; includes per-node migrator tags and the top-level `source_sha` envelope field consumed by `promote-build-payload.sh`. |
| `promote-build-payload.sh` | Apply resolved digests to a deploy-branch overlay via `promote-k8s-image.sh`. Emits `promoted_apps=<csv>` to `$GITHUB_OUTPUT` **incrementally after each successful promotion** plus a `trap EXIT` belt-and-suspenders — the output survives mid-script abort (bug.0328). Records per-app `source_sha` into `.promote-state/source-sha-by-app.json` in a second pass so map failures cannot shadow `promoted_apps`. Partial map failures surface as `::warning::`; total map failure hard-fails the script so provenance decay is loud (bug.0321 Fix 1+4). |
| `promote-k8s-image.sh` | Lower-level overlay digest writer. Used directly by `promote-and-deploy.yml` and indirectly by `promote-build-payload.sh`. Treat as a primitive; never bypass it to edit overlay YAML by hand. |
| `update-source-sha-map.sh` | Shared primitive that merges a single `app → source_sha` entry into `.promote-state/source-sha-by-app.json` on the deploy branch (bug.0321 Fix 4). Called per promoted app by `promote-build-payload.sh` and by the `promote-and-deploy.yml` promote-k8s loop. |
Candidate / preview / production flight:
| Script | Purpose |
| --------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `aggregate-rollup.sh <env>` | task.0376 — compute `current-sha = git merge-base $(deploy/<env>-<node> tips)` + merge per-node `source-sha-by-app.json` into the rollup preserving unaffected entries. Called by `promote-and-deploy.yml`'s `aggregate-{preview,production}` job; rebase-retries on push contention. `CURRENT_SHA_IS_MERGE_BASE` + `ROLLUP_MAP_PRESERVES_UNAFFECTED`. |
| `report-candidate-status.sh` | Post commit status on the PR head for candidate-flight progress. (Pre-task.0376 lease primitives `acquire-candidate-slot.sh` / `release-candidate-slot.sh` retired; per-node `concurrency: flight-<env>-<node>` is the lease — `BRANCH_HEAD_IS_LEASE`.) |
| `wait-for-candidate-ready.sh` | Endpoint readiness gate (HTTPS `/readyz`) — run inside the workflow. **Refuses to run** unless `ARGOCD_SYNC_VERIFIED=true` is set in the environment (runtime-enforced gate ordering, bug.0321 Fix 4). Set `ARGOCD_SYNC_VERIFIED=skip` for local/test callers that do not have Argo in the loop. |
| `wait-for-argocd.sh` | SSH into the VM and block until every app in `PROMOTED_APPS` (CSV, or the full catalog when unset) reports `sync.revision == EXPECTED_SHA && Healthy` **AND its new ReplicaSet has reached desired count** per app (`rollout_check`: `updatedReplicas >= spec.replicas` AND `availableReplicas >= spec.replicas`; bug.0326, [ci-cd.md Axiom 15](../../../docs/spec/ci-cd.md)). `EXPECTED_SHA` MUST be the deploy-branch tip SHA (not the source-app commit). The new-RS gate is required because Argo-level Healthy fires while the old ReplicaSet is still Ready during rolling updates — `/readyz` from those pods serves the prior BUILD_SHA. It deliberately does **not** run `kubectl rollout status` / wait for the old RS to drain (slow terminations false-fail healthy deploys). On success, writes `ARGOCD_SYNC_VERIFIED=true` to `$GITHUB_ENV` so downstream steps see the marker. |
| `smoke-candidate.sh` | Candidate HTTPS smoke (operator, poly, resy — `/readyz`, `/livez`) |
| `wait-for-in-cluster-services.sh` | `kubectl rollout status` on every k8s Deployment (operator/poly/resy/scheduler-worker) via SSH. Authoritative per-app rollout gate — catches "overlay unchanged, Argo no-op, pods stale". |
| `verify-buildsha.sh` | End-of-deploy contract probe: curls **`/version`** on node-apps, asserts **`.buildSha`** matches the expected SHA (task.0345 / #978). Map mode: `SOURCE_SHA_MAP` → `.promote-state/source-sha-by-app.json`. When **`NODES`** is set (workflows pass `promoted_apps`), verifies only that subset vs the map (task.0349 / affected-only). Single-SHA fallback when the map file is missing. Scheduler-worker and migrator filtered out (no Ingress). Runs in `verify-deploy` / `verify-candidate`. |
| `deploy-infra.sh` | SSH + rsync `infra/compose/**` from a git ref, then `compose up`. `--ref <git-ref>` (default `main`) |
| `assert-target-substrate.sh` | Read-only preflight for app flights. Reads `infra/catalog/$TARGET.yaml`, dispatches by `.type`, and currently implements the `type=node` substrate assertion (VM/k3s/Argo reachable, local catalog/overlay/AppSet present, scoped DNS, live edge/Caddy route, Service NodePort, Deployment-consumed Secret/ExternalSecret, node DB). `type=service` and `type=infra` fail explicitly until their own contracts exist. |
| `flight-preview.sh` | Always dispatch the preview flight (`promote-and-deploy env=preview`) for the merged SHA — latest-wins, no lease |
| `create-release.sh` | Cut `release/YYYYMMDD-<sha>` from preview `current-sha`, open release PR into `main` |
| `verify-deployment.sh` | Post-deploy verification |
Support:
| Script | Purpose |
| ----------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `validate-dsns.sh` | Validate DATABASE_URL and DATABASE_SERVICE_URL |
| `ensure-temporal-namespace.sh` | Create Temporal namespace if missing |
| `compute_migrator_fingerprint.sh` | Content-hash for migrator cache |
| `check-gitops-manifests.sh` | Validate k8s manifest correctness |
| `check-gitops-service-coverage.sh` | Ensure all services have k8s manifests |
| `test-image.sh` | Container livez smoke (used ad-hoc / by legacy paths) |
| `build.sh`, `build-service.sh`, `push.sh` | Legacy build scripts retained as callables by `build-multi-node.yml` fallback; new work uses `build-and-push-images.sh` |
| `deploy.sh` | Legacy full-stack SSH deploy; **retired** — use `deploy-infra.sh` + Argo CD reconciliation instead |
### Provision (`scripts/setup/`, `infra/provision/`)
| Resource | Purpose |
| ------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `.github/workflows/provision-env.yml` | **Canonical new-env entry** — runner-owned bootstrap.sh → provision-env-vm.sh (VM + DNS + OpenBao/ESO + deploy). Guide: [`docs/guides/create-env.md`](../../../docs/guides/create-env.md) (incl. known e2e gaps) |
| _(no script — by design)_ | **Node-app compute needs NO VM provisioning** — an Akash-lane node app is created via the operator compute API against the existing env's state substrate (see Deployment targets section). `provision-env.yml` provisions the STATE substrate, not app compute. |
| `provision-env-vm.sh` | Current env provisioner (catalog-driven; candidate-\*/preview/production) |
| `provision-test-vm.sh` | Legacy one-command VM setup (candidate-a only) |
| `infra/provision/cherry/` | OpenTofu modules for Cherry Servers VMs |
| `infra/provision/cherry/k3s/` | k3s bootstrap (cloud-init, Argo CD, ksops) |
### VM SSH access — policy first, path second
**SSH is not a general tool in this repo. Reproducibility via infrastructure-as-code is the lifeblood — every ad-hoc SSH action is a latent drift source.** The rules below are hard, not aspirational:
1. **SSH is read-only by default.** Inspect pods, tail logs, `kubectl get`, `curl` from inside the network. Never `kubectl apply`, never `kubectl edit`, never `docker compose up` by hand, never edit files on the VM. The only sanctioned write path is a commit to git (code branch or `deploy/*`) that Argo / the pipeline reconciles.
2. **The only environment where write-mode SSH is permitted is `candidate-a`.** Even there it's a last resort for a 5-minute experiment that must be captured as a script, OpenTofu change, or GitOps commit **before the session ends**, per the repo-wide "if it isn't in git, it didn't happen" rule. A write-mode SSH to candidate-a that leaves no diff behind is a bug.
3. **Never SSH to production.** Not for reads, not for writes, not for diagnostics. If you need production signal, query Grafana/Loki via the MCP, check `/readyz` / `/livez` from outside the network, or read `deploy/production` state from git. If those don't answer the question, the question is asking you to violate the IaC contract — escalate, don't shell in.
4. **Preview reads are discouraged.** Prefer Loki / Grafana / `/readyz` from outside. A preview SSH is acceptable only when an external signal is provably missing and the read genuinely can't wait for instrumentation to land.
**Where SSH coordinates live (when you're allowed to use them)**
The sanctioned path — and the only one an agent should rely on — is what [`docs/guides/multi-node-deploy.md`](../../../docs/guides/multi-node-deploy.md) documents: `scripts/setup/provision-test-vm.sh` writes the SSH private key + VM IP to `.local/` (gitignored) at provision time.
| File | Purpose |
| ----------------------------- | ------------------------------------------------------------- |
| `.local/{env}-vm-key` | SSH private key (ed25519) for `root@<VM_IP>` |
| `.local/{env}-vm-ip` | VM IP address (single line) |
| `.local/{env}-vm-secrets.env` | Provision-time secrets (Argo bootstrap, sealed-secrets, etc.) |
| `.local/{env}-vm-age-key` | age key for sealed-secrets / sops on the VM |
`{env}` matches `DEPLOY_ENV` in `provision-test-vm.sh` — currently `candidate-a`, `preview`, `production`, `test`. The provisioner public key is committed at `infra/provision/cherry/base/keys/cogni_{env}_deploy.pub`.
Usage (candidate-a only; read-only): `ssh -i .local/candidate-a-vm-key root@$(cat .local/candidate-a-vm-ip) "kubectl -n argocd get applications"`.
**`.local/` lives on the provisioner device, not in worktrees.** It's gitignored, so a freshly cloned worktree (anything under a conductor / agent workspace path) will not have it. The canonical copy lives in the dev's primary cogni-template clone — reach for keys there, not in the active worktree. If you don't know your own primary-clone path, ask the human running the device; never hard-code home-directory paths into committed files (this skill is in a public repo).
**Legacy `canary` naming.** Some provisioner machines still have old
`.local/canary-vm-*` files, but current Cogni monorepo candidate-a should prefer
`.local/candidate-a-vm-key` and `.local/candidate-a-vm-ip` when present. Fall
back to `canary-vm-*` only if the current files are absent and you have verified
the IP matches `cogni-candidate-a.vm.cognidao.org`.
If `.local/{env}-vm-key` isn't present for the env you need, this device isn't the provisioner for that env. Do not improvise — don't source shell aliases from personal dotfiles, don't paste IPs from Slack, don't copy keys from a teammate. Either re-run `provision-test-vm.sh` (candidate-a only) or drive the change through the pipeline.
## Enforcement rules
When reviewing code that touches CI/CD, deploy, or infra:
- **Reuse existing scripts.** If `promote-k8s-image.sh` already updates overlays, don't write a new one. If `deploy-infra.sh` already handles Compose infra, extend it — don't create a parallel path.
- **No manual VM state.** Every change must be in a script or manifest. Ask: "if we destroy and reprovision this VM, does this change survive?"
- **Re-point ALL deploy pointers on VM migration (deploy-pointer drift).** When an env's VM is rebuilt/migrated to a new box, **three** pointers must be updated together, or CI→VM access silently dies: (1) the Cloudflare `<env>.vm.cognidao.org` A record, (2) the GH env secret `VM_HOST`, (3) the GH env secret `SSH_DEPLOY_KEY`. The app path is independent (Cloudflare→cluster keeps serving), so a stale pointer is INVISIBLE until a deploy/reconcile/promote runs — then it fails as a cryptic `exit 255 / Connection timed out / Permission denied`. **Incident (2026-06-10):** the Jun-3 prod migration `.202→.152` left all three stale → every prod reconcile/promote failed silently for days; cognidao.org stayed green the whole time. Fix is loud-by-default now: `reconcile-env-substrate.sh` SSH-preflights and emits a clear diagnosis. Verify `VM_HOST` resolves to the live Cherry VM (project 254586) and `SSH_DEPLOY_KEY` authenticates before assuming a deploy bug. promote-and-deploy + deploy-infra should grow the same preflight (follow-up).
- **No new mutable tags.** All image references use digests or `{env}-{sha}` tags.
- **No new long-lived code branches.** Feature branches are short-lived. Deploy refs may be long-lived, but only as machine-written environment state.
- **No PRs for deploy state.** Deploy branches get direct commits.
## Anti-patterns to flag
- **Assuming a node app ships via overlay/AppSet.** Check the node's deploy lane first: Akash-lane nodes (`deploy_provider: akash`) have NO deploy branch, overlay, AppSet, or Argo Application — their deploy surface is the operator compute API (Deployment-targets section). Advising an overlay bump for one is a category error.
- **App-adjacent sidecars via k8s pod injection.** A node's second image is an extra service in the same Akash lease (`SIDECAR_IS_SDL_SERVICE`, ci-cd.md Axiom 25) — the overlay/kustomize injection lane was closed with PR #1884; don't resurrect it.
- Rebuilding in production instead of promoting proven digest
- **Growing the frozen deploy brain** (see Freeze Policy, top). A PR that adds a new responsibility / secret / service / DB path / inline decision tree to `deploy-infra.sh`, a net-new infra-mutating `scripts/ci/*.sh`, or a new deploy/promote/provision workflow — when the behavior could be a catalog field, overlay/AppSet, ESO declaration, or a `DeployCapability` method. Flag it; route it per the request→home table. (Bug fixes + catalog-driven + guard-tightening are fine in place.)
- **NODE_RESOLUTION_IS_A_DB_READ (north-star): the operator resolves nodes (list, id→slug, id→URL) from its OWN DB registry — never a static `COGNI_NODE_ENDPOINTS` map, and never by splicing a configmap at node-formation.** The operator app has `resolveNodeRegistry`/`listRoutableNodes` (see [cicd-platform-boundary.md](../../../docs/spec/cicd-platform-boundary.md) "resolve the node via the shared registry"); the ADDRESS comes from `NodeAddressPort` (`node-address.adapter.ts`), which resolves the node's declared `deployment_provider.<env>` — `http://<slug>-node-app:3000` for a k3s node, the node's public canonical host for one placed off-cluster (`PLACEMENT_DECIDES_THE_ADDRESS`, bug.5106). Never hardcode `internalNodeAppUrl` in a new operator→node client: that assumption silently dropped every attribution receipt for an Akash-placed node. The **static `COGNI_NODE_ENDPOINTS` configmap exists only for the DB-less `scheduler-worker`** (no DB → must consume a catalog-rendered map). Flag any operator-side feature that: reads `COGNI_NODE_ENDPOINTS` in the app, adds nodes to a static endpoint map, or edits `github-repo-write.ts` / a base configmap to keep such a map in sync — route it to a registry read instead. Reference: the receipt-delivery gateway (#1924) shipped copying the worker's static-map pattern into the operator app → empty node epochs until it was moved to the registry (story.5001).
- `:latest` or mutable tags in deploy manifests
- One-off SSH edits not captured in scripts
- PRs for machine-written deploy-branch updates
- Auto-release PR on every E2E success
- Ungated promotion (CI red but code advances)
- `cancel-in-progress: true` on deploy workflows
- Unbounded health-check polling without timeout
- Duplicating logic that already exists in `scripts/ci/`
- Running `deploy-infra.sh` from `candidate-flight.yml` or any app digest
flight. App flight can assert substrate; infra levers provision substrate.
- Naming a generic deployable gate after one catalog shape. `node` checks are a
branch under `assert-target-substrate.sh`, not the generic flight abstraction.
- **Silent-success on no-op runs.** A workflow that exits 0 without doing the work paints a green check that operators misread as "deployed". Every no-op path must produce a visually-distinct outcome: exit-code convention (`1=error, 2=no-op`) + a gated follow-up job that surfaces as skipped, OR a loud `::warning::` + step summary. Never silently succeed after a skip.
- **Status / summary logic in YAML.** Decision logic (exit-code → status name, summary markdown, banner copy) belongs in `scripts/ci/*.sh`. Workflows call the script and gate jobs on its `$GITHUB_OUTPUT`. Inline `case $rc in ...` + `echo >> $GITHUB_STEP_SUMMARY` in YAML is a smell — move it into the script.
- **`/readyz 200` treated as rollout proof.** `/readyz` is Service-level: answered by any running pod, old or new. Real rollout proof is **`/version.buildSha` matches the expected SHA** (`verify-buildsha.sh`; task.0345) — the ONLY provider-agnostic proof, and the only one that exists on the Akash lane. On the k3s lane, `kubectl rollout status` (`wait-for-in-cluster-services.sh`) or the new-RS replica-count assertion in `wait-for-argocd.sh` (`rollout_check`) are acceptable k8s-specific variants (never `/readyz`).
- **Argo `health.status == Healthy` treated as rollout proof** (bug.0326). Argo's Healthy fires as soon as enough pods are Ready — including old-RS pods during a rolling update. `/readyz` from those pods returns the prior BUILD_SHA, so verify-buildsha fails on a green-upstream flight. `wait-for-argocd.sh` now asserts the new ReplicaSet has reached desired count (`rollout_check`: `updatedReplicas >= spec.replicas` AND `availableReplicas >= spec.replicas`) per promoted app after the Application-level Healthy check ([ci-cd.md Axiom 15](../../../docs/spec/ci-cd.md)); don't regress that to Application-health-only.
- **Skipped verification treated as success** (bug.0328; per-node since task.0376). A terminal consumer of a skipped `verify-candidate`/`verify-deploy` must distinguish real no-op from contradiction: candidate-flight's `report-status` keys on `has_flight_targets` (skip is green only when there were no flight targets); preview/prod's `aggregate-{preview,production}` assert every promoted cell has a `verified-<node>.txt` marker (`aggregate-decide-outcome.sh`, [ci-cd.md Axioms 14 & 19](../../../docs/spec/ci-cd.md)). Skipped + had-work = hard-fail.
- **Hand-maintained target lists in workflows** (bug.0328). Every `ALL_TARGETS=(...)` or `promote_app operator ... / poly ... / resy ...` array inlined in a workflow `run:` block is drift waiting to happen — the next new node gets half-shipped when someone updates the lib but forgets the workflow. Source `scripts/ci/lib/image-tags.sh`, iterate `ALL_TARGETS` / `NODE_TARGETS`.
- **Step-level `if:` for verification gates.** GitHub treats a skipped _step_ inside a running job as contributing to job success. When a verification should be allowed to skip (e.g. empty `promoted_apps`), model it as a _job-level_ gate with `needs:` and `if:` — the job then surfaces as visibly skipped (grey in the checks list), not green. Step-level `if: promoted_apps != ''` is the silent-green primitive that bug.0321 hunted down.
- **Gate-ordering by convention, not by runtime.** "X must run before Y" embedded in a comment rots at the next refactor. Enforce it structurally: the upstream script writes a marker to `$GITHUB_ENV` (e.g. `ARGOCD_SYNC_VERIFIED=true`); the downstream script refuses to run without it. `wait-for-argocd.sh` → `wait-for-candidate-ready.sh` is the reference pattern (bug.0321 Fix 4).
- **Cross-PR contract verification via overlay inspection alone.** Production promotions copy preview's overlays, which can mix digests from different PR head SHAs (affected-only CI). A single `EXPECTED_BUILDSHA` is meaningless there. The canonical pattern: write a per-app `source_sha` map to `.promote-state/source-sha-by-app.json` at promotion time, copy it forward preview → production, read it in the verifier. Artifact provenance travels with the artifact.
- **Deploy workflows triggering app-level business logic.** `candidate-flight-infra.yml` / `deploy-infra.sh` / `promote-and-deploy.yml` exist to reconcile infrastructure state — Compose stacks, k8s overlays, image digests. They MUST NOT `curl` an app endpoint to kick off business work (data sync, cache warm, third-party mirror, index rebuild). When the deploy script becomes a cron, every property of the deploy pipeline (lease semantics, retry budget, timeout, secret blast radius, "you can't dispatch infra without WORK_ITEMS_NOTION_TOKEN") is now coupled to whatever app feature added the curl. Apps schedule themselves — internal cron, scheduler-worker job, on-startup hook, or a dedicated workflow with its own trigger. Reference incident: PR #1177 added a Notion mirror sync as a post-deploy `curl` step in `candidate-flight-infra.yml`; reverted in full because "env-gated, no impact" is a lie when the gate is in the deploy pipeline. **devops-expert MUST be a required code reviewer on any PR touching `.github/workflows/*`, `scripts/ci/**`, `infra/**`, or `deploy/\*` — including PRs whose primary purpose is an app feature that just "happens to" add an env var or a curl step to a deploy file.**
Is this your skill, or is something wrong with this listing? Request removal or report an issue. Author removals are honored within 72 hours.
No comments yet. Be the first to comment!
