Coordinator Rolling Out To Nodes

---
name: coordinator-rolling-out-to-nodes
description: "Use when rolling a build across all nodes — fleet deploy, rolling the nodes, deploy everywhere, ship to every node, canary then the rest, promote the canary, staged rollout, deploying after a merge batch. Orchestrates the staged order and the promotion gate between stages, and delegates each node to the coordinator-deploying-a-node runbook. Load this BEFORE dispatching any node."
---

# Coordinator: Rolling Out a Build Across All Nodes — staged order, canary, promotion gate

> This skill is the ORCHESTRATION layer. It does not replace
> `coordinator-deploying-a-node` — it calls it, once per node.

## 0. Universal invariants (read first — these outrank everything below)

These hold for every deployment regardless of project or topology. The adapter supplies *how*, these define *what must be true*.

1. **Build once; distribute identical bytes.** One pinned immutable artifact (SHA/version/checksum triple) is built once at one SHA on the build host; every node installs from that same artifact. No node re-derives it. Pin exact SHA, never tip. Adapter: `fleet.deployment.thrum.artifact` or `fleet.deployment.products.<id>.artifact`.
2. **Target-local mutation.** The node-local coordinator performs the mutation (install/migrate/restart) locally on its own host. Never drive a node's build/install/restart from another host. The orchestrator dispatches and gates; the node executes.
3. **Immutable identifier + checksum.** Artifact is identified by SHA + version + checksum; verify BEFORE install that the delivered bundle is the one the orchestrator built. Adapter: `artifact.checksum`, `artifact.version_file`, `artifact.commit_file`.
4. **Canary evidence does not transfer.** A canary pass proves the *build* is sound; it does not prove any other node's *state* is sound. Every node takes its own BEFORE readings.
5. **Verify by observed effect, never exit code.** Install/restart success is proven by post-mutation observation (version equals pin, NEW PID, service status reports pin, migration moved), not by installer exit 0. Adapter: `install.verify_by_effect`, `health.probes`.
6. **Capture before/after.** For every node, capture BEFORE state (current version/state, health counters) before mutation and AFTER state after; compare.
7. **Account for every target.** Roster completeness: enumerate topology from config, disposition each node (rolled on pin / deferred with owner sign-off + notify local coordinator), and never normalize unmeasured rows.

Additional parameterized invariants (skill-owned, adapter-valued):

- Pull-current first (ff-only) before reasoning about state. Adapter: `preflight.pull_command`.
- Backup before touching anything; sizing via adapter formula. Adapter: `backup.command`, `backup.sizing_formula`.
- Never bypass the storage engine's file ownership. Adapter: `storage_access.via`, `storage_access.forbidden`.
- Install-before-restart ordering; re-render between install and restart. Adapter: `templates.order`.
- Convergence wait from mechanism, not impatience. Adapter: `health.convergence_wait`.

## 1. Enumerate topology (from adapter — never hard-code)

Read the project's deployment adapter inside `.thrum/config.json` (`fleet.deployment`; human-readable reference at the deployment-adapter reference). Never hard-code topology or commands in the dispatch or in this skill — read them from that single source.

- For Thrum fleet plus connected daemons: `fleet.deployment.thrum.topology` (`canary`, `waves`, `excluded`, `connected_daemons`, `identity_resolution`).
- For a customer product `<id>`: `fleet.deployment.products.<id>.topology` (same shape, per-product `canary`/`nodes`/`waves`).

Resolve each node's `daemon_id` / `ssh_alias` / `coordinator` via the adapter's `identity_resolution` method (daemon_id join, not name). If topology is empty, STOP — do not invent nodes.

**Why every node carries an `ssh_alias`:** so the rolling coordinator can reach the node directly when the messaging or pane-mirror path fails. The main case is an agent stuck on a permission prompt or dialog. Read the pane with `ssh <alias> tmux capture-pane -p -t <session>`, then answer a prompt you recognise as safe with `tmux send-keys`. Never approve blindly, and never pick a "don't ask again" option. Re-capture afterwards to confirm the pane moved on. A node flagged `blocked-on-human` gets unblocked this way in the same turn; don't wait on it or reorder the roll around it. The alias is for pane interaction and artifact distribution only. The node still builds, installs and restarts itself, through its own coordinator.

> Example — Thrum project: topology canary and waves are defined in `topology.canary` / `topology.waves` (seven boxes, one excluded), connected daemons per `topology.connected_daemons`. Build host and build command are adapter values, not skill literals.

## 2. Pin the artifact (build once at SHA/version/checksum)

Derive the immutable pin on the build host (adapter `artifact.build_host`): current `HEAD` SHA, version, and checksum. When multiple related artifacts (e.g., core plus connected-repo bundles) are involved, pin each immutably and distribute the same set to every relevant node. Announce the pin in the dispatch so every node can verify it.

Artifact identity lives in `artifact.output`, `artifact.version_file`, `artifact.commit_file`, `artifact.checksum`. Never use a floating tag like tip; always the pinned SHA.

## 3. Preflight (control-plane current, forward-only, backup sizing via adapter)

- Verify control-plane checkout is current: `preflight.pull_command` (must be ff-only; if not fast-forwardable, STOP and surface).
- Forward-only check per node: `preflight.forward_only_check` — prove the pin is an ancestor of the node's current SHA (plus a control that must FAIL, e.g., an invented SHA, to prove the check discriminates).
- Backup sizing via adapter: `backup.sizing_formula` — compute `free_at_completion` per the formula before running `backup.command`; if the projected peak exceeds the safe threshold, STOP and escalate.

## 4. Build once → distribute identical artifact (mechanism via adapter)

Build once per artifact on its `artifact.build_host` via `artifact.build_command` → verify the produced bundle's version/checksum/commit sidecar → distribute the identical bytes to every node's inbound drop path (`artifact.distribution.drop_path`) via the adapter's `artifact.distribution.mechanisms` (e.g., drop-folder or copy). No node re-derives or rebuilds. Verify BEFORE unpack that the delivered bundle matches the pin.

> Example — Thrum project: build once via the adapter's `artifact.build_command` on the adapter's `artifact.build_host`, distribute via the adapter's `artifact.distribution.mechanisms` to `artifact.distribution.drop_path`, bundle carries sidecars verified per adapter.

## 5. The staged order — canary, waves, prepare-and-hold, promotion gate

> 🔴 **If any node fails during the rollout — canary or any wave — STOP THE ROLLOUT.** Do not roll, restart, or recommend
> restarting any further node (including the build host) until the failure is root-caused and fixed or the
> owner explicitly decides to continue. A node that went down and was recovered by hand is a FAILURE, not a green.

- **Canary first, alone.** One node (adapter `topology.canary`) rolls first. Its health must be observed green before any wave proceeds.
- **Waves with coordinated dependency/order/rollback.** Subsequent waves follow `topology.waves`. When related daemons share ordering (e.g., connected-repo daemons that must restart together), respect the adapter-defined dependency and restart order; a failure in one dependent daemon triggers coordinated rollback per adapter `rollback` / `backup.restore_command`.
- **Prepare-and-hold + release BY NAME.** If the mechanism supports it, prepare all nodes in a wave (acquire + verify artifact, take BEFORE snapshot, but do not restart) then release by explicit node name, not by broadcast.
- **Promotion gate.** After each stage, verify by observed effect on that node (see node skill) before promoting. On no-go, halt the rollout and roll back the affected node(s) per adapter.

## 6. Dispatching a node — what the message must carry

Dispatch each node to its own local coordinator (resolved via `identity_resolution`), never via remote shell for the mutation itself. The dispatch message must carry:

- Pinned artifact identifier (SHA/version/checksum) and drop-path location.
- Which adapter section to use (`fleet.deployment.thrum` vs `fleet.deployment.products.<id>`).
- Expected pin for verification and the BEFORE-snapshot request.
- Hold/release semantics if using prepare-and-hold.

The node's local runbook is `coordinator-deploying-a-node` — reference that skill by name in the dispatch.

## 7. After the last node — roster completeness, phase reconciliation, test gate

- Enumerate the topology again from the same adapter source and disposition every entry (rolled on pin / deferred with owner sign-off + notify local coordinator). Missing or unmeasured rows are not normalized.
- Reconcile registry/phase: verify every daemon reports the pin (adapter `health.probes` + `install.verify_by_effect`), and that agent phases have converged.
- Run the project's gate that is appropriate for the change (e.g., `make gate` or equivalent) before declaring success.

## 8. Holds and red flags

- If `fleet.deployment` is absent or `fleet.deployment.products` and `fleet.deployment.thrum` are both empty, STOP — no topology is defined.
- If the control-plane pull is not fast-forwardable, STOP.
- If `fleet.deployment.thrum.products` exists (vestigial), STOP — products live only at `fleet.deployment.products`.
- If a node reports success by exit code alone without observed-effect verification, treat as not-verified.
- Never wrap a service restart in a blind timeout on the node (see node skill `service.never_timeout` and `migration.signal_rules`).
- 🔴 A **macOS** node restarted with the plain `service.restart_command` instead of `service.mac_restart_command` (Example — Thrum adapter: `service.mac_restart_command`). The daemon loses its Local Network permission and its LAN peers go dark while it looks healthy. Put the mac command in every macOS node's dispatch, and use it yourself when you roll the build host last.