Coordinator Deploying A Node

---
name: coordinator-deploying-a-node
description:
  "Use when deploying a build to a single node, migrating or not — deploying a
  build, schema-migrating deploy, pro-bundle install plus service restart on a
  node, roll out a build to a node, ship to a node, single-node canary. Loads
  the deploy runbook — verifying the artifact actually CONTAINS the fix,
  install-before-restart ordering, verify-by-effect, and preventing store
  corruption on migration."
# source: claude-plugin/skills/coordinator-deploying-a-node/SKILL.md
---

## Coordinator: Deploying a Build to a Single Node — migrating or not

> This runbook covers EVERY deploy to a single node, migrating or not. Most
> sections apply to every deploy; only migration-weight and store-corruption
> specifics are bump-only.

### 0. Pull-current first (ff-only)

The absolute first action before anything else: fetch the control-plane
repository and fast-forward to the orchestrator's pin.

- Read the project's deployment adapter inside `.thrum/config.json`
  (`fleet.deployment`; reference at the deployment-adapter reference) to obtain
  `preflight.pull_command` and `preflight.pin_command`.
- Run `preflight.pull_command` (must be ff-only; if not fast-forwardable, STOP
  and surface — never blind-merge in the shared tree).
- Verify the delivered artifact actually CONTAINS the fix at the pinned SHA
  (adapter `artifact.commit_file` sidecar).

This ensures the node's coordinator reasons from a checkout that is actually
current before evaluating backup, install, or migration.

> Example — Thrum project: pull via
> `git fetch origin && git merge --ff-only origin/<branch>` before any backup or
> install.

### 1. Who executes — resolve the node's local coordinator (via adapter identity)

The node's own coordinator drives the deploy locally on its own host. Never
drive a node's build/install/restart from another host.

- Resolve WHO via `topology.identity_resolution` (adapter
  `fleet.deployment.thrum.topology.identity_resolution` or
  `fleet.deployment.products.<id>.topology`) — a `daemon_id` join plus hostname
  and absence of `.thrum/redirect`, never by agent name or worktree name.
- When connected-repo daemons share a host, respect the adapter-defined restart
  order within that host.

If the target node has no working coordinator, state why explicitly before using
any fallback.

### 2. Back up before touching anything (via adapter backup command + sizing formula)

- Run the project's backup command (`fleet.deployment.thrum.backup.command` or
  `fleet.deployment.products.<id>.backup.command`) and verify per
  `backup.verify`.
- Size BEFORE you run it on a bloated node: apply the adapter's
  `backup.sizing_formula`
  (`free_at_completion = free_now - zip(stale current/) + size(stale current/) - size(live lane)`)
  — the stale `current/` is removed before the fresh export, so peak is not
  double-count. Retention runs last and does not bound the peak.
- On failure, roll back via `backup.restore_command`; never delete the store.

> Example — Thrum project: backup via `thrum backup`, sized per the formula
> above, verified with `sqlite3 ?mode=ro&immutable=1` read.

### 3. Storage access — through the engine or control-plane CLI, never file-bypass

Access the state store only through the engine or the control-plane CLI in
read-only mode. Never bypass the storage engine's file ownership.

- Via: `storage_access.via` (e.g., CLI or `?mode=ro`).
- Forbidden: `storage_access.forbidden` (e.g., hand-copy of live `-wal`/`-shm`,
  concurrent `.backup` on live store).
- A bare read-write open can fire a checkpoint that truncates the write log out
  from under live readers → sustained corruption. Always use `?mode=ro` where
  applicable.

If the project has no WAL/SHM, this section is N/A — skip and note.

### 4. Never wrap service start/restart in a blind timeout

🔴 **On macOS, use `service.mac_restart_command`, not
`service.restart_command`.** (Example — Thrum adapter:
`service.mac_restart_command` preserves Local Network (TCC) permission.) A
daemon restarted from a terminal or agent pane gets its Local Network (TCC)
permission attributed to that terminal and is silently denied LAN peer dials
(`no route to host`): peers go dark while the node itself looks healthy, and
`PPID 1` does NOT prove it was re-parented correctly. Check the node's OS before
choosing the command. After a mac restart, verify the peers on the LAN
reconnect, not just the version.

Run the project's service restart command
(`fleet.deployment.thrum.service.restart_command`, or
`service.mac_restart_command` on macOS, or product equivalent) without a blind
timeout wrapper. A timeout that kills a migrating restart mid-migration
manufactures a half-migrated store.

- Adapter flag `service.never_timeout` is true.
- Signal-window rules are adapter data: `migration.signal_rules` (`SIGQUIT`
  fatal-always, `SIGUSR1` fatal-before-socket). Respect them — do not send fatal
  signals in the forbidden window.

> Example — Thrum adapter: use the configured `service.mac_restart_command` on
> macOS; use `service.restart_command` otherwise. Never wrap either in
> `timeout`.

### 4a. EULA gate — before the binary swap

EULA gate (0.11+): before swapping the binary, record acceptance for the box
user by running the new build once with `THRUM_ACCEPT_EULA=<bundled version>`.
Confirm that `~/.thrum/license/eula-acceptance-<version>-*.json` exists and that
a plain `thrum inbox` works. Skipping this breaks every agent's CLI on the box.

### 5. Install from the delivered artifact (never rebuild on the node)

Install from the identical bundle the orchestrator built, delivered to
`artifact.distribution.drop_path`.

- Verify BEFORE unpack that the delivered bundle's version/checksum/commit
  sidecar equals the orchestrator's pin (adapter `artifact.checksum`,
  `artifact.version_file`, `artifact.commit_file`).
- Run `install.command` (from adapter). The node does not compile, sign, or
  rebuild — it installs.
- Collect BEFORE readings (current version, state counters, checkpoint counts,
  health probes) before the install.

> Example — Thrum project: install from the adapter's
> `artifact.distribution.drop_path` via the adapter's `install.command`.

### 6. Measure from the running process, never the checkout

Read version/state from the running service, not from the filesystem checkout.

- Verify via `install.verify_by_effect` probes: version equals pin, NEW PID,
  service status reports pin, sidecar equals pin.
- Do not trust installer exit 0 alone.

### 7. Migrate via service restart — verify by effect

Migration is performed by restarting the service after install; the service's
own boot migration does the schema/data move.

- Calibrate weight via `migration.weight_calibration` (heavy bulk-export vs
  light add-column) — do not assume a fixed duration.
- Known signatures that require a cold-restart recovery:
  `migration.sigbus_signatures` (e.g., WAL-index recover signatures) and
  `migration.on_sigbus`.
- Observe that the restarted service reports the new version/state and that
  health probes have converged (adapter `health.probes`,
  `health.convergence_wait`).

### 8. Verify post-migration (liveness, health probes, checkpoint counts, agent liveness)

Run adapter `health.probes` and compare BEFORE/AFTER:

- Service alive and reporting pin.
- State counters converged per `health.convergence_wait` (mechanism-derived,
  e.g., `3m`).
- Crash log empty, expected agents live.

For tandem daemons that share a host or depend on each other, verify the
coordinated set per adapter.

### 8a. Re-render configuration/templates if the build changes them

If the build carries template or role changes, re-render between install and
restart:

- Commands: `templates.refresh_command` and `templates.deploy_command`.
- Order: `templates.order` (`refresh→deploy` between `install.sh` and daemon
  restart).
- Verify by effect (render output, then service restart).

> Example — Thrum project: `thrum roles refresh` then `thrum roles deploy`
> between install and restart.

### 8b. Plugin/skill refresh if the deploy carries skill changes

If the bundle carries skill/plugin changes, refresh the plugin cache:

- Refresh driver: `plugin.refresh` (version bump required — cache keyed by
  version).
- Manifests: `plugin.manifests` (all skill-carrying manifests bumped in the same
  commit as the new skills).

Do not hand-edit generated marketplace manifests.

### 8c. Reconcile registry/phase after restart

Re-verify the daemon registry and agent phases after restart (adapter
`health.probes` includes phase reconciliation). Confirm the pinned version is
durable across a clean cold-boot probe where applicable.

### 8d. Verify the GUI shows every agent (by effect)

A build can decode `agent.list` fine on the daemon and still hide agents in the
GUI (a strict UI decode once hid 485 of 538 on a canary). Compare what the
daemon lists with what the GUI renders, on the node:

1. Expected numbers, run ON the node (python3 only, read-only, calls the
   daemon's own `agent.list` over its socket):
   `python3 <skill>/resources/agent-count-expected.py` prints `rail_expected`,
   `fleet_expected` and `records_lacking_proof`. Do not use `thrum team --all`
   as the baseline: it is `team.list`, a different set (stale rows included).
2. Rendered numbers. Needs Playwright plus a browser. The script tries
   Playwright's bundled Chromium, then the newest installed `chromium-<rev>`
   under the Playwright browsers dir (`PLAYWRIGHT_BROWSERS_PATH`, else the
   per-user `ms-playwright` cache; a Playwright upgrade can leave only other
   revisions), then an installed Google Chrome. It prints the route used
   (`browser=...`) on stderr and never installs a browser, so a node near its
   disk guard is not touched.
   `node <skill>/resources/agent-count-gui.mjs <ui-url>` (the `ui` line of
   `thrum daemon status`; the script header says where Playwright must resolve)
   prints `rail` and `fleet`; it exits 3 with a message when no browser is
   usable. Without a browser, open `<ui-url>` at 1280x720 and read by hand:
   - rail: the heading `Agents (N)` above the left-rail list (`h3` in the agent
     list);
   - Fleet: add the five badges `category-count-Working`, `-Standby`,
     `-EmptySession`, `-RuntimeExited`, `-Worktree` (the Stale and Archived
     badges come from a separate query and are not part of the sum);
   - notice: the box `incomplete-agents-notice`, if present.
3. PASS = `rail == rail_expected` AND `fleet == fleet_expected`. The GUI must
   render ALL records, complete or not: a record lacking its daemon or repo id
   is shown, not dropped. If the notice is present it is information only.
   Report `records_lacking_proof` as an info line ("N records lack repo/daemon
   id"); it is not a failure and it will stay above zero until every home daemon
   is upgraded.
4. FAIL = either number is lower than expected (the hidden-fleet class), or the
   GUI shows 0 where the daemon lists agents. Stop the roll and report both sets
   of numbers.

Rules mirrored from `AgentList.tsx` and `FleetGrid.tsx` (rail: local-origin
rows, no users, no retired rows, deduplicated by bare name; Fleet: every
non-stale, non-archived record). If those components change the filter, update
`agent-count-expected.py` and its test in the same change.

### Red flags — STOP

- Pull not ff-only.
- Backup skipped or undersized.
- Storage accessed via forbidden path (`storage_access.forbidden`).
- Restart wrapped in a blind timeout or killed with `SIGQUIT`/`SIGUSR1` in a
  fatal window (`migration.signal_rules`).
- Install rebuilt locally instead of from the delivered artifact.
- Post-migration health not verified by observed effect.
- Template re-render skipped when the build changed templates (`templates.order`
  violated).
- GUI agent counts not compared with `agent.list` after the roll, or a lower
  rail or Fleet count accepted (section 8d).