Coordinator Deploying A Node

---
name: coordinator-deploying-a-node
description: "Use when deploying a build to a single node, migrating or not — deploying a build, schema-migrating deploy, pro-bundle install plus service restart on a node, roll out a build to a node, ship to a node, single-node canary. Loads the deploy runbook — verifying the artifact actually CONTAINS the fix, install-before-restart ordering, verify-by-effect, and preventing store corruption on migration."
---

# Coordinator: Deploying a Build to a Single Node — migrating or not

> This runbook covers EVERY deploy to a single node, migrating or not. Most sections apply to every deploy; only
> migration-weight and store-corruption specifics are bump-only.

## 0. Pull-current first (ff-only)

The absolute first action before anything else: fetch the control-plane repository and fast-forward to the
orchestrator's pin.

- Read the project's deployment adapter inside `.thrum/config.json` (`fleet.deployment`; reference at the
  deployment-adapter reference) to obtain `preflight.pull_command` and `preflight.pin_command`.
- Run `preflight.pull_command` (must be ff-only; if not fast-forwardable, STOP and surface — never blind-merge in the
  shared tree).
- Verify the delivered artifact actually CONTAINS the fix at the pinned SHA (adapter `artifact.commit_file` sidecar).

This ensures the node's coordinator reasons from a checkout that is actually current before evaluating backup, install,
or migration.

> Example — Thrum project: pull via `git fetch origin && git merge --ff-only origin/<branch>` before any backup or install.

## 1. Who executes — resolve the node's local coordinator (via adapter identity)

The node's own coordinator drives the deploy locally on its own host. Never drive a node's build/install/restart from
another host.

- Resolve WHO via `topology.identity_resolution` (adapter `fleet.deployment.thrum.topology.identity_resolution` or
  `fleet.deployment.products.<id>.topology`) — a `daemon_id` join plus hostname and absence of `.thrum/redirect`, never
  by agent name or worktree name.
- When connected-repo daemons share a host, respect the adapter-defined restart order within that host.

If the target node has no working coordinator, state why explicitly before using any fallback.

## 2. Back up before touching anything (via adapter backup command + sizing formula)

- Run the project's backup command (`fleet.deployment.thrum.backup.command` or
  `fleet.deployment.products.<id>.backup.command`) and verify per `backup.verify`.
- Size BEFORE you run it on a bloated node: apply the adapter's `backup.sizing_formula`
  (`free_at_completion = free_now - zip(stale current/) + size(stale current/) - size(live lane)`) — the stale
  `current/` is removed before the fresh export, so peak is not double-count. Retention runs last and does not bound the
  peak.
- On failure, roll back via `backup.restore_command`; never delete the store.

> Example — Thrum project: backup via `thrum backup`, sized per the formula above, verified with
> `sqlite3 ?mode=ro&immutable=1` read.

## 3. Storage access — through the engine or control-plane CLI, never file-bypass

Access the state store only through the engine or the control-plane CLI in read-only mode. Never bypass the storage
engine's file ownership.

- Via: `storage_access.via` (e.g., CLI or `?mode=ro`).
- Forbidden: `storage_access.forbidden` (e.g., hand-copy of live `-wal`/`-shm`, concurrent `.backup` on live store).
- A bare read-write open can fire a checkpoint that truncates the write log out from under live readers → sustained
  corruption. Always use `?mode=ro` where applicable.

If the project has no WAL/SHM, this section is N/A — skip and note.

## 4. Never wrap service start/restart in a blind timeout

🔴 **On macOS, use `service.mac_restart_command`, not `service.restart_command`.** (Example — Thrum adapter:
`service.mac_restart_command` preserves Local Network (TCC) permission.) A daemon restarted from a terminal or agent
pane gets its Local Network (TCC) permission attributed to that terminal and is silently denied LAN peer dials
(`no route to host`): peers go dark while the node itself looks healthy, and `PPID 1` does NOT prove it was re-parented
correctly. Check the node's OS before choosing the command. After a mac restart, verify the peers on the LAN reconnect,
not just the version.

Run the project's service restart command (`fleet.deployment.thrum.service.restart_command`, or
`service.mac_restart_command` on macOS, or product equivalent) without a blind timeout wrapper. A timeout that kills a
migrating restart mid-migration manufactures a half-migrated store.

- Adapter flag `service.never_timeout` is true.
- Signal-window rules are adapter data: `migration.signal_rules` (`SIGQUIT` fatal-always, `SIGUSR1`
  fatal-before-socket). Respect them — do not send fatal signals in the forbidden window.

> Example — Thrum adapter: use the configured `service.mac_restart_command` on macOS; use `service.restart_command`
> otherwise. Never wrap either in `timeout`.

## 4a. EULA gate — before the binary swap

EULA gate (0.11+): before swapping the binary, record acceptance for the box user by running the new build once with
`THRUM_ACCEPT_EULA=<bundled version>`. Confirm that `~/.thrum/license/eula-acceptance-<version>-*.json` exists and that
a plain `thrum inbox` works. Skipping this breaks every agent's CLI on the box.

## 5. Install from the delivered artifact (never rebuild on the node)

Install from the identical bundle the orchestrator built, delivered to `artifact.distribution.drop_path`.

- Verify BEFORE unpack that the delivered bundle's version/checksum/commit sidecar equals the orchestrator's pin
  (adapter `artifact.checksum`, `artifact.version_file`, `artifact.commit_file`).
- Run `install.command` (from adapter). The node does not compile, sign, or rebuild — it installs.
- Collect BEFORE readings (current version, state counters, checkpoint counts, health probes) before the install.

> Example — Thrum project: install from the adapter's `artifact.distribution.drop_path` via the adapter's `install.command`.

## 6. Measure from the running process, never the checkout

Read version/state from the running service, not from the filesystem checkout.

- Verify via `install.verify_by_effect` probes: version equals pin, NEW PID, service status reports pin, sidecar equals pin.
- Do not trust installer exit 0 alone.

## 7. Migrate via service restart — verify by effect

Migration is performed by restarting the service after install; the service's own boot migration does the schema/data move.

- Calibrate weight via `migration.weight_calibration` (heavy bulk-export vs light add-column) — do not assume a fixed duration.
- Known signatures that require a cold-restart recovery: `migration.sigbus_signatures` (e.g., WAL-index recover
  signatures) and `migration.on_sigbus`.
- Observe that the restarted service reports the new version/state and that health probes have converged (adapter
  `health.probes`, `health.convergence_wait`).

## 8. Verify post-migration (liveness, health probes, checkpoint counts, agent liveness)

Run adapter `health.probes` and compare BEFORE/AFTER:

- Service alive and reporting pin.
- State counters converged per `health.convergence_wait` (mechanism-derived, e.g., `3m`).
- Crash log empty, expected agents live.

For tandem daemons that share a host or depend on each other, verify the coordinated set per adapter.

## 8a. Re-render configuration/templates if the build changes them

If the build carries template or role changes, re-render between install and restart:

- Commands: `templates.refresh_command` and `templates.deploy_command`.
- Order: `templates.order` (`refresh→deploy` between `install.sh` and daemon restart).
- Verify by effect (render output, then service restart).

> Example — Thrum project: `thrum roles refresh` then `thrum roles deploy` between install and restart.

## 8b. Plugin/skill refresh if the deploy carries skill changes

If the bundle carries skill/plugin changes, refresh the plugin cache:

- Refresh driver: `plugin.refresh` (version bump required — cache keyed by version).
- Manifests: `plugin.manifests` (all skill-carrying manifests bumped in the same commit as the new skills).

Do not hand-edit generated marketplace manifests.

## 8c. Reconcile registry/phase after restart

Re-verify the daemon registry and agent phases after restart (adapter `health.probes` includes phase reconciliation).
Confirm the pinned version is durable across a clean cold-boot probe where applicable.

## 8d. Verify the GUI shows every agent (by effect)

A build can decode `agent.list` fine on the daemon and still hide agents in the GUI (a strict UI decode once hid 485
of 538 on a canary). Compare what the daemon lists with what the GUI renders, on the node:

1. Expected numbers, run ON the node (python3 only, read-only, calls the daemon's own `agent.list` over its socket):
   `python3 <skill>/resources/agent-count-expected.py` prints `rail_expected`, `fleet_expected` and
   `records_lacking_proof`. Do not use `thrum team --all` as the baseline: it is `team.list`, a different set (stale
   rows included).
2. Rendered numbers. Needs Playwright plus a browser. The script tries Playwright's bundled Chromium, then the newest
   installed `chromium-<rev>` under the Playwright browsers dir (`PLAYWRIGHT_BROWSERS_PATH`, else the per-user
   `ms-playwright` cache; a Playwright upgrade can leave only other revisions), then an installed Google Chrome. It
   prints the route used (`browser=...`) on stderr and never installs a browser, so a node near its disk guard is not
   touched. `node <skill>/resources/agent-count-gui.mjs <ui-url>` (the `ui` line of `thrum daemon status`; the script
   header says where Playwright must resolve) prints `rail` and `fleet`; it exits 3 with a message when no browser is
   usable. Without a browser, open `<ui-url>` at 1280x720 and read by hand:
   - rail: the heading `Agents (N)` above the left-rail list (`h3` in the agent list);
   - Fleet: add the five badges `category-count-Working`, `-Standby`, `-EmptySession`, `-RuntimeExited`, `-Worktree`
     (the Stale and Archived badges come from a separate query and are not part of the sum);
   - notice: the box `incomplete-agents-notice`, if present.
3. PASS = `rail == rail_expected` AND `fleet == fleet_expected`. The GUI must render ALL records, complete or not: a
   record lacking its daemon or repo id is shown, not dropped. If the notice is present it is information only. Report
   `records_lacking_proof` as an info line ("N records lack repo/daemon id"); it is not a failure and it will stay
   above zero until every home daemon is upgraded.
4. FAIL = either number is lower than expected (the hidden-fleet class), or the GUI shows 0 where the daemon lists
   agents. Stop the roll and report both sets of numbers.

Rules mirrored from `AgentList.tsx` and `FleetGrid.tsx` (rail: local-origin rows, no users, no retired rows,
deduplicated by bare name; Fleet: every non-stale, non-archived record). If those components change the filter, update
`agent-count-expected.py` and its test in the same change.

## Red flags — STOP

- Pull not ff-only.
- Backup skipped or undersized.
- Storage accessed via forbidden path (`storage_access.forbidden`).
- Restart wrapped in a blind timeout or killed with `SIGQUIT`/`SIGUSR1` in a fatal window (`migration.signal_rules`).
- Install rebuilt locally instead of from the delivered artifact.
- Post-migration health not verified by observed effect.
- Template re-render skipped when the build changed templates (`templates.order` violated).
- GUI agent counts not compared with `agent.list` after the roll, or a lower rail or Fleet count accepted
  (section 8d).