Coordinate Landing

---
name: coordinate-landing
description: >
  Coordinate atomic landing, recovery, rollback, and release of a cross-repo Epic. Load whenever
  one Epic spans several repositories, including planning its landing saga.
---

# The landing saga

A single feature often spans several repos — a `golib` change plus the two services that consume it.
Under the a-novel model those repos are **independently versioned and independently merged**, so
"land them together" is not free: it is a **saga**, a coordinated sequence with compensating actions
when a step fails. This skill maps that saga: what the pieces are called, the one invariant they all
serve, and the operator procedures that need core or a repository admin.

The contributor-facing half — what a held or frozen Pull Request means to the person who opened it —
is published in
[`a-novel-kit/.github` › docs/board-lifecycle.md](https://github.com/a-novel-kit/.github/blob/master/docs/board-lifecycle.md).
This skill is the operator's view.

The saga is enforced by the `a-novel-kit/workflows` governance actions, driven by two triggers: the
per-PR / per-merge-group events, and a **reconcile sweep** that runs every ~15 minutes as a
level-triggered, self-healing floor. Nothing here is a bespoke distributed-transaction engine — it is
a small set of GitHub-native mechanisms (a required check, the merge queue, check-runs, a dispatch
action) composed to make a multi-repo landing behave like one.

---

## The Epic Atomicity Rule (INV-1)

> **All of an Epic's member PRs land together, or none of them do.**

An Epic is a planning issue; its member PRs each carry the `epic:<N>` label. INV-1 is the invariant
the whole saga exists to protect: a consumer must never merge while the dependency it needs is still
unmerged (or vice-versa), because that leaves a repo un-buildable. Every mechanism below is either an
_enforcer_ of INV-1 (merge-gate, the merge queue) or a _compensator_ for a violation of it
(epic-freeze recovers, epic-rollback undoes). When a saga decision is unclear, ask which side of
INV-1 it serves.

INV-1 is what makes the release train safe to run in **any order**: once an Epic has landed
atomically, its repos are mutually consistent, so releasing them is order-independent. Cross-**Epic**
ordering — Epic B depends on Epic A's release — is _not_ INV-1's job; that is `manage-versions`.

---

## Vocabulary — freeze it here, use it everywhere

Use these terms identically in code, PRs, issues, and conversation. No synonyms; never one word for
two things. The vocabulary is a contract, and a reused name is a future bug.

| Term                                       | Meaning                                                                                                                                                                                                                                                                                     |
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Epic**                                   | The planning issue grouping a multi-repo change; its number is `N`.                                                                                                                                                                                                                         |
| **`epic:<N>` membership**                  | The label binding a PR to Epic N. Author-role-gated (only a maintainer can add it), so membership is trusted. It defines the set until activation freezes it — see **Activation snapshot**.                                                                                                 |
| **Activation snapshot**                    | The member set frozen into the Epic issue body once it has held merge-ready long enough to settle. From then on the snapshot _is_ the set: a PR de-labelled, closed, or relabelled afterwards stays a member.                                                                               |
| **Wave**                                   | One frozen set landing. A PR labelled after the freeze joins the **next** wave; the snapshot retires once every member merged (or, after a rollback, none is open).                                                                                                                         |
| **Atomic landing**                         | All member PRs merging together — INV-1 satisfied.                                                                                                                                                                                                                                          |
| **merge-gate**                             | The required status check that **holds** an `epic:<N>` PR until the whole member set is ready, approved and passing `review-gate`, then lets the merge queue land them together. A standalone (unlabelled) PR fast-paths to pass. On engagement of the halt it posts `failure` on every PR. |
| **merge queue**                            | GitHub's native queue. The merge-gate re-evaluates over each frozen `gh-readonly-queue/...` head so the set greens and commits together.                                                                                                                                                    |
| **Partial landing**                        | An Epic where some members merged but others did not — an INV-1 violation (e.g. a member left the queue and did not re-enter).                                                                                                                                                              |
| **epic-freeze / partial-landing detector** | The action + sweep pass that detects a partial landing and **freezes** every surviving sibling (posts `failure` on their heads + dequeues live groups) so no further member lands.                                                                                                          |
| **Grace window**                           | The 45-minute interval (3× the 15-min sweep) a stray sibling has to re-enter the queue before the freeze trips — absorbs normal queue churn.                                                                                                                                                |
| **Roll-forward**                           | Re-enqueuing a _landable_ stray within grace (enable auto-merge). Recovery **forward**, not a rollback — the preferred repair.                                                                                                                                                              |
| **epic-rollback**                          | The human-triggered, admin-gated, VCS-layer compensator: reconstruct the current wave's merged `epic:<N>` ledger, revert each squash newest-first under a **fresh** rollback-Epic, and land them in reverse through the merge-gate.                                                         |
| **Release train**                          | One admin dispatch that releases every repo an Epic landed in — derive each repo's bump, drive its `release.yaml`, record the tag.                                                                                                                                                          |
| **Receipt**                                | The tag a repo's release cut, recorded on the Epic. The release train's output **and** its idempotent-resume ledger.                                                                                                                                                                        |
| **AGENT_KILL_SWITCH**                      | The org-wide fail-safe emergency halt (an org variable).                                                                                                                                                                                                                                    |
| **`automation:paused`**                    | The per-Epic pause label on the Epic issue — holds THAT Epic's automation, best-effort.                                                                                                                                                                                                     |
| **Blast-cap**                              | The per-Epic distinct-repo cap that trips a loud alert (freeze) or a hard abort (rollback) when an Epic's fan-out is suspiciously wide.                                                                                                                                                     |
| **Saga**                                   | The whole coordinated land → detect → recover → release → archive lifecycle.                                                                                                                                                                                                                |
| **Coordinator**                            | The set of governance actions that drive the saga (the reconcile sweep + the per-event workflows).                                                                                                                                                                                          |

---

## Why membership freezes

Before landing starts, membership is the live `epic:<N>` label search. That works until a member is
de-labelled mid-landing: GitHub indexes no "ever carried label X", so the PR vanishes from the set,
and the detector reads the survivors as a clean landing while a member sits abandoned. The set is
therefore captured into the Epic body once it has held merge-ready long enough for the label index to
settle, and from that point the snapshot is the authority.

Two consequences worth knowing before operating an Epic:

- **A de-labelled member is still a member.** Removing the label does not remove a PR from an
  in-flight landing. To stop the landing, use `automation:paused` on the Epic.
- **The snapshot names members; it does not authorise them.** Each member must still show it carried
  `epic:<N>` — currently, or in its own label history. A named PR that never did voids the whole
  snapshot, because the Epic body is editable and its members become freeze targets.

---

## What a merge group is not

A queued PR is validated as a **merge group** — a synthetic `gh-readonly-queue/<base>/pr-<N>-<sha>`
ref — not as the pull request. The group has no author and no labels, so every PR-level exemption
evaporates there. A ruleset `bypass_actors` entry is one: a bot allowance that lets a dependency PR
merge does not survive the queue, and the base branch's required checks come back as hard
requirements. Never assume a bypass covers the queue.

GitHub offers no way to scope a required status check to pull requests only. A flaky third party
therefore cannot be exempted for the queue; the fix is to drop it as a required check and read its
result from inside a check you control.

When a group sits pending, separate the two failure modes first: `commits/<sha>/check-runs` (Actions
jobs) and `commits/<sha>/status` (third-party commit statuses) are different APIs. A green job list
beside an empty status list means your upload job worked and the provider never reported back. The
group then waits on a status that will never arrive, blocking every PR queued behind it until the
queue's check-response timeout expires.

---

## The saga lifecycle

```
author → LAND (merge-gate + queue, atomic) → DETECT (reconcile sweep)
                                                 ├─ whole?  → RELEASE (release train) → ARCHIVE
                                                 └─ partial → RECOVER (freeze + roll-forward within grace)
                                                                └─ unrecoverable → ROLL BACK (human-approved)
```

1. **Author.** A maintainer labels each member PR `epic:<N>`.
2. **Land.** The merge-gate holds each member until the whole set is ready, approved and passing
   `review-gate`, then the merge queue lands them together — INV-1 satisfied atomically. A
   standalone PR is unaffected.
3. **Detect.** The reconcile sweep (every ~15 min, level-triggered) re-derives each open Epic's state
   from live GitHub truth — it never trusts a stored flag, so it self-heals after any missed webhook.
4. **Recover.** On a detected partial landing it **freezes** the surviving siblings and **rolls
   forward** any landable stray within the grace window. A frozen sibling holds (its required check
   goes red) until the Epic is whole again or a human intervenes.
5. **Release.** Once whole, the **release train** releases the Epic's repos from one admin dispatch and
   records a tag receipt per repo.
6. **Archive.** Each repo's `release.yaml` clears its awaiting-release board items after it ships.

---

## Operator runbook

Every entry is admin-gated at the point of action; the destructive ones (rollback) additionally force
a human approval. All the write actions honor `AGENT_KILL_SWITCH` and `automation:paused`. Check the
operator's role first with [check-access](../check-access/SKILL.md): a developer reports the
incident to core instead of acting.

### Halt everything (incident brake)

Set the **`AGENT_KILL_SWITCH`** org variable (in the affected org) to any value that is **not** an
off-token — e.g. `on`. Effect, org-wide and immediate on the next event/sweep:

- **merge-gate posts `failure` on every PR** — nothing merges (the halt rests on the gate holding
  _every_ PR, so it can't fast-path a standalone).
- Every board writer, auto-merge arm, freeze poster, and rollback **no-ops or refuses**.

**Lift** by setting the value back to an off-token — canonically **`off`** (a created org variable
cannot be empty, so `off` is the resting value; unset behaves the same but isn't discoverable). The
switch is **fail-safe**: a fat-fingered or garbage value halts, not runs. It is a cooperative in-action
flag, not a security boundary — if the incident is a compromised App, revoke the installation / rotate
`AGENT_BOT_PRIVATE_KEY` instead.

### Pause one Epic

Add the **`automation:paused`** label to the Epic issue. Holds THAT Epic's automation (merge-gate holds
its members; the detector and rollback skip it) while other Epics keep moving. **Best-effort** — a
label-read blip fails open (the pause is skipped that pass); it is an operator convenience, not the
hard halt. Remove the label to resume.

### Recover a partial landing

Usually **automatic**: the sweep freezes the surviving siblings and rolls a landable stray forward
within grace. Intervene only when:

- **A stray can't land** (merge conflict, failing checks). The freeze holds the whole surviving set
  (their required check is red) — fix the stray and let it re-enter the queue, or escalate to a
  rollback if the landed subset is genuinely broken.
- **The freeze is very wide** (blast-cap tripwire fired: a loud red sweep). A freeze spanning more
  distinct repos than the cap is almost certainly a mis-scoped Epic — it still posts (fail toward
  freezing), but investigate the Epic's membership first.

### Roll back an Epic (INV-1 genuinely violated)

Dispatch **`epic-rollback`** for Epic N. It is admin-only, `dry_run`-default, and typed-confirm
(`revert-epic-<N>`). Always **dry-run first** — it prints the reconstructed ledger + the planned
per-repo reverts and does no writes. Then run live:

- It reconstructs the **current wave's** merged `epic:<N>` ledger from the snapshot's wave boundary
  on (earlier waves landed whole and are left alone; an unreadable boundary stops the run), using
  REST-authoritative squash SHAs, git-reverts each squash **newest-first**, opens one revert PR per
  repo, and groups them under a **fresh** rollback-Epic `epic:<M>` through the unchanged merge-gate.
- It labels Epic N **`rolled-back`**. merge-gate then retires N's frozen wave once **no member is
  open**, on the next PR labelled `epic:<N>`, so the Epic can carry a new wave. Close or land any
  member still open first. A new wave's freeze clears the label, so it never applies to a later wave.
- The App authored the reverts, and **GitHub 422s a self-approval**, so the wave **PARKS pending a
  human approval** — forced four-eyes on a destructive op. Review + approve each revert PR; the gate
  then greens and the wave lands in reverse, atomically.
- A revert conflict opens a **HELD draft** placeholder that holds the whole wave (no partial rollback);
  finish it by hand. The blast-cap **aborts loud** above the cap (a rollback that wide is opt-in — fail
  toward NOT reverting).

A rollback does not un-merge history — each revert is a new forward commit, so a mistaken rollback is
itself revertible.

### Release an Epic (the release train)

Dispatch **`release-train`** for Epic N (admin-gated, `dry_run`-default). Rehearse first (the dry run
dispatches each repo's `release.yaml` with `dry_run=true` and cuts nothing), then run live:

- It reconstructs the Epic's landed repos, derives each repo's semver **bump** from its
  conventional-commit range (fix→patch, feat→minor, `!`/`BREAKING CHANGE`→major), and drives each
  repo's own `release.yaml` — any order (INV-1 makes them independent).
- It records a **receipt** (the cut tag) per repo on the Epic. **Idempotent resume**: a re-dispatch
  skips every repo already shipped for this Epic (derived from _live tags_, never the receipt), so a
  partial train re-cuts only the unshipped.
- A live train waits up to an hour at each repo's protected `release` environment and links the
  cut in its summary: approve it there. A rehearsal never meets the environment. A cut nobody
  approves in time is **pending**, not failed — approve the run, then re-dispatch to record the
  receipt. The receipt marks the tag; the tag's own run publishes the release. A **botched cut** (tag pushed, Release
  missing) is flagged loudly for manual repair, never silently skipped.

### Cross-repo hotfix

A bug on a released line that spans several repos is **not** a special "hotfix train" — it is a
**standard Epic, run fast**: a maintainer labels the fix PRs `epic:<N>`, the merge-gate lands them
atomically, and the release train releases them. There is deliberately **no cross-repo hotfix orchestrator** — the Epic
machinery already gives atomicity plus a coordinated release. The _single-repo_ hotfix path
(`hotfix.yaml`: fix on the default branch → backport into `release/vX.Y` → cut on merge) and its vocabulary live in
`manage-versions`.

---

## Version coordination — the two rules `manage-versions` owns

The saga lands and releases a set of repos; keeping them _version_-compatible across that release is
`manage-versions`' domain. Two rules from there govern how a saga is shaped:

- **Publish-before-rollout.** When one repo's change is needed by another, the **dependency PR merges
  and releases FIRST**, and the consumer re-pins to the released tag before _its_ PR merges. Within a
  single Epic the merge-gate lands the set together, but the _release_ order across repos still honors
  this: release the dependency, re-pin, then release the consumer (the release train derives bumps
  per repo; cross-repo re-pins are `manage-versions`').
- **Expand→contract** (staged breaking change). A breaking change never ships in one step: **expand**
  — add the new path alongside the old, non-breaking, and release; migrate every consumer; **contract**
  — remove the old path in a later release. Drafted ahead as `blocked-by` sub-issues. This is how a
  cross-repo breaking change stays landable atomically at every step: no single merge breaks a
  consumer, so INV-1 holds throughout.

See `manage-versions` for the mechanics (exact `go.mod` pins, pseudo-version development against an
unreleased dep, the tag-push release, Renovate re-bumps) and its **landing-failed runbook** for the
version-recovery decision tree.

---

## How this composes

```
plan-feature (creates the Epic + Task sub-issues)
   └─ implement-feature (per-repo branches, one PR per Task; a maintainer labels each epic:<N>)
        └─ THE SAGA (this skill): merge-gate lands the set atomically (INV-1)
             ├─ partial landing → epic-freeze recovers, or epic-rollback undoes (human-approved)
             ├─ manage-versions: publish-before-rollout · expand→contract · landing-failed runbook
             └─ release-train releases the Epic's repos → archive
```

- **Enforcer skills:** none — read the `a-novel-kit/workflows` actions for the ground truth.
- **`resolve-pr-feedback`** for the conversation on a **held or frozen** PR — explain _why_ it is held
  (waiting on its Epic set / frozen by a partial landing), not just _that_ it is.
- **`manage-versions`** for anything version-shaped in the rollout.

---

## Principles

The mechanisms are described above; these are the judgment calls they encode.

- **INV-1 is the north star.** Every mechanism enforces "land together or not at all," or compensates
  for a violation.
- **Recover forward before you roll back.** Rollback is the last resort, and it is human-approved.
- **The sweep is the floor, level-triggered.** Add no stateful shortcut.
- **Halt is fail-safe and cooperative.** Revoke the App for a real compromise.
- **Freeze the vocabulary, use it consistently.**