Archie E2e

---
name: archie-e2e
description: Boot a live Archie instance from the current branch, drive E2E scenarios headlessly through the archie-debug MCP (nonce → create_task → wait_for_task → approve), capture per-scenario evidence files, and tear down cleanly. Use for Forge Stage 4 QA, verifying a branch against a live instance, running the basic or edit-mode E2E scenario, or when asked to "run the e2e harness", "boot archie for testing", or "verify this AC live".
---

# archie-e2e — boot-from-branch E2E harness

Verify acceptance criteria against a live Archie instance booted from the branch under test. The lifecycle is four steps, each with one documented invocation:

1. **Boot** — `npx tsx tools/e2e/boot.ts` builds and starts the instance detached, waits for `/health`.
2. **Drive** — execute scenario recipes through the `archie-debug` MCP (no scripts; you make the MCP calls).
3. **Evidence** — pipe a per-scenario payload to `npx tsx tools/e2e/evidence.ts`, which validates and writes the JSON + markdown pair.
4. **Teardown** — `npx tsx tools/e2e/teardown.ts` stops everything and verifies no project containers remain.

## Prerequisites

> Running in a cloud sandbox / CI behind a TLS-intercepting proxy (e.g. Claude Code on the web)? See `docs/guides/e2e-in-cloud-sandbox.md` for the cold-start setup (CA trust, key mapping, bind-mount ownership, `setsid` for long-running processes). The steps below assume direct internet egress.

- Docker (Desktop or daemon) running locally; `docker compose version` works.
- `.env` at the repo root with a non-empty `ANTHROPIC_API_KEY`. **No Slack tokens are needed** — scenarios enter through the CLI/API channel.
- The `archie-debug` MCP available (registered in `.mcp.json` as `npx tsx tools/debug-mcp/server.ts`). The harness consumes it as-is and never modifies `tools/debug-mcp/`.
- For the edit-mode and merge-approval scenarios: at least one configured engineering repo in the workdir (see the recipes' prerequisite notes).
- macOS Docker Desktop caveat: a wedged `docker-credential-desktop` helper can stall `docker compose up --build` during registry auth — upstream of the harness's bounded wait, so boot appears to hang before any diagnostics. Verify with `docker-credential-desktop list </dev/null`; if it hangs, point `DOCKER_CONFIG` at a scratch dir without `credsStore` for the run (leave your real `~/.docker/config.json` untouched). The scratch config also drops the `desktop-linux` context, so additionally set `DOCKER_HOST=unix://$HOME/.docker/run/docker.sock` or every docker command will fail to reach the daemon (observed in the 2026-07-05 QA run).

### Booting from a git worktree

A worktree checkout is a fresh tree, so `workdir/` and `claude-data/` do not exist in it and the boot creates empty ones. An instance booted that way has no history and no per-channel state: the CLI task list has nothing to paginate, and — because the channel store is what makes announce-once work — **the instance re-announces every `Archie…` canvas into its real Slack channel on first contact.** That is a live, user-visible post caused purely by test setup.

Before the first boot in a new worktree:

```bash
ln -s /path/to/main/checkout/workdir      workdir
ln -s /path/to/main/checkout/claude-data  claude-data
cp -R /path/to/main/checkout/secrets      secrets     # a real copy, NOT a symlink
```

**Do all three, and link `workdir` whole — that alone resolves the setup.** In particular it is what supplies the plugins: main's `workdir/plugins` is itself a symlink to a local `archie-plugins` checkout, so linking main's `workdir` gets you the configured agents and their repos for free. Creating `workdir/` yourself instead — or letting the boot create it — leaves `workdir/plugins` empty, and **that does not fail the boot**: `ARCHIE_PLUGINS` is unset in local dev so nothing clones, and `initWorkdir`'s guard only tests that the directory *exists*. The instance comes up healthy with **zero agents**, and the first symptom is a repo-change scenario where the PM never requests edit mode — which reads like a product bug rather than setup.

`workdir` is designed for concurrent access by many containers, so sharing it between checkouts is correct rather than risky — and copying it is not an option at ~20G.

**`secrets/` is the exception and must be a real directory.** `Dockerfile.dev` does `COPY secrets/ /tmp/ca-src/`, and Docker's build context does not follow symlinks at the context root: a symlinked `secrets` fails the build with `"/secrets": not found`. Bind mounts *do* follow host symlinks, which is why `workdir` and `claude-data` are fine. The same asymmetry is why `docker-compose.yml` re-mounts `./workdir/plugins` explicitly — a symlink *inside* a mounted tree does not resolve in the container.

Also set a free `PORT` in the worktree's `.env`. Main's 3000 is usually taken by another checkout's container; `boot.ts` will relocate around a squatter on its own, but the `archie-debug` MCP resolves its URL once at spawn, so an explicit port is one less thing to reconcile.

**Rollback:** delete `.claude/skills/archie-e2e/` and `tools/e2e/` — the harness has no side effects and nothing else references them (plus one `e2e-evidence/` line in `.gitignore`).

## 1. Boot

```bash
npx tsx tools/e2e/boot.ts [--timeout-seconds N]
```

The boot runs a strict ordered sequence; each step gates the next, so no failure ever falls through into the poll loop:

1. **Preflight** (mandatory, first): `.env` exists at the repo root and `ANTHROPIC_API_KEY` is non-empty. Failure exits non-zero naming the missing item, with no compose invocation at all.
2. **Port preflight**: the target port is probed before any compose invocation (skipped when `ARCHIE_URL` is set explicitly — that target is the operator's to manage). Free → proceed. Published by **this project's own archie** (checked against the container's published ports, not mere existence) → `docker compose down` first and boot fresh on the same port (the harness never adopts a running instance — it may be stale code; every boot builds from the current checkout). Held by **anything else** (a foreign archie from another worktree, an unrelated service) → a free port is auto-picked and the boot relocates there, leaving the squatter untouched.
3. **Compose up, exit code trapped**: `docker compose up --build -d` (detached — `npm run docker:dev` is foreground, don't use it here), with `PORT` and `GIT_SHA` passed in the compose environment. A non-zero exit prints the diagnostics block and exits non-zero **before a single `/health` poll** — a failed build never spends any of the wait cap.
4. **Health poll**: `GET {baseUrl}/health` every 5s until 200. Bounded by a default cap of **600s**, overridable via `--timeout-seconds` or `E2E_BOOT_TIMEOUT_SECONDS` (flag beats env beats default). Each tick also checks `docker compose ps --format json`: an `archie` container that is exited, dead, restarting (crash loop), or missing fails immediately instead of waiting out the cap.
5. **Checkout attestation**: `/health` must report the `git_sha` the boot passed (`git rev-parse HEAD`, `-dirty` suffix on an unclean tree). A healthy instance reporting anything else fails the boot — positive proof the instance was composed from the checkout under test, not a stale build or another checkout. Quote the `Attested: instance composed from <sha>` line in evidence.

On success it prints the resolved base URL, the `/health` body, and a final `ARCHIE_URL=http://localhost:<port>` line. **When the port was auto-picked, the driver MUST target that URL — and note the MCP constraint:** the archie-debug MCP resolves its URL once at spawn (`ARCHIE_URL` → `PORT` env → `PORT` from `.env` → `http://localhost:3000`), so an MCP already attached to your session (spawned before boot) CANNOT be retargeted by exporting a variable afterwards. Either drive via a fresh stdio client you spawn yourself with `ARCHIE_URL` in its environment (`ARCHIE_URL=... npx tsx tools/debug-mcp/server.ts`, as the QA runner does), or restart the session/MCP so it picks the value up. On the default port none of this matters — the resolutions already agree.

**Failure diagnostics** (printed on any compose-up or poll-phase failure, always followed by a non-zero exit): the `docker compose ps` table plus the last 100 lines of the archie container's logs (`docker compose logs --no-color --tail=100 archie`). Read the log tail first — missing env keys, plugin clone failures, and crash loops all surface there.

**Boot time observations**: cold `--build` boot: ~45s build + ~2min to healthy (M-series Mac, 2026-07-05 QA run of forge run pr-167-mcp-file-bridge); warm boot (image cached): _not yet measured_. Tuning guidance: set `--timeout-seconds` to roughly **2× the observed cold time** (~330s) for CI-like runs, and near the warm time for tight iteration once measured. The generous 600s default remains a safe ceiling.

## 2. Drive scenarios (archie-debug MCP recipes)

Scenarios are agent-driven: you call the MCP tools directly, following these recipes. The recipe names below are canonical and **must be used verbatim as the evidence files' `scenario` field**.

**Constraints (all recipes):**

- **Serial only** — run one scenario at a time; parallel runs are unvalidated.
- **Nonce window** — `wait_for_task`'s nonce scan covers only the **25 most-recent tasks**, so on a long-lived instance old nonces fall out of the window; remedy: fresh boot before a scenario session.
- Mint each nonce fresh: `E2E-$(openssl rand -hex 4)`.

### Recipe: `basic-nonce`

A read-only question reaches `completed` with an observable PM reply. Verifies the create → correlate → wait → assert loop (brief AC2).

1. Mint `NONCE=E2E-$(openssl rand -hex 4)`.
2. `create_task` with the nonce embedded in the message, e.g. `"[${NONCE}] What agents are configured in this instance? Reply with a short list and do not modify anything."`
3. `wait_for_task(nonce: NONCE)` — first call correlates the nonce to a task id and returns `TASK=<id>`.
4. While `STATE=pending`: call `wait_for_task(task_id: <id>, cursor: <CURSOR from the previous call>)` again — each call waits up to ~45s server-side and returns a resumable `CURSOR`.
5. Terminal: assert `STATE=completed`, and a `PM_REPLY` line was observed (or fetch `get_events(task_id)` and assert a `message` event with `data.from === 'pm-agent'`).
6. Excerpts for evidence: `get_log(task_id, tail: 40)` for the transcript (must contain the nonce), `get_events(task_id)` for `task:created` … `task:completed`.

### Recipe: `edit-mode-approval`

A change request against a configured repo trips the edit-mode gate, is approved via the API path, and proceeds to completion (brief AC3).

**Prerequisite:** the workdir has at least one configured engineering repo (the PM only requests edit mode when a code change targets a repo the plugin config declares). If the prerequisite is unmet, do NOT fake a pass — report the scenario as `BLOCKED`, stating: what is missing (no configured engineering repo in the workdir), what was attempted, and the remedy (configure a repo via the plugins setup, reboot, re-run). A BLOCKED scenario produces no evidence file; it appears in the run report only.

1. Mint a fresh nonce as above.
2. `create_task` with a small, real change request against a configured repo, e.g. `"[${NONCE}] In <repo>, add a comment line '// archie-e2e touch' to the top of README-adjacent file X and open a PR."`
3. `wait_for_task(nonce: NONCE)`, then resume with `task_id` + `cursor` while `STATE=pending`.
4. On `STATE=approval_requested`: read the approval type from the `APPROVAL_TYPE=` line of the `wait_for_task` output, then call `approve(task_id: <id>, type: "edit_mode", approve: true)` (this is the `POST /api/tasks/:id/approve` path).
5. Continue the `wait_for_task` loop; assert the task reaches `STATE=completed`.
6. Excerpts for evidence: events must show `approval:requested` (`data.approvalType: "edit_mode"`) followed by `approval:resolved` and eventually `task:completed`; the transcript records the approval decision line.

### Recipe: `merge-approval-deny`

An explicit merge request in a repo without `autoMerge: true` trips the `merge` approval gate, is denied via the API path, and no merge ever executes — the deny path keeps QA from really merging anything (pr-merge-policy AC3).

**Prerequisite:** same as `edit-mode-approval` — the workdir has at least one configured engineering repo, and that repo must not resolve to `autoMerge: true` (the shipped default: any repo whose declaring agents don't all set the flag). If the prerequisite is unmet, do NOT fake a pass — report the scenario as `BLOCKED`, stating: what is missing (no configured engineering repo in the workdir, or only auto-merge repos), what was attempted, and the remedy (configure a repo via the plugins setup, reboot, re-run). A BLOCKED scenario produces no evidence file; it appears in the run report only.

1. Mint a fresh nonce as above.
2. `create_task` with a small, real change request against a configured repo that ends in an open PR, e.g. `"[${NONCE}] In <repo>, add a comment line '// archie-e2e touch' to the top of README-adjacent file X and open a PR. Do not merge it."`
3. `wait_for_task(nonce: NONCE)`, then resume with `task_id` + `cursor` while `STATE=pending`. On `STATE=approval_requested` with `APPROVAL_TYPE=edit_mode`: `approve(task_id: <id>, type: "edit_mode", approve: true)` and keep waiting.
4. Wait until the task settles with the PR opened (`STATE=completed`, PM reply announcing the PR). Note the PR number and repo from the PM reply or the knowledge log — the approve call in step 7 needs exactly this identity.
5. `send_message(task_id: <id>, message: "[${NONCE}] Please merge that PR.")` — the PM's `merge_pull_request` call posts the merge approval prompt instead of merging.
6. Resume the `wait_for_task` loop; assert `STATE=approval_requested` with `APPROVAL_TYPE=merge`. The knowledge log now carries the decision finding `Merge approval requested for <github>#<pr_number>` — this names the identity to resolve against.
7. `approve(task_id: <id>, type: "merge", approve: false, github: "<github>", pr_number: <pr_number>)` with the identity of the PR the scenario drove open — the API rejects merge-type resolutions that omit `github`/`pr_number`.
8. Continue the `wait_for_task` loop; assert the task settles (`STATE=completed`) and **no merge occurred**: the knowledge log contains the denial finding `Merge denied by user — PR not merged`, and neither the log nor the events contain a merged completion (`PR … merged on user approval`) for this PR.
9. Excerpts for evidence: events must show `approval:requested` (`data.approvalType: "merge"`) followed by `approval:resolved` (`data.approve: false`) with no merge in between; transcript lines for the merge-approval request and the denial finding.

## 3. Capture evidence

Each scenario produces a `<scenario>.json` (canonical) + `<scenario>.md` (rendered for reviewers) pair, written by the validated writer:

```bash
# assemble the payload, then:
cat payload.json | npx tsx tools/e2e/evidence.ts --out-dir <dir>
# or: npx tsx tools/e2e/evidence.ts --in payload.json --out-dir <dir>
```

**Destination:** `--out-dir` flag → `E2E_EVIDENCE_DIR` env → default `./e2e-evidence/` at the repo root. The default is **gitignored** local scratch; Forge Stage 4 passes `--out-dir openspec/changes/<change>/qa-evidence/`, which **is** committed. One out-dir with `ac_ids` routing inside the files is the default convention; per-AC directories (`qa-evidence/<AC-id>/`) also work by pointing `--out-dir` per invocation.

**All-or-nothing semantics:** stdin is read fully before parsing; truncated or malformed JSON produces a classed error ("truncated JSON input from stdin" / "invalid JSON from stdin: …"), a non-zero exit, and no files. An invalid payload exits non-zero naming every validation error and writes nothing. The pair is written atomically (temp-file + rename) and transactionally — both files land or neither. A half-written evidence file is silent poison for the reviewer judging pass/fail from the file alone; the writer makes that state unrepresentable.

### Schema: `archie-e2e-evidence/v2` (field by field)

| Field | Type | Meaning |
|---|---|---|
| `schema` | `"archie-e2e-evidence/v2"` | Schema tag, exact string. |
| `scenario` | string, kebab-case | Canonical recipe name (`basic-nonce`, `edit-mode-approval`, `merge-approval-deny`); names the output files. |
| `ac_ids` | string[], non-empty | Brief AC ids this scenario verifies, e.g. `["AC2"]`. |
| `started_at` / `finished_at` | ISO 8601 strings | Scenario wall-clock bounds. |
| `environment.base_url` | string | Resolved base URL the scenario ran against. |
| `environment.git_branch` | string | Branch under test (`git branch --show-current`). |
| `environment.git_commit` | string | Commit under test (`git rev-parse --short HEAD`). |
| `nonce` | string | The minted nonce, e.g. `E2E-a1b2c3d4`. |
| `task_id` | string | Correlated task id from `wait_for_task`. |
| `terminal_state` | enum | `completed \| stopped \| approval_requested \| pending \| not_found`. |
| `assertions` | array, non-empty | Each: `id`, `description`, `expected`, `observed` (all non-empty strings), `pass` (boolean). |
| `excerpts.transcript` | string[] | Verbatim transcript lines (from `get_log`, rendered from `events.jsonl` — `knowledge.log` is not served over the API) the assertions rest on. |
| `excerpts.events` | array | Verbatim event objects (from `get_events`) the assertions rest on. |
| `result` | `"pass" \| "fail"` | Overall verdict; the validator enforces it equals the AND of assertion passes. |

The markdown companion is rendered mechanically from the JSON in the same invocation (metadata header, assertion table, fenced excerpts, verdict); the JSON is canonical if they ever seem to diverge.

## 3b. Egress check (run on any branch that touches the sandbox or bumps the SDK)

```bash
npx tsx tools/e2e/egress-check.ts        # requires a booted instance
```

Asserts the documented network boundary on the live instance: it copies `tools/e2e/egress-probe.mts` into the container, spawns one throwaway sandboxed agent (Haiku, two Bash calls) using the real `src/agents/sandbox.ts`, and requires **both** directions — a host off the allowlist must be blocked, and a host on it must be reachable. Exits non-zero naming which direction failed, with the probe transcript.

Why it exists: the allowlist is enforced by the Claude CLI, not by our code, and it silently stopped being enforced between CLI 2.1.156 and 2.1.157 while our configuration was unchanged. Archie picked that up transitively in a lockfile refresh and ran with unrestricted agent egress for six weeks. No unit test can catch that class of regression — only a live assertion can. Run this after any SDK bump, any `src/agents/sandbox.ts` or `spawn.ts` change, and before shipping anything that relies on the isolation claims in `SECURITY.md`.

A failure here is a release blocker, not a flake. If the non-allowlisted host is reachable, agent egress is open.

## 3c. Tool-approval-gate check (run on any branch that touches the gate, spawn hooks, or bumps the SDK)

```bash
npx tsx tools/e2e/tool-gate-check.ts     # requires a booted instance
```

Asserts the MCP tool approval gate end to end on the live instance, using the `gatecheck` example plugin (a stub MCP server whose one mutation appends to `workdir/e2e/gate-marker.log`, observable from the host; its policy, including the button title, is the `archie` block on the server in `examples/plugins/.mcp.json`): a gated call must produce a `tool_call` approval request **without executing**, approving via the API must let the agent's retry execute **exactly once**, and the fixture's `allow`-tier tool must pass ungated. Same rationale as the egress check: the interception lives in the Claude CLI (`PreToolUse` under `bypassPermissions`), is version-coupled, and no unit test can notice the CLI silently ceasing to deliver MCP calls to hooks. A failure is a release blocker for anything relying on the gate.

Prerequisite: the booted workdir includes the `gatecheck` example plugin (`npm run example:setup` links `examples/plugins`, which ships it). The check is self-driving over the HTTP API — no MCP session needed — and takes one model round-trip per phase (~2-5 min total).

## 4. Teardown

```bash
npx tsx tools/e2e/teardown.ts
```

Runs `docker compose down`, then verifies via `docker compose ps --all --format json` that **no containers remain for the project** — survivors (including stopped-but-not-removed ones) are listed by name and the command exits non-zero. On success it prints a confirmation line suitable for pasting into evidence: `Teardown clean: docker compose ps --all reports no containers for this project.` Never end a scenario session without a clean teardown exit.