E2e Template Testing

---
name: "e2e-template-testing"
description: "End-to-end validation of coordinator and agent template changes"
domain: "development"
confidence: "high"
source: "manual"
---

## Context

Squad's coordinator prompt (`squad.agent.md`) and agent charters (e.g.
`scribe-charter.md`) are shipped as templates in `.squad-templates/`. Changes to
these files affect how every squad session behaves — but unit tests can't catch
prompt-level regressions because the prompts are interpreted by an LLM at
runtime.

This skill describes how to validate template changes end-to-end by running real
squad sessions against a locally-built CLI that includes your modified templates.

## When To Use

- You changed `.squad-templates/squad.agent.md` (coordinator prompt)
- You changed `.squad-templates/scribe-charter.md` or other agent charters
- You changed `.squad-templates/notes-protocol.md` or helper scripts
- You added new conditional blocks (e.g. state-backend-aware spawn templates)
- You modified the init scaffolding that writes templates to target repos

## Prerequisites

- **Node.js** ≥20, **npm** ≥10
- **Git** CLI
- **GitHub Copilot CLI** (`copilot` or `ghcs`) installed
- A local clone of the squad repo on your feature branch

## Workflow

### Step 0 — Post initial tracking comment (**FIRST action — before anything else**)

If `PR_NUMBER` and `REPO` are both set, **the absolute first thing you do** — before
fast-fail checks, before building, before creating any repos — is post the initial
tracking comment with all steps marked as `:hourglass_flowing_sand: Pending`.

This gives reviewers immediate visibility that a run is in progress and what to expect.

```powershell
$runStart = Get-Date
$body = @"
## E2E Progress - PR $env:PR_NUMBER

| Step | Status | Started | Duration |
|---|---|---|---|
| 1. Fast-fail checks (build :cd: link :cd: ``squad version``) | :hourglass_flowing_sand: Pending | --:-- | -- |
| 2. Create test repo(s) | :hourglass_flowing_sand: Pending | --:-- | -- |
| 3. ``squad init`` + file verification | :hourglass_flowing_sand: Pending | --:-- | -- |
| 4. Run sessions | :hourglass_flowing_sand: Pending | --:-- | -- |
| 5. Verify outcomes | :hourglass_flowing_sand: Pending | --:-- | -- |
| 6. Record verdicts + post final comment | :hourglass_flowing_sand: Pending | --:-- | -- |

| Symbol | Meaning |
|---|---|
| :hourglass_flowing_sand: | Not started |
| :arrows_counterclockwise: | Running |
| :white_check_mark: | Passed |
| :x: | Failed |
| :warning: | Passed with caveats |

*Run started: $($runStart.ToString('HH:mm')) — all steps pending*
"@

$tmpFile = [System.IO.Path]::GetTempFileName()
$utf8NoBom = New-Object System.Text.UTF8Encoding $false
[System.IO.File]::WriteAllText($tmpFile, $body, $utf8NoBom)
$response = gh api "repos/$env:REPO/issues/$env:PR_NUMBER/comments" --method POST --field "body=@$tmpFile" | ConvertFrom-Json
$env:COMMENT_ID = $response.id
Remove-Item $tmpFile -Force
Write-Host "Progress comment posted — ID: $($response.id)"
```

After posting, immediately update the comment to mark Step 1 as `:arrows_counterclockwise: Running` (do NOT wait — this is a two-step sequence: post all-pending, then immediately update to Step 1 running). Then proceed to Step 1.

See **Progress Reporting** for the full comment lifecycle and update patterns.

**⚠️ STOP before continuing:** If posting the comment fails (network error, auth error), abort the run and report the failure. Do not proceed silently without a tracking comment.

### Step 1 — Build the CLI from your branch

```bash
cd /path/to/squad          # your feature branch
npm install
npm run build -w packages/squad-sdk && npm run build -w packages/squad-cli

# Link so `squad` command uses your local build (workspace flag — no cd required)
npm link -w packages/squad-cli
```

Verify: `squad version` output includes the `-preview` suffix (e.g., `x.y.z-preview`),
confirming the local dev build is active. If the output shows a plain semver without
`-preview`, the globally-installed npm package is still in use — re-check the link step.
See [CONTRIBUTING.md — Making the `squad` Command Use Your Local Build](../../../CONTRIBUTING.md)
for the full guidance on local dev versioning.

### Step 2 — Create a disposable test repo

```bash
mkdir /tmp/sq-test-1 && cd /tmp/sq-test-1
git init
echo "# Test Project" > README.md
echo '{"name":"test-project","version":"1.0.0"}' > package.json
mkdir src
echo "export function hello() { return 'world' }" > src/index.ts
git add -A && git commit -m "init: test project"
```

Keep the project small — you only need enough for the coordinator to recognize a
codebase and hire a team.

### Step 3 — Init a squad with your modified templates

```bash
squad init
# If testing a specific feature (e.g. state backends):
# squad init --state-backend git-notes
```

Verify the init produced the expected files:
```bash
ls -la .squad/
cat .squad/team.md          # should have ## Members with 3+ agents
cat .squad/config.json      # should reflect any CLI flags you passed
```

### Step 4 — Run a real session and capture output

Use the Copilot CLI's `-p` flag with `--allow-all-tools` for non-interactive sessions.
`--allow-all-tools` is **required** for automated/non-interactive runs — without it,
tool calls (including file writes) prompt for confirmation and block.

```powershell
# PowerShell (Windows)
copilot --agent squad --allow-all-tools -p "Picard, decide what testing framework to use. Write your decision." `
  2>&1 | Tee-Object evidence/session-task.log
```

```bash
# Bash (macOS/Linux)
copilot --agent squad --allow-all-tools -p "Picard, decide what testing framework to use. Write your decision." \
  2>&1 | tee evidence/session-task.log
```

Alternatively, set the `COPILOT_ALLOW_ALL=1` environment variable instead of the flag.

For multi-turn workflows, run sequential sessions:
```powershell
# Session A: give the team a task
copilot --agent squad --allow-all-tools -p "prompt A" 2>&1 | Tee-Object evidence/session-A.log

# Session B: verify state persisted
copilot --agent squad --allow-all-tools -p "What decisions has the team made?" 2>&1 | Tee-Object evidence/session-B.log
```

### Step 5 — Verify the outcome

Check that your template change had the expected effect. Common checks:

```bash
# State location (for state-backend changes)
git notes --ref=squad list              # git-notes backend
git ls-tree -r squad-state              # orphan backend
ls .squad/agents/*/history.md           # worktree backend

# Coordinator behavior (grep session log)
grep "STATE_BACKEND" evidence/session-task.log
grep "spawn" evidence/session-task.log

# File tree diff
git diff --stat HEAD~1                  # what changed on working branch
git log --all --oneline                 # commits across all branches
```

### Step 6 — Record the verdict

Create an `evidence/verdict.md` in each test repo:

```markdown
## Test: [scenario name]
**Backend:** worktree | git-notes | orphan | two-layer
**Branch:** [your feature branch]
**Result:** PASS | PARTIAL | FAIL
**Duration:** Xm Ys

### What was verified
- [ ] Coordinator identified feature correctly (from session log)
- [ ] Agent was spawned via `task` tool (not simulated)
- [ ] team.md has ## Members with 3+ agents
- [ ] State landed in correct location
- [ ] No unexpected side effects

### Evidence files
- session-task.log — full session output
- git-log.txt — `git log --all --oneline`

### Notes
[anything unusual or noteworthy]
```

Record the wall-clock time from the start of Step 1 (fast-fail checks) to the end
of Step 6 (verdict posted). This is the full E2E run duration for this scenario.

## Progress Reporting

Use this section only when you are running E2E validation for an open PR. If
`PR_NUMBER` and `REPO` are both set, post and maintain a live tracking comment
in the PR thread. If either value is missing (for example, a local-only run),
skip progress reporting silently.

### Start the tracking comment (Step 0 — see Workflow above)

The initial comment must be posted as **Step 0** — the absolute first action before
anything else. See the Step 0 block in the Workflow section for the exact code.

The subsections below describe how to **update** the comment at each step boundary.
For reference, here is the initial all-pending comment body posted in Step 0:

1. Post a PR comment before Step 1 begins:

```bash
gh pr comment "$PR_NUMBER" --repo "$REPO" --body "## E2E Progress\n\n| Step | Status | Started | Duration |
|---|---|---|---|
| 1. Fast-fail checks (build · link · \\`squad version\\`) | ⏳ Pending | --:-- | -- |
| 2. Create test repo(s) | ⏳ Pending | --:-- | -- |
| 3. \\`squad init\\` + file verification | ⏳ Pending | --:-- | -- |
| 4. Run sessions | ⏳ Pending | --:-- | -- |
| 5. Verify outcomes | ⏳ Pending | --:-- | -- |
| 6. Record verdicts + post final comment | ⏳ Pending | --:-- | -- |
\n| Symbol | Meaning |
|---|---|
| ⏳ | Not started |
| 🔄 | Running |
| ✅ | Passed |
| ❌ | Failed |
| ⚠️ | Passed with caveats |"
```

2. Capture the comment ID immediately after posting it:

```bash
COMMENT_ID=$(gh api "repos/$REPO/issues/$PR_NUMBER/comments" --jq '.[-1].id')
```

3. Treat Step 1 as in progress as soon as the comment exists. Update the body so
   Step 1 shows `🔄 Running` and every later step remains `⏳ Pending`.

### Update the tracking comment after every step boundary

1. When marking a step `🔄 Running`, record `$startTime = Get-Date` and store the
   `HH:MM` start time in that row's **Started** column.
2. Edit the existing comment in place; do not post a new progress comment:

```bash
gh api --method PATCH "repos/$REPO/issues/comments/$COMMENT_ID" --field body="..."
```

3. When marking a step `✅`, `❌`, or `⚠️`, compute
   `$duration = (Get-Date) - $startTime` and format it as
   `"{0}m {1}s" -f [int]$duration.TotalMinutes, $duration.Seconds`.
4. Update the completed step row to `✅`, `❌`, or `⚠️`, keep its original
   `HH:MM` value in **Started**, and write the formatted duration in **Duration**.
5. Keep all previously completed rows unchanged.
6. Mark the next step as `🔄 Running` and set its **Started** value.
7. Leave later steps as `⏳ Pending` with `--:--` for **Started** and `--` for
   **Duration**.
8. If a step fails and you stop early, still update the comment so the failed step
   shows `❌` with its original start time and computed duration, and Step 6
   becomes `🔄 Running` while you prepare the final verdict.

### Use this status legend in the comment

| Symbol | Meaning |
|---|---|
| ⏳ | Not started |
| 🔄 | Running |
| ✅ | Passed |
| ❌ | Failed |
| ⚠️ | Passed with caveats |

### Use exact step names and order

Keep these six rows in this exact order every time you update the comment:

1. Fast-fail checks (build · link · `squad version`)
2. Create test repo(s)
3. `squad init` + file verification
4. Run sessions
5. Verify outcomes
6. Record verdicts + post final comment

### Handle Windows comment bodies safely

On Windows PowerShell 5.1, use the **`--field body=@file`** pattern to post comment
bodies. Write the content to a temp file using UTF-8 **without BOM**, then pass
`--field "body=@$tmpFile"` to `gh api`. This is more reliable than piping JSON
through `--input -` on PS 5.1, which can silently corrupt multi-byte characters
even with `[Console]::OutputEncoding = UTF8`.

Key rules:
- Use `New-Object System.Text.UTF8Encoding $false` (the `$false` disables the BOM).
  `[System.Text.Encoding]::UTF8` writes a BOM which GitHub renders as a stray
  character (``) at the start of the comment.
- Use `--field "body=@$tmpFile"`, NOT `--input -` or `--input filename`, for
  comment body updates. The `@` prefix tells `gh` to read the field value from
  the file rather than treating the path as a literal string.
- Clean up the temp file after posting.
- Scrub any local absolute paths from the body before posting (see PII Protection
  section).

```powershell
$step1StartTime = Get-Date
$step1Started = $step1StartTime.ToString('HH:mm')
$step1Duration = (Get-Date) - $step1StartTime
$step1DurationText = "{0}m {1}s" -f [int]$step1Duration.TotalMinutes, $step1Duration.Seconds
$step2StartTime = Get-Date
$step2Started = $step2StartTime.ToString('HH:mm')
$body = @"
## E2E Progress

| Step | Status | Started | Duration |
|---|---|---|---|
| 1. Fast-fail checks (build · link · `squad version`) | :white_check_mark: Passed | $step1Started | $step1DurationText |
| 2. Create test repo(s) | :arrows_counterclockwise: Running | $step2Started | -- |
| 3. `squad init` + file verification | :hourglass_flowing_sand: Pending | --:-- | -- |
| 4. Run sessions | :hourglass_flowing_sand: Pending | --:-- | -- |
| 5. Verify outcomes | :hourglass_flowing_sand: Pending | --:-- | -- |
| 6. Record verdicts + post final comment | :hourglass_flowing_sand: Pending | --:-- | -- |

| Symbol | Meaning |
|---|---|
| :hourglass_flowing_sand: | Not started |
| :arrows_counterclockwise: | Running |
| :white_check_mark: | Passed |
| :x: | Failed |
| :warning: | Passed with caveats |
"@

$tmpFile = "$env:TEMP\e2e-comment-body.md"
$utf8NoBom = New-Object System.Text.UTF8Encoding $false
[System.IO.File]::WriteAllText($tmpFile, $body, $utf8NoBom)
gh api --method PATCH "repos/$env:REPO/issues/comments/$env:COMMENT_ID" --field "body=@$tmpFile"
Remove-Item $tmpFile -Force
```

### Progressive Verdicting — Post After Each Scenario (Critical)

**Do NOT batch all scenario results to the end.** This is the most common cause
of lost verdicts. After each scenario completes, **immediately** PATCH the
tracking comment with that scenario's result before moving to the next one.

The pattern for each scenario:

```powershell
# After scenario N completes — PATCH immediately, before starting scenario N+1
$scenarioNDuration = "{0}m {1}s" -f [int]((Get-Date) - $scenarioNStartTime).TotalMinutes, ((Get-Date) - $scenarioNStartTime).Seconds
# ...rebuild the full comment body with this scenario updated to PASS/FAIL/PARTIAL...
$tmpFile = [System.IO.Path]::GetTempFileName()
$utf8NoBom = New-Object System.Text.UTF8Encoding $false
[System.IO.File]::WriteAllText($tmpFile, $body, $utf8NoBom)
gh api --method PATCH "repos/$env:REPO/issues/comments/$env:COMMENT_ID" --field "body=@$tmpFile"
Remove-Item $tmpFile -Force
Write-Host "Scenario N verdict posted"
```

This guarantees that even if the AI model connection drops mid-run, the last
successfully PATCHed state is always visible in the PR.

### Agent Run Time Budget

**⚠️ Critical: Background agents lose their AI model connection after ~15 minutes
of continuous execution.** This is a platform limit, not a bug in your code.
The verdict stage appears to "hang" because the connection drops right at the end
when the agent has been running too long.

**Per-agent scenario budget:**

| Scenario type | Estimated time | Budget |
|---|---|---|
| Static checks only (file existence, grep, size) | 1-3 min | 4 per agent |
| `squad init` + file verification (no copilot session) | 3-5 min | 3 per agent |
| `squad init` + one `copilot --agent squad` session | 8-15 min | **1 per agent** |
| Build + link + one copilot session | 12-20 min | **1 per agent** |

**Rule: Limit yourself to 1 scenario that includes a `copilot --agent squad` session
per agent run.** For a plan with multiple copilot-session scenarios, run them in
separate agents — not in sequence within a single agent.

If your scenario plan has N copilot-session scenarios, request N separate sims
agents to run them in parallel (one scenario each). Static scenarios may be
batched up to 4 per agent.

**If you are running a scenario with a `copilot --agent squad` session:**
- Run the build and link ONCE at the start (shared across all static scenarios)
- Run the copilot session immediately after the repo is set up
- PATCH the comment with the result immediately after the session ends
- Then proceed to static scenarios while you still have connection budget

### Replace the tracking comment with the final verdict

When you reach Step 6, replace the tracking comment body entirely with the final
structured verdict table. Do not post a separate final comment. The tracking
comment is the final verdict comment.

Include a summary row at the bottom of the final table showing the total elapsed
time for the full run:

```text
| **Total** | — | HH:MM | Xm Ys |
```

**If the connection drops before Step 6:** The last progressive verdict PATCH
already shows the partial state. The next agent run should read the existing
comment, pick up where it left off, and add remaining scenario rows rather than
starting fresh.

## Test Matrix Template

Use this matrix when planning validation for a template change. Not every change
needs every row — pick the scenarios relevant to your modification.

| # | Scenario | What to verify | Duration |
|---|----------|----------------|----------|
| 1 | Basic init + task | Templates applied, agent spawned, work produced | — |
| 2 | Cross-branch persistence | State survives `git checkout` (if state-backend) | — |
| 3 | Scribe behavior | Scribe commits to correct target | — |
| 4 | PR cleanliness | Feature branch PR has no leaked state files | — |
| 5 | Migration path | Existing squad picks up new template behavior | — |
| 6 | Edge case: empty repo | Init works in repo with single commit | — |
| 7 | Edge case: monorepo | Init works in subdirectory of monorepo | — |

Note: Keep `—` during planning, then replace it with the actual elapsed time when
recording the verdict for each scenario.

## Tips

- **Name test repos descriptively:** `sq-test-notes-crossbranch`, not `test1`.
- **Always capture session logs.** Without logs, you can't debug failures.
- **One scenario per repo.** Don't reuse repos across unrelated tests — state
  leaks between tests make results unreliable.
- **Clean up after.** Delete test repos when done. They accumulate fast.
- **Windows users:** Use PowerShell. `Tee-Object` replaces `tee`. Paths use `\`.

## Fast-Fail Rules

These checks must pass before running any scenario. If any fail, stop
immediately and report the failure — do **not** attempt workarounds or mark
scenarios as SKIPPED.

0. **Clean stale SDK before building.** Before running `npm run build`, remove any
   stale published copy of `@bradygaster/squad-sdk` that may have been installed
   into `packages/squad-cli/node_modules/`. This local copy shadows the workspace
   symlink in the root `node_modules/` and causes TypeScript to see the published
   version instead of the local source. Run from the repo root:
   ```powershell
   $stale = "packages\squad-cli\node_modules\@bradygaster\squad-sdk"
   if (Test-Path $stale) { Remove-Item -Recurse -Force $stale; Write-Host "Cleaned stale SDK" }
   ```
   This is safe to run unconditionally — if the path doesn't exist, the command is
   a no-op. The root `node_modules\@bradygaster\squad-sdk` workspace symlink remains
   intact and npm will use it automatically.
1. **Build must succeed.** Run `npm run build` from the repo root. A build
   failure blocks all scenarios; report `BUILD_FAILED` and stop.
   - If the error is `tsc: not found` or similar missing-binary errors, run
     `npm install` first to reconcile `node_modules` with the lock file, then
     retry. This can happen after `git checkout HEAD -- package-lock.json`
     restores the lock file without reinstalling.
2. **CLI must link successfully.** `cd packages/squad-cli && npm link` must exit
   0. If it fails, report `LINK_FAILED` and stop.
3. **`squad version` must run.** After linking, `squad version` must output a
   version string. If not, report `CLI_NOT_FOUND` and stop.

Do **not** mark scenarios as SKIPPED due to build or environment errors —
that obscures real failures from reviewers. SKIPPED is only acceptable when the
user explicitly requests it.

## PII Protection — Mandatory

When posting evidence to PR comments, issues, or any shared document:

- **Never include absolute paths** that contain a local username (e.g.,
  `C:\Users\username\...` or `/home/username/...`).
- **Use `~` notation** for home-relative paths: `~/AppData/Local/Temp/...`
  or `~/tmp/sq-test-1`.
- **Repo-internal paths use `<repo-root>` as the prefix.** If evidence files live
  inside the repository (e.g. `.e2e/`, `tmp/`, or any subdirectory of the repo),
  write them as `<repo-root>\.e2e\pr-1035\evidence` — not as `~\..\..\...` backward
  navigation. `<repo-root>` is a clear, portable placeholder for the repository root
  that does not expose the machine's directory layout.
- **Scrub before posting.** Replace any occurrence of the local machine path
  prefix (everything up to and including the username segment) with `~`.

Example — ❌ wrong: `C:\Users\johndoe\AppData\Local\Temp\sq-e2e-pr1035\evidence`
Example — ✅ right: `~/AppData/Local/Temp/sq-e2e-pr1035/evidence`
Example — ❌ wrong: `~\..\..\repos\squad\.e2e\pr-1035\evidence`
Example — ✅ right: `<repo-root>\.e2e\pr-1035\evidence`

This applies to all evidence tables, verdict files, and PR comments.

## Anti-Patterns

- **Skipping the local build.** If you test with the published CLI, you're
  testing the old templates, not your changes.
- **Posting absolute paths in PR comments.** Always scrub to `~`-relative paths
  before sharing. See PII Protection above.
- **Marking scenarios SKIPPED due to environment issues.** Fix the environment
  (use fast-fail rules above) or report BUILD_FAILED — never silently skip.
- **Testing only the happy path.** Template changes often break edge cases (empty
  repos, monorepos, cross-branch). Test at least 2-3 scenarios.
- **Trusting session output alone.** Always verify git state independently —
  agents can claim they wrote something without actually doing it.
- **Reusing test repos.** Prior state bleeds into later tests. Start fresh.
- **Batching all scenario verdicts to the end.** The AI model connection drops
  after ~15 minutes. Always PATCH the comment after each scenario so partial
  results are never lost. See Progressive Verdicting above.
- **Running multiple `copilot --agent squad` sessions in one agent.** Each
  session takes 5-15 minutes; combined with build time, you'll hit the ~15-minute
  connection budget. One copilot session per agent — split into parallel agents
  if your plan has more.

## Sandbox / Permission Notes

### Always pass `--allow-all-tools` in non-interactive mode

The Copilot CLI requires explicit permission to run tools automatically. In
interactive mode the user approves each tool call; in non-interactive mode (`-p`)
those prompts cannot be displayed and writes fail silently or with a "Permission
denied and could not request permission from user" error.

Fix: always include `--allow-all-tools` (or `--yolo` / `--allow-all`) in Step 4
commands, or export `COPILOT_ALLOW_ALL=1` before running E2E sessions.

This also applies when `copilot --agent squad` is launched as a subprocess from
inside a Copilot CLI background agent (e.g. Sims running via the `task` tool) —
the flag is still needed.

### `--allow-all-paths` for repos outside the CWD

By default the CLI restricts file access to the current directory tree. If the
coordinator needs to read files from a parent repo while running in a disposable
test repo, add `--allow-all-paths`:

```powershell
copilot --agent squad --allow-all-tools --allow-all-paths -p "..."
```

Or use the combined shorthand: `--allow-all` / `--yolo`.

## Confidence

high — Validated through 12 real E2E test sessions during state-backend
development (PR #1004). `--allow-all-tools` requirement confirmed in PR #1035.