Runtime Canary

---
name: runtime-canary
description: >-
  Check that Claude Code, Codex, and Copilot CLI find and run deployed skills, by deploying this checkout into a
  throwaway home and starting each runtime there. Use it before a pull request that changes skill paths,
  allowed-tools, runtime adapters, or agents. Each run calls a model.
allowed-tools: ["Bash(python -B tools/runtime_canary.py *)", "PowerShell(python -B tools/runtime_canary.py *)"]
---

# Runtime canary

`tools/runtime_canary.py` deploys this checkout and the fixture skill `runtime-canary-probe` into a fresh home under the temporary directory, which `deploy.py --canary-home` allows from a worktree. Nothing is deployed anywhere else. It then starts each installed runtime with that home as its working directory, so the runtime finds the deployed skills as project skills, gives it a fixed prompt naming one skill, and prints one fact per line. The script does every deterministic step; your part is choosing what to check and saying what a failure means. Its docstring lists every output line and what the run isolates: configuration, not credentials, so the runtimes use the user's own sign-ins, and each `RUNTIME` line's `KEPT` names what still loaded.

It is a manual gate, never part of `tests/run_validation.py`: each run calls a model, and a run can fail because the model chose not to run the command.

## 1. Choose the skills

The script names the shipped skills a change reaches and starts nothing. For a pull request:

```bash
python -B tools/runtime_canary.py --changed-since origin/main
```

For a release, give `--changed-since` no commit, so it reads from the last release, the highest `vX.Y.Z` tag `HEAD` holds ("Before tagging" in `docs/releasing.md`). Each `CHANGED <skill> "<why>"` line names a skill and the first changed path that reaches it, through a hidden dependency, a shared asset, an agent, or the adapter renderer where it says so; its docstring states the rules. `NOTHING_CHANGED` means the change needs no canary. The fixture always runs first. For a pull request, if more than three skills are named, ask the user which to keep: the cost is one model run per runtime per skill. For a release, run every named skill without asking, at most three to a run. Naming a hidden dependency, such as `code-review-core`, itself costs no model run and prints only its discovery lines and, per runtime, its `NOT_ATTEMPTED` and `MATRIX` lines.

## 2. Run it

From the worktree, with the chosen skills:

```bash
python -B tools/runtime_canary.py "<skill>" "<skill>"
```

Add `--runtime <name>` (repeatable) to check fewer runtimes, `--discovery-only` to list skills without running any model, `--codex-sandbox unelevated` to rerun Codex in the other Windows sandbox mode after an elevated run was `BLOCKED`, or `--keep-home` to keep the home's transcripts after a run that passed. The first line is `HOME "<dir>"`.

## 3. Say what each line means

One success is evidence; one failure is a question. Explain each failure from its line and, when needed, the `TRANSCRIPT` it names (its `.stderr.txt` sits beside it), before anyone changes code:

- `DEPLOYED <source id>` or `DEPLOY_FAILED <source id> "<log>"`: the canary deploys this checkout, then the fixture source, into the home, one line each. `DEPLOY_FAILED` stops the run with exit 1 before any runtime starts; the log file it names, under the home, holds the deployer's output and its reason, so read it first.
- `SKIPPED <runtime> "<reason>"`: the runtime is not installed where `deploy.py check` looks, so the canary neither lists nor runs it. On its own it fails nothing; only when every runtime asked for prints nothing else does `NOTHING_CHECKED` fail the run.
- `SUPPLIED <runtime> "<key>=<value>"`: a setting the canary passes in place of the configuration it switches off, such as the Windows sandbox mode Codex needs, which `--codex-sandbox` chooses. "Windows sandbox mode" in `docs/codex-support.md` explains it.
- `RAN "<script>" "<cwd>"`: the runtime ran the script from the canary's copy of the skill, or of a skill it declares in `skill_deps`. This is the pass, whatever the script exited with, since the model chooses its arguments. The installed copy never counts. The canary records Python and Bash scripts however the runtime started them, so a Bash script stopped at its usage check by `--help` still counts, but only a file the interpreter could open: a missing script, a folder, `python -c`, and `python -m` never count. `exit=<n>` follows when the recorder saw the script end (Python 3.12 or later, or a Bash script that sets no `EXIT` trap of its own), and `timed-out` when the session ran out of time after the script started. A non-zero exit is not a failure, but one that looks like a crash, such as a failed import or `exit=1` from a script that does not compile, is a defect in the deployed copy that validation cannot see; read the transcript and say so.
- `NOT_ATTEMPTED "<reason>"`: the skill declares `none` for that runtime in its `deploy-meta` `runtime_support`, so the canary started nothing.
- `UNSUPPORTED "copilot -p cannot start a user-only skill; ..."`: the skill's adapter lets only the user start it, and Copilot's headless mode cannot, so the canary spent no run on it. This is a known runtime limit, not a skill defect; "Headless sessions" in `docs/copilot-support.md` describes it and when to recheck it.
- `BLOCKED "<policy>"`: the runtime's own execution policy refused a command before any script ran, so the skill was never tested. A Codex policy that refuses every shell command, even reading the skill, shows up this way. The line names the Windows sandbox mode Codex ran with and points at "Windows sandbox mode" in `docs/codex-support.md`, which shows how to check the mode without a model. Report the line, not a skill defect.
- `FAILED "... denied: <tool>: <command>"`: Claude Code refused the command because the skill's `allowed-tools` does not grant it. Name the tool: on Windows the model may run the command through `PowerShell` rather than `Bash`.
- `FAILED "ran the installed copy <path>, not the canary copy"`: the runtime followed the copy under the real home. The change was not exercised; say so instead of blaming it.
- `FAILED "ran <path> instead of a script under <dir>"`: it ran a script from the wrong place, which is a path defect.
- `FAILED "ran <script> but it wrote no marker; ..."`: only the fixture says this. The path was right, but a sandbox stopped the write, or the read of the run's identifier the marker must carry; a marker line without it never counts.
- `FAILED "ran <script> but it exited with code <n>"`: only the fixture says this, since its arguments are known and it must exit 0. The path was right, so read the `.stderr.txt` for why the script stopped.
- `FAILED "cannot read what the run recorded in <file>: ..."`: the canary could not read its own marker or recorder file afterwards, so the run proves nothing either way. Check the file's permissions before rerunning.
- `FAILED "never ran a script from the skill"`: the transcript shows whether the runtime loaded the skill or the model declined, or whether the first script is in a language the canary does not record, which is anything but Python and Bash. Rerun once before treating it as a defect.
- `FAILED "timed out after <n> seconds before running a script from the skill"`: the session ran out of time before any script ran, as a hung runtime or a model still deciding does. Read the transcript for where it stopped, then rerun once, with a larger `--timeout` when it was still working.
- `FAILED "exited with code <n> before running a script from the skill"`: the runtime itself ended with that code before any script ran, as a runtime that could not start the session or stopped on an error does. Read the `.stderr.txt` and the transcript for its reason before rerunning once.
- `MATRIX <runtime> <skill> AGREES <level>` or `DISAGREES <level> "<what ran instead>"`: the run compared with the skill's declared runtime support, the table in "Runtime support" in `docs/skills.md`. `full` expects `RAN`; `partial` expects `UNSUPPORTED` where the runtime lacks `user-only-start` and `RAN` otherwise; `none` expects `NOT_ATTEMPTED`. Any `DISAGREES`, a skill that declares nothing, any `UNDISCOVERED` or `DISCOVERY_FAILED` line below, in a `--discovery-only` run too, or a `NOTHING_CHECKED` line makes the canary exit 1; `DISCOVERY_NOT_ATTEMPTED` and `PROJECT_COPY` never do. A disagreement on a rerun-able failure is the failure above; one that persists means the declaration or the skill is wrong, so say which.
- `DISCOVERED`, `UNDISCOVERED`, `SHADOWED`, and `DISABLED`: whether the runtime lists the canary's copy enabled, as `deploy.py verify` requires for `FOUND`; which other enabled copies of the same name it lists (`SHADOWED`); and which copies it lists turned off in its settings. A disabled canary copy prints `DISABLED` and `UNDISCOVERED`. Copilot's listing names one copy per name, the one it will use, so Copilot prints `SHADOWED` only when another copy wins over the canary's; a missing Copilot `SHADOWED` line does not mean no other copy exists, which its `KEPT` folders may still hold. Codex and Copilot list user-only skills too. Claude Code is listed only by a run, so `--discovery-only` prints no Claude lines.
- `DISCOVERY_FAILED <runtime> "<reason>"`: the canary could not read that runtime's skill list, so the run shows nothing about what it finds and fails; the reason says why, such as no answer in time, and every other line still prints.
- `DISCOVERY_NOT_ATTEMPTED "<reason>"`: the deployer gave the skill no runtime adapter, as it deploys a hidden dependency, so `deploy.py verify` expects no runtime to list it and the canary asks none to. The canary reads this from the home's manifest, so a skill whose recorded adapter a runtime does not list is still `UNDISCOVERED`.
- `PROJECT_COPY "<path>"`: the runtime lists the home's own deployed copy under `.claude/skills` as a project skill, because the canary starts it with the home as its working directory. Copilot does this for a skill with no adapter. No session started outside the home sees that copy, so it is not a shadow and not a deployment problem.
- `NOTHING_CHECKED "<reason>"`: no runtime was listed or run, so the run proves nothing and fails; the reason names why for each runtime, such as one `SKIPPED`, a listing that failed, Claude Code under `--discovery-only`, or a runtime asked for no skill it could list or start; rerun with what the reason says was missing.

## 4. Record the result

The run ends with a record block, `RECORD_BEGIN` to `RECORD_END`: each runtime's `VERSION`, then the lines a reader of the pull request needs, with the throwaway home written as `<temp>`, this checkout as `<source>`, and the profile folder as `<home>`. Paste it into the pull request's Validation section as printed, and add your reading of each failure below it.

After a run that passed, the script removes the home and prints `REMOVED`. After one that failed, or with `--keep-home`, it prints `HOME_KEPT` and leaves the home and its transcripts. Once they are read, delete it, unless the user wants them. The script deletes only a home it made, one holding the `.runtime-canary/home.json` marker each run writes naming it, and prints `REMOVED`, or `FAILED "<dir> holds no .runtime-canary/home.json naming it, ..."` for any other folder, which it keeps:

```bash
python -B tools/runtime_canary.py --remove-home "<dir>"
```