Dev Testing

---
name: dev-testing
description: Capsem testing policy and workflow. Use whenever running or writing tests. For VM, hypervisor, frontend, or Python specifics see the dev-testing-* skills.
---

# Testing

Read `tests/README.md` before adding or moving test fixtures. Test-only config
belongs under `tests/fixtures/`, not root `config/`.

## Where a new guard belongs

A test that records an architectural mistake -- something that must not be
repeated, checkable from source -- is a Citadel guard, in `tests/citadel/`, not
another file at `tests/` root. It runs in the fast phase, so it costs seconds
rather than waiting on an asset build.

Two rules the existing guards follow:

State the reason in a named `*_RATIONALE` appended to the assertion, so a
violation teaches. `test_db_boundary.py` is the model. Where the reason is
already stated canonically -- `config/gate.toml [boundary]`, AGENTS.md, a skill
-- cite it instead of restating it.

Write the adversarial case, not just the conformance case. A guard proved only
by inputs it was built for is not proved. `test_workflow_enforcement.py` carries
fourteen evasions and four legitimate shapes because enumerating bad spellings
let five through; the rule is now a whitelist for that reason.

## Test execution

Write code first with TDD and minimal focused feedback. During routine edits
run the regression being changed and only necessary file/behavior checks.
Broad workspace, contract, Citadel, VM, installed-package and performance runs
belong at major phase boundaries, not after every edit or commit. Pending full
proof must not block independent implementation. Qualify the complete phase
before calling it qualified, and retain the exact-source merge/release gates.
The bounded wrapper provides timeout and cleanup without a default machine
lease; reserve `--machine-lease` for explicitly isolated qualification work.

Finish a focused build/test command before editing its inputs. Cargo may have
already compiled a test when a later edit lands, so its green result can cover
older bytes. If inputs changed during compilation or execution, that result
does not verify the final source; rerun the focused regression after the edit.

SDK acceptance launchers use `uv run --frozen --no-sync` against the environment
prepared by the gate. Snapshot refresh changes source mtimes; an implicit sync
can rebuild the editable SDK and fetch its backend inside the sealed namespace.
Dependency preparation belongs to the earlier environment owner.

Linux managed-supervisor acceptance hands the private manager's runtime and PID
explicitly through the privilege transition. The worker checks its socket peer,
native owner and network namespace before restoring the manager environment.
Never adopt an ambient user manager outside the sealed worker's namespace:
its service can be healthy while localhost is unreachable from the test.
Pass that authenticated manager's runtime and bus address explicitly into the
service unit too. A user manager's default unit environment can point at a
different session; worker-only binding leaves a healthy service unable to
verify its own restart policy.

Focused Ruff commands must select `--config build_system/pyproject.toml`, as
the gate does. Auto-discovery from a root-level test used a different Python
version and reversed `tomllib` import grouping, passing locally then failing
the canonical source phase.

| Command | What | VM? |
|---------|------|-----|
| `just fast-test` | Incomplete source feedback only | No |
| `just focus-test <group>` | One existing owner; `rust` selects changed crates and reverse dependents | Depends on group |
| `just install` | Complete local macOS package build and native install for hands-on testing | Yes |
| `just test` | Complete local verification; required, passing, before a release | Yes |

A release normally needs passing `just test`; explicitly approved release `force`
may follow focused test-only fixes. Source guards, audits and hosted proof remain.
`just fast-test` and `just focus-test <group>` are broader feedback tools for a
phase boundary or a failure that needs their scope. During TDD use the smallest
native test directly; use complete verification at major phase boundaries.

Pick the narrowest lens, in this order: one test file run directly with pytest
in the latest `cache/worktrees/<run>` (it reuses that run's built assets; rebuild
and re-sign only what changed), then `focus-test <owner>` (`kingslanding` for
containers, `assets` to rebuild the guest), then `functional --slow` once on the
final head. Ask the user before adding a new owner.

The full gate is a construction boundary, not the edit loop. During TDD,
reproduce the failure with the smallest focused test, run that test red/green,
and batch adjacent parity fixes before paying for the complete gate. Run it
only when explicitly requested. Later fixes use focused owners; a gate-policy
edit is not permission for another full run. The one full run a change always
needs by default is the passing `just test` a release of it requires. Use the
documented operator-approved release override only when explicitly authorized.

`just test` deliberately accepts committed or uncommitted developer work. It
records `HEAD` plus a digest of every tracked and untracked non-ignored source
byte, then fails if either changes while the gate runs. Generated output stays
under ignored build directories. The local proof therefore covers the exact
source state the developer asked to test without forcing commit choreography.
Automatic gate benchmark output belongs under ignored
`cache/target/tests/benchmarks/`. Historical benchmark publication uses the owning
pytest/benchmark command and an explicit review; it is not a Just convenience
recipe.

## Release CI invariant

Release CI reuses the same checked-in private modules as local `just test`:

- `_test-fast`
- `_test-static`
- `_test-artifacts`
- `_test-functional`
- `_test-glowup`
- `_test-release-contracts`

The build scope is selective; the quality scope is not. Binary CI builds only
packages and resolves the channel's runtime by recorded digest. Runtime CI
builds the runtime for one channel and resolves the current package by recorded
digest. The resolved complementary artifacts are staged into the production
test harness, not replaced by source-built substitutes.

Every activated pairing must pass artifact validation, all VM suites,
Winterfell/MCP lifecycle, IronBank, injection, integration, benchmarks, full
`capsem-doctor`, exact native install, and update glow-up. A staged runtime
whose minimum package is not yet satisfied may run only the self-consistency,
integrity, isolation, and boot proof; the following binary lane must run the
complete functional and glow-up modules before activation.

Both macOS and Linux must install their exact publishable native packages,
including their real post-install scripts, before publication. Notarization and
the public stable-to-nightly switch/upgrade glow-up remain mandatory
end-to-end proof.

On Apple Silicon macOS, `just test` owns the pre-publication macOS package
boundary through `build_system/packaging/macos/macos_release_glowup.py`: it builds the package with the
production assembler, installs that exact file in a disposable headless Tart
guest, verifies the receipt, app bundle, complete binary cohort, service and
gateway health, then extracts the same package on the physical Mac and boots a
real Capsem guest VM from its exact binary/runtime payload to a shell marker.
The local package is unsigned; its postinstall ad-hoc signs the installed
Mach-O payload with the required entitlements. Local qualification must not
load Developer ID material or create a signing keychain. The tagged publication
workflow alone signs, notarizes, staples, and installs the final publishable
package.
Tart macOS guests do not support nested virtualization, so these are two
explicit halves of one script rather than a claimed nested proof. `just fast-test`
deliberately excludes Tart and therefore cannot be used for release.

Rust is pinned to `1.97.1` across the workspace file, workflow steps,
host-builder, and bootstrap. Change all pin surfaces together in a deliberate
toolchain-bump PR. RustSec and JavaScript bulk advisories are blocking in
`just fast-test`, local `just test`, ordinary CI, and both release lanes,
as well as the scheduled/manual audit. A new advisory fails the candidate
until it is remediated or explicitly reviewed in checked-in scanner policy.

Linux proof is host-aware: a cross-built non-host package receives structural
validation in qualification and exact native installation in its tagged release
job. `CAPSEM_REQUIRE_LINUX_DEB_PROOF=1` must not reject that non-host package
before the host package reaches its mandatory KVM proof. The hosted arm64 runner
does not expose `/dev/kvm`, so it proves exact package/service operation while
the x86_64 runner additionally owns the guest-shell marker.

Expensive harnesses need a cheap clean-environment bootstrap proof at the
start of `just test`, before Docker/Colima or artifact preparation. The one
private `_test-fast` module is also called by `just fast-test`, ordinary CI, and
both release lanes. It owns YAML/workflow and source syntax, source contracts,
dependency audits, Clippy, Python lint/type checks, and JavaScript/frontend
checks; no caller may reproduce a subset inline. Only a green fast gate may
build the Linux install-test image. That preflight must use a
container-owned `UV_PROJECT_ENVIRONMENT` and prove `python -m pytest` launches
before VM, package, or asset work consumes hours. Keep an ordering contract.
It is fail-fast infrastructure
validation only: the later Docker/systemd install E2E remains mandatory and
must still exercise the installed package and post-install behavior.
Use the configured localhost hostname for sealed install probes: Docker's
random hostname is absent from `/etc/hosts` with `network=none`, so sudo warns.

Copied macOS system executables need `sign_binary(path, relocated=True)` from
`tests/helpers/sign.py`. Their original Apple signature may verify strictly
yet still carry launch constraints that cause SIGKILL after relocation.
This mode replaces only ordinary fixture signatures; selected runtime,
release, and frozen-cohort artifacts must remain immutable consumers.

Seatbelt can silently strip set-id bits even when `chmod` succeeds. Verify a
fixture's real mode before blaming admission. Test mode decisions with concrete
inputs through the shared production predicate, keeping real filesystem tests
for both consumers; do not mock `Path.stat` or relax the sandbox to set a bit.

## Local/CI execution parity

Read `references/local-ci-parity.md` before editing any release workflow, gate
recipe, or CI job. It holds the Ironbank parity rule (every portable release
gate must be owned by `just test`), scanner/tool pinning, Docker platform and
prune discipline, common cache contracts, and the source-guard contracts.

## TDD workflow

Focused Ruff checks must include `--config build_system/pyproject.toml`, as
the canonical source gate does. Implicit configuration can classify imports
differently and produce changes that fail the actual repository policy.

Write tests first:
1. Write failing tests that capture expected behavior
2. Verify they fail for the right reason
3. Write minimal implementation to pass them
4. Refactor

Without a failing test first, tests can pass by accident. Cargo fixtures, including tiny probes, use `tests/helpers/bounded.py`: its `env` argument applies explicit probe compiler and target overrides after containment; never serialize inherited secrets. The wrapper owns the finite child timeout and cleanup; routine probes do not acquire a machine lease. Do not add an outer subprocess timeout.

## Functional slice proof matrix

Use the matrix below for qualification at each major phase boundary. During implementation run only the focused checks needed for the changed behavior and record outstanding phase proof. A green unit suite or a benchmark is not a substitute for complete phase qualification.

| Category | What it proves | Minimum expectation |
|----------|----------------|---------------------|
| Unit/contract | Pure logic, parser state machines, schema migration, helper APIs | Red/green tests for normal and edge behavior at the smallest useful boundary |
| Functional | The feature works through its production-facing API, not just private helpers | Exercise the real module boundary with realistic inputs and assert outputs plus side effects |
| Adversarial | The feature preserves security, privacy, and policy invariants when attacked | Malformed, oversized, denied, missing, racing, timeout, permission, and leak-prevention cases |
| E2E/VM | The user-visible path works in a real Capsem session | Boot/run a VM or use the black-box CLI/MCP/service path, then inspect externally visible behavior |
| Telemetry | Audit data is present, accurate, and queryable | Query `session.db` or logger readers for required rows, fields, decisions, errors, and attribution |
| Performance | Hot paths stayed inside the accepted budget | Benchmarks or timing assertions with recorded numbers and regression criteria |

If a category is genuinely impossible or deliberately deferred, record it as missing with a reason, owner, and follow-up task. Silent deferral is the bug. "Covered by later E2E" is not enough unless the tracker names the later test and the current milestone is explicitly scoped as internal-only.

For policy, MITM, MCP, telemetry, networking, filesystem, process lifecycle, or sandbox-boundary work, the functional slice matrix is mandatory. The tests should prove not only that the happy path succeeds, but also that enforcement happens at the intended boundary: a blocked MCP tool does not dispatch, a blocked return does not leak, a denied URL does not reach the network, a malformed frame does not poison the stream, and telemetry records the truth.

## Ironbank ledger tests

Use `/ironbank` for release-critical VM, network, model, MCP, credential
broker, package-manager, doctor, benchmark, and security acceptance proof.
Ironbank lives in `tests/ironbank/` and is full black-box: tests are written
from public contracts, CLI help, docs, generated schemas, hermetic fixtures,
route responses, logs, DB rows, and installed package metadata. Do not inspect
Rust/product internals to decide expected behavior.

Ironbank cannot use:
- `skip`, `skipif`, `slow`, optional markers, or public-network dependencies
- status-code-only replay
- row-exists checks
- parser-only assertions
- manual OAuth/client runs as release proof

One deterministic stimulus must prove the whole ledger path: client result,
parsed facts, CEL/security decision, detection/enforcement rows, protocol DB
rows, structured logs, status counters, UDS route, HTTP route, and UI-facing
JSON. Every emitted DB/log/route field is exact-value asserted, covered by a
typed invariant, or explicitly marked not applicable. Unknown fields fail the
test until the field ledger is updated.

Package-manager tests prove function. Installing `zstd`, for example, means
compressing known bytes, decompressing them, and comparing the exact output;
not just checking dpkg output.

## Logged-data DB ownership

Telemetry and security ledgers are database-owned. Service routes, UI handlers,
MCP helpers, and benchmark harnesses must not build their own logged-data
projection caches and must not open SQLite directly. They may own query intent
(for example the fields a route needs), but they call the logger DB object to
execute it. The logger DB object owns connection threads, `mem`/disk table
layout, write buffering, flush, reload-from-disk behavior, WAL tuning, and
future FTS5/search tables.

Do not move route-specific SQL into `DbWriter` or turn the DB layer into a pile
of route helper methods such as `stats_detail_payload()` just to hide SQL. The
boundary is execution and storage mechanics:

```rust
db.ready().await?;
db.query(sql, params).await?;
db.write(event).await?;
```

`db.write(event).await` means the DB object accepted the event into its
producer buffer. Tests that assert read-after-write rows must use the DB flush
barrier or shutdown/reopen. Do not paper over visibility with sleeps, route
projections, or direct SQLite readers.

Empty table means empty result. Missing table or column means the schema
contract is broken and must fail loudly; never add compatibility branches that
treat missing ledger shape as empty data.

Regression tests must guard the boundary. If a route needs ledger data, add a
test that proves the route uses the DB object and a source guard that rejects
raw `rusqlite` opens, direct `DbReader::open`, and service-owned projection
state in production route code. Add a companion guard that prevents
route-specific DB writer methods or missing-schema fallbacks from being
introduced.

## Mock server boundary

`crates/capsem-mock-server` is the single reusable local fixture server for
benchmarks, doctor, protocol recording/replay, gateway/integration tests, and
Ironbank. It owns mock protocol responses and deterministic local upstream
behavior. Tests may launch it through `build_system/scripts/test/mock_server.py`,
`tests/helpers/mock_server.py`, or `CAPSEM_MOCK_SERVER_BASE_URL`.

Do not add another local HTTP/MCP/OAuth/model mock server for a feature. Extend
the shared mock server and its fixtures instead, then assert the route through
the relevant black-box test.

## Parallel VM and process lifecycle tests

Read [parallel-lifecycle-tests.md](references/parallel-lifecycle-tests.md) before
changing VM concurrency, companion supervision, or Apple VZ lifecycle tests.

## Adversarial testing

Capsem is a security product. Every security-relevant feature needs tests that actively try to break invariants. Think like an attacker:
- Can a corp-blocked domain be snuck through another provider's list?
- Does an overlapping wildcard in allow+block always deny?
- Does malformed input (empty strings, unicode, huge payloads, invalid JSON) get rejected?
- Can path traversal escape the VirtioFS sandbox?
- Can a guest process modify its own binaries?

Stress-test boundary conditions. Write tests for the attacks you'd attempt yourself.

### Security invariants to verify in tests

When touching security-relevant code, check these invariants have test coverage:

| Invariant | What to test | Where |
|-----------|-------------|-------|
| VirtioFS share is `guest/` only | `session_dir/guest/` exists, symlinks resolve, host-only files (`session.db`, `serial.log`) are outside the share | `capsem-core::lib::tests` |
| UDS sockets are 0600 | After bind, verify permissions exclude other users | `capsem-process` |
| Process env is cleared | `env_clear()` called, only allowlisted vars passed | `capsem-service` spawn tests |
| No `process::exit` on guest I/O | Control channel close causes loop break, not exit | `capsem-process` |
| Sensitive logs are 0600 | `serial.log` created with restricted permissions | `capsem-process` |
| Gateway auth on all routes | Every route except `GET /` returns 401 without token | `capsem-gateway::auth::tests` |
| Auth rate limiting | 429 after threshold, resets after window | `capsem-gateway::auth::tests` |
| CORS rejects external origins | Only localhost/127.0.0.1/tauri allowed | `capsem-gateway::tests` |
| Body size limit | 413 for >10MB payloads | `capsem-gateway::proxy::tests` |
| VM ID validation | Path traversal (`../`), dots, spaces, null bytes rejected | `capsem-gateway::terminal::tests` |
| Rootfs read-only | runtime rootfs asset mounted ro, guest binaries 555 | `capsem-doctor` in-VM tests |
| Suspend reports errors | IPC failure and timeout both return 500, not silent success | `capsem-service` tests |

## Test fixture anti-pattern: masking races with polling

If all test fixtures wait/poll before asserting, the tests will never catch server-side race conditions. For every endpoint that talks to a VM socket, write at least one test that calls it IMMEDIATELY after provision (no `wait_exec_ready`, no `ready_vm` fixture). The server must handle readiness internally.

**Pattern to avoid** (masks the bug -- server never needs wait logic because client always waits):
```
fixture calls provision -> fixture polls wait_exec_ready -> test calls exec
```

**Required test pattern** (catches the bug -- if server doesn't wait, test fails):
```
test calls provision -> test immediately calls exec -> server handles wait
```

See `tests/capsem-service/test_svc_exec_ready.py` for the regression tests that enforce this.

### wait_exec_ready is a single call, not a loop

`wait_exec_ready` (in `tests/helpers/service.py`, `tests/helpers/mcp.py`, `tests/capsem-gateway/test_gw_e2e.py`) makes one exec call with the server-side timeout passed through. The server's `handle_exec` calls `wait_for_vm_ready` internally, which polls until the VM is ready. Do NOT add client-side retry loops -- that creates a double-wait where each retry can block for the full server timeout (30s client retries x 30s server wait = pathological cascade). One wait, one place.

For gateway readiness and selected native-binary provenance, read
[native gateway fixtures](references/native-gateway-fixtures.md).

### Exec latency regression gate

`tests/capsem-serial/test_boot_timing.py::test_exec_latency_within_gate` asserts that provision-to-first-exec completes within `EXEC_LATENCY_GATE`. If this test fails, investigate boot time (process.log boot_timeline spans), not the wait mechanism.

## Where tests live

- **Rust unit: sibling `tests.rs` file, not inline `mod tests { ... }`.** See the next subsection.
- Rust integration: `crates/capsem-core/tests/`
- In-VM diagnostics: `guest/artifacts/diagnostics/test_*.py` (see dev-testing-vm)
- Hypervisor: KVM + Apple VZ tests (see dev-testing-hypervisor)
- Frontend: `web/app/src/lib/__tests__/` (see dev-testing-frontend)
- Python (builder): `tests/test_*.py`
- Python integration (service daemon): `tests/capsem-*/` directories, each with its own conftest.py and pytest marker
- Ironbank release ledger: `tests/ironbank/` (black-box only; no Rust
  implementation-derived expectations)

### Rust unit tests: sibling `tests.rs` pattern

**Every Rust module keeps its unit tests in a sibling `tests.rs`, not an inline `mod tests { ... }` block.** The parent module declares:

```rust
// foo.rs  OR  foo/mod.rs
// ... production code ...

#[cfg(test)]
mod tests;
```

and the tests go in `tests.rs` in the same directory:

```rust
// tests.rs -- sibling of foo.rs or child of foo/
use super::*;

#[test]
fn roundtrip() { ... }
```

**Why.** Inline `#[cfg(test)] mod tests { ... }` blocks are appended at the bottom of prod files and commonly hit 50–99% of the file's line count. That means every Read, grep, and scroll to reach production code walks past thousands of test lines first. Several modules in this codebase hit 4,000+ lines that way before extraction. Agents and humans both read faster when prod code isn't buried.

**Mechanics.**
- `tests.rs` is a submodule of the parent file -- `use super::*;` works, private items are visible, `#[cfg(test)]` on the `mod tests;` declaration still gates compilation.
- For files that don't yet have a sibling directory (e.g. `lib.rs`, `foo.rs`), put `tests.rs` next to them in the same `src/` directory.
- For files that are already `foo/mod.rs`, put `tests.rs` inside `foo/`.
- Attributes on the inline `mod tests` block (e.g. `#[allow(unused_imports)]`) move onto the declaration: `#[cfg(test)]\n#[allow(unused_imports)]\nmod tests;`.

**Extraction recipe** (for any remaining inline `mod tests { ... }`):
1. Move the block body (everything between the outer `{` and `}`) into a new sibling `tests.rs`.
2. Dedent one indentation level so contents read as top-level items.
3. Replace the old inline block with `#[cfg(test)] mod tests;` (plus any attributes that were on the original).
4. `cargo test -p <crate>` -- should pass identically.

**When to push back.** If you see a new PR or agent output adding an inline `mod tests { ... }` block, request it be moved to `tests.rs` before merge. Exceptions are narrow: tiny helper modules under ~50 lines total where inline tests plus prod code fit on one screen, or a module that's already a test-only helper.

### Source contracts must read the sibling, not the production file

A Python contract asserting that some Rust test *exists* must read the sibling
`tests.rs`, never the production `.rs` the test moved out of. Use
`tests/rust_sources.py`:

```python
from rust_sources import production, sibling_tests

assert "pub enum Status" in production(RELEASE_GRAPH)          # prod symbol
assert "release_graph_enums_reject_unknown" in sibling_tests(RELEASE_GRAPH)
```

Keep the two sources **separate**. Several contracts assert a symbol is
*absent* from production (`"Removed" not in source`), and a test module
legitimately names the thing it proves is rejected -- concatenating them lets a
fixture falsify a claim about shipped code.

`sibling_tests()` resolves `mod tests;` the way Rust does (`foo.rs` →
`foo/tests.rs`; `main.rs`/`lib.rs`/`mod.rs` → `tests.rs` beside it) and raises
when the module is missing rather than passing on an empty string. The helper is
not named `tests_of` on purpose: pytest collects `test*`, so the obvious name
becomes a phantom failing test in every importer.

`tests/test_rust_test_name_assertions.py` enforces this repo-wide and fails in
seconds. It resolves each assertion's target through the AST, per function
scope, so a contract that legitimately names a relocated test while asserting it
against a test module or spec document is not flagged.

**Why it matters.** This layout change broke sixteen contracts under
`tests/capsem-release/`, then five more under `tests/capsem_install/` that run
only inside the Docker install gate -- invisible until forty minutes into a
release run. Nothing about the failure pointed at a moved function; it read as a
broken release.

## Integration test suites

All Python integration tests live under `tests/capsem-*/` and use pytest markers. Each suite has a dedicated `just` recipe.

| Suite | Directory | Marker | VM? | What it tests |
|-------|-----------|--------|-----|---------------|
| Service API | `capsem-service/` | `integration` | Yes | HTTP endpoints: provision, list, info, exec, logs, file I/O, delete |
| CLI | `capsem-cli/` | `integration` | Yes | CLI subcommands via subprocess |
| npm MCP | `ironbank/test_mcp_settings_ledger.py` | `ironbank` | Yes | Packed npm stdio -> authenticated gateway -> guest MCP ledger proof |
| Session DB | `capsem-session/` | `session` | Yes | Telemetry: net/model/tool/mcp/fs events |
| Isolation | `capsem-isolation/` | `isolation` | Yes | Multi-VM filesystem + network isolation |
| Security | `capsem-security/` | `security` | Yes | Binary perms, codesigning, asset integrity, env blocklist |
| Config | `capsem-config/` | `config` | Yes | Limits, resource bounds, hot-reload |
| Bootstrap | `capsem-bootstrap/` | `bootstrap` | No | Setup flow, dev tools, asset checks |
| Stress | `capsem-stress/` | `stress` | Yes | 5 concurrent VMs, rapid create/delete |
| Build chain | `capsem-build-chain/` | `build_chain` | Yes | cargo build -> codesign -> pack -> manifest -> boot |
| Guest | `capsem-guest/` | `guest` | Yes | Network, services, filesystem, env inside guest |
| Cleanup | `capsem-cleanup/` | `cleanup` | Yes | Process killed, socket removed, session dir removed |
| Codesign | `capsem-codesign/` | `codesign` | No | All binaries signed, entitlements present (FAIL not skip) |
| Serial | `capsem-serial/` | `serial` | Yes | Console logs, boot timing < 30s |
| Session lifecycle | `capsem-session-lifecycle/` | `session_lifecycle` | Yes | DB exists, schema, events, survives shutdown |
| Config runtime | `capsem-config-runtime/` | `config_runtime` | Yes | CPU/RAM applied in guest, blocked domains |
| Recipes | `capsem-recipes/` | `recipe` | No | just run-service, just doctor, cargo build |
| Recovery | `capsem-recovery/` | `recovery` | Yes | Stale socket/instances, orphaned process, double service |
| Rootfs artifacts | `capsem-rootfs-artifacts/` | `rootfs` | No | Artifact files, build context, doctor consistency |
| Session exhaustive | `capsem-session-exhaustive/` | `session_exhaustive` | Yes | Per-table data validation, cross-table FK integrity |
| Install | `capsem_install/` | `install` | No | Native package installer: layout, auto-launch, service install, manifest placement, update, uninstall, lifecycle, reinstall, error paths |

`just test` is the only public complete local verification. `just fast-test`
is incomplete feedback, `just focus-test` selects a closed existing owner, and
`just install` is the complete local product install. Do not add another
composite; use the owning native test for narrower diagnosis.

Public Just recipes, Capsem CLI command paths, and service HTTP method/path
pairs are exact approval-gated surfaces. Any change must pass
`tests/test_public_surface_contract.py` and requires explicit approval before
editing `config/public-surface.toml`.

## Test matrix and coverage

Read `references/test-matrix.md` for the per-crate Rust CI matrix, coverage
enforcement, and Python suite tiers. Rust workspace and per-crate floors are
owned only by `config/gate.toml`; the per-crate checker rejects stale headroom,
so meaningful coverage gains raise their ratchets in the same change.

## Fast debug with capsem MCP tools

Read `references/mcp-debug-tools.md` for interactive VM debugging through the
capsem MCP server: tool table, one-shot vs iterative workflows, and common
session-DB queries. MCP tools are for fast targeted checks; just recipes are
for complete validation at major phase boundaries and before merge or release.

## End-to-end validation is not optional

At the major phase boundary for changes touching guest binaries, network policy, telemetry, MCP, or VM lifecycle:

1. `just exec "capsem-doctor"` -- verifies sandbox integrity inside the VM
2. After telemetry/logging changes: run a real session and verify the exercised
   protocols through the typed timeline, history, statistics, and security
   routes. Use `check_session.py --db` only on a coherent, quiesced evidence
   copy after its ledger worker has stopped.

## When tests fail

Never dismiss a test failure as "pre-existing" or "unrelated." Every failure must be investigated. Follow the dev-debugging workflow:

1. **Do not change the test to make it pass.** The test is evidence. Changing the assertion to match broken behavior destroys that evidence.
2. **Reproduce and diagnose first.** Understand *why* it fails before writing any fix. See the dev-debugging skill for the full methodology: reproduce with a test, diagnose root cause, then fix comprehensively.
3. **Fix the code, not the test.** If the test is genuinely wrong (not the code), explain in detail why the test's expectation is incorrect before changing it.

### Measuring a gate's result

**Never take the last line of a multi-part result as the result.** Two shapes of
one mistake, both of which report success while the thing measured failed:

**`$?` after a pipe is the pipe's status.** `just test | tail` reports what
`tail` did. Redirect, then read the code separately:

```bash
just test > /tmp/gate.log 2>&1; echo "EXIT=$?"
```

**`tail -n1` across a multi-part result returns the last part, not the whole.**
`cargo test -p capsem-service` runs three test binaries; the last prints
`0 passed`, so `| tail -1` reads as though the crate had no tests while 91 and
264 passed above it. Aggregate instead of sampling:

```bash
cargo test -p capsem-service 2>&1 | grep -E "^test result:"   # every binary
```

Both errors are silent and both flatter you: one turns a failed gate into a
pass, the other turns a passing crate into a phantom regression. If a command
can emit more than one verdict, read them all.

Read the *first* real error, not the recipe cascade under it — `grep -aE "^FAILED|^E "` lands on the cause, while the trailing `error: Recipe ... failed` lines are only the unwind.

`build_system/tests/gate/test_exit_status_integrity.py` keeps this out of committed recipes,
scripts, and workflows, and requires `set -o pipefail` in any bash recipe that
pipes. It cannot see an agent's ad-hoc shell — that part is on you.

### Fixtures use the wrapper, never the raw variable

Redirect Capsem paths with `paths::CapsemPathsGuard::redirect(root)`. It sets
`CAPSEM_HOME`, `CAPSEM_RUN_DIR`, and `CAPSEM_ASSETS_DIR` from one root and
restores on drop, so a fixture cannot set one and inherit the rest.

Read logs with `telemetry::read_log_tail`, including in assertions: a test that
opens a `*.log` path directly stops exercising what the product does the moment
that stream rotates.

Both are enforced by `build_system/tests/gate/test_path_and_log_wrappers_are_mandatory.py`. See
`/dev-rust-patterns` "One rule, one function" for why.

### Verify with the gate's environment, not a bare shell

`just test` exports `CAPSEM_HOME`, `CAPSEM_RUN_DIR`, and
`CAPSEM_BENCHMARK_OUTPUT_ROOT`. A test that reads ambient state passes in your
shell and fails in the gate:

```bash
CAPSEM_HOME="$PWD/cache/target/tests/home/.capsem" \
CAPSEM_RUN_DIR="$PWD/cache/target/tests/home/.capsem/run" \
  cargo test -p <crate>
```

A fixture that overrides `CAPSEM_HOME` must override `CAPSEM_RUN_DIR` too —
the run dir takes precedence over the home-derived default, so setting only the
first sends production code to the ambient run directory while the fixture
writes into a temp one. Bisect by exporting one variable at a time; that names
the culprit in two runs instead of guessing.

### Cache limits belong in the cache contract

A number copied next to a rule drifts from it silently. Three separate gate
failures in one session traced to this: a coverage floor copied into a test, a
guest kernel major copied beside its pin, and a Docker fixture restating a
storage assumption outside the cache policy.

Each read as a broken product, and each surfaced minutes-to-an-hour into a gate rather than at the edit. Derive the value from its source, or name it once and pin config and contract together:

```python
maximum = load_policy(ROOT).runtimes["docker"].max_size_bytes
assert registry.contract("docker").max_size_bytes == maximum
```

Prove it derives rather than hardcodes: change the source value and confirm the test *follows* instead of breaking.
Read `/dev-cache` for the common cache model; tests must not recreate backend
accounting or machine-capacity rails.

## Source guards and testable design

See [source-guards-and-design.md](references/source-guards-and-design.md) for
platform gating, exact guard exclusions and separating pure decisions from runtime adapters.