Dev Rust Patterns

---
name: dev-rust-patterns
description: Rust patterns and hard-won lessons. Use when writing Rust in capsem-core, capsem-app, or capsem-agent: async/tokio, non-blocking I/O, cross-compilation.
---

# Rust Patterns

## Data paths: zero-copy first, no speculative hashing or JSON

Read [data-path performance](references/data-path-performance.md) before changing
transport, admission, serialization, hashing or buffer ownership. Prefer moving
or borrowing bytes and one bounded encoding into the final frame. Hashing and
JSON require a concrete, documented contract; neither is a default safety step.

## Async / non-blocking

Capsem uses tokio for all async I/O. The MITM proxy, vsock manager, and file monitor are all async.

Publish matching in-memory ownership inside the serialized blocking admission,
before releasing its I/O permit. A reader can observe the durable record before
the async caller resumes after `spawn_blocking`; publishing its ticket after
`.await` exposed valid managed-session claims to `recovery_pending`. The
controller regression pauses that continuation and claims through the real store.

### Never block the tokio runtime

Long-running synchronous work (FUSE request processing, disk I/O, compression) must run on a dedicated thread via `tokio::task::spawn_blocking` or a dedicated `std::thread`. Blocking inside a tokio task starves other tasks.

The VirtioFS FUSE server runs on its own thread for this reason -- FUSE ops are synchronous by nature (read, write, lookup) and can't be made async without significant complexity.

### Blocking-in-async anti-pattern (systemic -- audit, don't spot-fix)

Any code path that does blocking I/O inside an async function or while holding a `tokio::sync::Mutex` is a bug. This causes the tokio worker thread to stall, freezing the entire gateway, UI, or network stack until the blocking operation completes.

**What counts as blocking I/O:**
- `std::process::Command` (subprocess execution)
- `std::fs::*` (read, write, copy, remove_dir_all, create_dir_all)
- `walkdir::WalkDir` (directory traversal)
- `blake3::Hasher` on large data (hash computation)
- `std::thread::sleep`

**The fix pattern** -- in `capsem-service`, route handlers go through `ServiceState::off_worker` (`crates/capsem-service/src/blocking.rs`), and `crates/capsem-service/src/tests/async_io_contract.rs` refuses direct blocking calls. Elsewhere, the shape is:
```rust
let result = tokio::task::spawn_blocking(move || {
    let rt = tokio::runtime::Handle::current();
    rt.block_on(async {
        let mut guard = mutex.lock().await;
        sync_blocking_work(&mut guard)
    })
}).await.unwrap_or_else(|e| /* handle panic */);
```

**Example site:** the fork clone (`crates/capsem-service/src/vm_files/fork.rs`) runs `clone_sandbox_state` -- fsync, clonefile, and a full tree walk -- on the blocking pool. Any new tree copy, hash, or directory walk reachable from an async handler needs the same treatment.

### Channel patterns

- `tokio::sync::mpsc` for producer-consumer (vsock data flow, telemetry events)
- `tokio::sync::broadcast` for fan-out (serial output to multiple subscribers)
- `tokio::sync::oneshot` for single-response request-reply (control messages)

### Logger DB boundary

`capsem-logger` owns SQLite connection/thread/storage mechanics. Rust service,
gateway, MCP, UI, and benchmark code must not call `rusqlite::Connection::open`
or `DbReader::open` directly for telemetry/security ledgers, and must not add
service-owned projection caches.

Callers may own query intent, but the DB object owns execution:

```rust
db.ready().await?;
db.query(sql, params).await?;
db.write(event).await?;
```

`db.write(event).await` is an accept boundary: the DB-owned producer buffer
has taken responsibility for the event. Read-after-write tests must use the DB
flush barrier or shutdown/reopen before asserting route-visible rows. Never add
route sleeps, route projection caches, or caller-owned SQLite reads to force
visibility.

Do not hide route SQL by adding route-specific helpers to `DbWriter`; the DB
writer is not a product route registry. Put connection threads, `mem`/disk
tables, batching, flushing, WAL tuning, and future FTS5/search in
the DB layer. Empty tables are fine. Missing tables or columns are schema
contract failures and must not be converted into empty results.

### Coalescing buffer

Terminal output uses a `CoalesceBuffer` (8ms window, 64KB cap) to batch small vsock reads into larger writes. This prevents xterm.js from choking on thousands of tiny updates. The pattern: accumulate into a buffer, flush on timer or size threshold.

### Graceful shutdown

Use `tokio::select!` with a cancellation token or shutdown signal. Every long-running task must respect shutdown. Dangling tasks after VM exit cause resource leaks.

## Cross-compilation

Guest binaries target `aarch64-unknown-linux-musl` and `x86_64-unknown-linux-musl`. Key gotchas:

- **Platform-specific types**: `libc::ioctl` request param is `c_ulong` on macOS but `c_int` on Linux. Use `as _` to let the compiler infer the correct type.
- **Linker**: `.cargo/config.toml` sets `linker = "rust-lld"` for both musl targets.
- **No std dependencies**: musl builds are fully static. Avoid crates that link to system libraries.
- **Test on both**: `cargo check --target aarch64-unknown-linux-musl` catches cross-compile errors without needing to boot a VM.

## Error handling

- Use `anyhow::Result` for application code (capsem-app, scripts)
- Use `thiserror` for library errors in capsem-core (typed, matchable)
- Propagate errors up, don't swallow them. If a function returns `Result`, the caller must handle it.
- Log errors at the point where you have context, then propagate. Don't log AND propagate (causes duplicate log lines).

## Host Unix boundary

Reusable host Unix mechanics belong in the small modules under
`capsem_foundation::unix`; application crates consume their typed APIs instead
of calling `nix` or `libc` directly. This keeps ownership, EINTR handling,
errno classification, CLOEXEC, no-follow behavior, and race checks identical
across the CLI, service, process, guard, credentials, and core.

- Accept `BorrowedFd` and return `OwnedFd` or `File`. Never return a raw
  descriptor whose close responsibility is implicit, and never pair
  `ManuallyDrop<File>` with a manual `close`.
- Expected races are typed outcomes: a missing process and lock contention are
  not generic errors. Keep expected disconnects quiet; log unexpected failures
  once where the caller knows the operation, object identity, and errno.
- Use `unix::contained::ContainedDir` for guest-influenced paths. Its
  descriptor-relative traversal, constrained open options, and regular-file
  checks must remain in the same syscall sequence; do not put a path-based
  check in front of a separate path-based action.
- Use `unix::fs` for owner-only directories, no-follow regular-file access,
  filesystem capacity, atomic private publication, permission bits
  (`set_mode`), durability (`sync`, `sync_before_barrier`,
  `sync_filesystem`) and copy-on-write cloning (`clone_file_into`,
  `copy_sparse`). Use `unix::tree_clone` to copy a directory tree a guest can
  write. Use `unix::lock` for advisory locks, including synchronous
  read-modify-write sections.
- When foundation lacks a primitive, add it to foundation with its tests;
  never reach for `nix` or `libc` at the call site "just this once".
- Inside foundation, `nix` is the syscall layer and `unix::errno::io` converts
  its errors. A raw `libc::` call is allowed only where `nix` cannot express it
  (no wrapper, or a platform gap such as `MSG_NOSIGNAL` on macOS), and each
  one is listed with that reason in `tests/citadel/unix_boundary_debt.toml`.
- Put each primitive in its own module with tests in the sibling `tests.rs`.
  Cover errno preservation, ownership after the source is dropped, symlink and
  path-replacement races, special files, exact modes, and concurrent callers.

When transferring connected sockets with `unix::router_channel`, retain the
original socket until a request-bound receipt confirms that the receiver owns
and validated its descriptor. A completed `sendmsg` only queues SCM_RIGHTS;
Darwin may collect sockets with no owner outside that queue. Bound the receipt
wait, reject wrong identities, directions and extra descriptors, and preserve
scope cancellation and positive child reaping. The native MCP regression is
`upstream_broker/mcp_spawn/tests.rs`: it delays adoption, checks that the original
stays alive, and proves it is released after the receipt. Keeping an extra
socket for the child's entire lifetime masks disconnects and leaks descriptors.
The same rule covers startup policy, proxy traffic/MCP/trace hints, ledger
clients, gateway connections and the aggregator's owner connection. Auditing
only the native MCP bootstrap missed these paths and left a macOS policy-update
broken pipe. `tests/citadel/test_descriptor_adoption_lifetime.py` guards the
early-drop sites, including private cable handoff: its table reference can be
replaced by a reconnect before adoption. Gateway and upstream grant tests drop the received copy
before acknowledging to prove the sender still owns its original, then assert
EOF after adoption to rule out retention leaks.

Direct Unix calls remain valid only in modules that implement a kernel or
platform ABI rather than reusable host policy, such as KVM/FUSE and guest audit
plumbing. `tests/citadel/test_unix_boundary.py` and its exact domain-ABI
inventory enforce that distinction; extending the inventory requires explicit
review, not a convenience exemption. The same guard inventories every raw
`libc::` call and `extern "C"` binding inside foundation itself, rejects a new
one that `nix` already wraps, and catches aliased imports (`use ::libc as sys`),
`extern "C" { fn ... }` blocks and renamed or table-form manifest
dependencies. Before it did, foundation's own internals were unscanned and
raw calls with `nix` wrappers accumulated there unnoticed.

## Guest RAM and VirtioFS performance

Before changing guest RAM, KVM storage or OCI boot, read
[Guest RAM and VirtioFS performance](references/guest-memory-performance.md).
Keep RAM and KVM slot offsets aligned to 2 MiB and request transparent huge
pages through `capsem_foundation::unix::guest_memory::GuestRam`. A regular
4-KiB-backed mapping once made kernel startup take 1.1 seconds and a 352-MB
EROFS scan wait 1.5 seconds in reads. The allocation and slot regression is
`guest_ram_and_kvm_slot_offsets_preserve_huge_page_alignment` in
`hypervisor/kvm/memory/tests.rs`; alignment alone is insufficient without the
per-mapping huge-page hint.

## Bidirectional I/O -- thread per direction

When bridging two blocking file descriptors bidirectionally (e.g., TCP socket to vsock in `net_proxy.rs`, or master PTY to vsock in `capsem-pty-agent`), doing both reads and writes in a single thread using `poll(2)` causes deadlocks. If both outgoing buffers fill simultaneously, a single thread blocks on writing and stops reading, creating mutual lockup. Always spawn a dedicated thread for at least one direction (`std::thread::spawn` for `fd_b -> fd_a` while the main thread handles `fd_a -> fd_b`).

## Serde -- avoid `serde_json::Value` on LLM payloads

The MITM proxy and ai_traffic parsers handle massive HTTP payloads (megabytes of tool calls, histories, images). Parsing these into `serde_json::Value` does full DOM allocation, which is inefficient and risks memory exhaustion.

**Rules:**
- Define targeted structs with `#[derive(Deserialize)]`. Serde skips and discards fields not in the struct without allocating memory for them.
- For struct fields that hold large, unconstrained JSON (tool call arguments, function responses, full model outputs) and are only converted to strings: use `Box<serde_json::value::RawValue>` instead of `serde_json::Value`. `RawValue` keeps the JSON as an unparsed string slice -- zero DOM allocation. Access the raw JSON string via `.get()`.
- Never add `serde_json::Value` fields to structs that parse LLM request/response bodies. If you only need a string representation, use `RawValue`. If you need to traverse nested fields, use a typed struct.
- Remove unused fields from deserialization structs -- they still force Serde to allocate.

**Example -- before (bad):**
```rust
struct FunctionCall {
    name: Option<String>,
    args: Option<serde_json::Value>,  // full DOM parse of potentially huge args
}
// later: let arguments = fc.args.as_ref().map(|v| v.to_string());
```

**After (good):**
```rust
struct FunctionCall {
    name: Option<String>,
    args: Option<Box<serde_json::value::RawValue>>,  // zero-copy string slice
}
// later: let arguments = fc.args.as_ref().map(|v| v.get().to_owned());
```

## Memory and resource management

- **File handle limits**: VirtioFS caps at 4096 open file handles, returns `EMFILE` beyond that.
- **Read size limits**: VirtioFS clamps reads to 1MB, gather buffers to 2MB.
- **Safe deserialization**: `read_struct` returns `Option<T>` with bounds checks in all builds (not just debug).
- **irqfd for interrupt delivery**: Guest interrupt signaling uses `irqfd` to avoid cross-thread syscall overhead.

## Concurrency patterns

- **Shared runtime caches**: Use `capsem-cache` primitives instead of a local
  map and eviction policy. Import capacities and freshness windows from the
  generated `capsem_proto::CACHE_*` constants; their only numeric source is
  `[cache]` in `config/data/runtime-limits.toml`.
- **Arc for shared state**: VM state, proxy config, and telemetry handles are `Arc`-wrapped for sharing across tasks.
- **Per-connection tasks**: The MITM proxy spawns a new tokio task per connection. Each task owns its TLS state and upstream connection. No shared mutable state between connections.
- **Config-owned runtime ceilings**: Network concurrency and in-memory cache
  policy are owned by
  `config/data/runtime-limits.toml` and generated into `capsem-proto`. Import
  the shared protocol constant in host and guest consumers; never duplicate a
  numeric capacity in individual crates. Ledger queue and batch capacities
  derive from the same supported-client ceiling, and bounded coalescing waits
  use the generated duration rather than a local sleep literal. Extend the
  runtime-limit Citadel guard for every new consumer. See `/dev-mitm-proxy` for
  the load proof required when a limit changes.

### Host-serialization locks for per-host critical sections

When a service orchestrates N sibling child processes on a single host and some operations cannot safely run two-at-a-time on that host -- whether because of a framework constraint (Apple VZ save/restore) or because of shared-resource starvation (VZ teardown + WAL checkpoint + virtiofs drain all competing for main-thread and I/O bandwidth) -- park a `tokio::sync::Mutex<()>` on the service's shared state struct and acquire it at the top of the handler for the whole duration of the critical section. `Mutex<()>` isn't a weird construction: the unit value is the lock-token, the type signals "pure serialization, no protected payload". `Semaphore::new(1)` is equivalent -- pick one and stay consistent.

Current instances in `crates/capsem-service/src/main.rs`:

- **`save_restore_lock`**: serializes Apple VZ `saveMachineStateToURL` / `restoreMachineStateFromURL` across sibling VMs. Concurrent save/restore corrupts the VirtioFS ring state on the unlucky VM, surfaces as ext4-on-loop0 I/O errors after resume. Held through `handle_suspend` (IPC + child-exit wait) and `handle_resume` (spawn + `wait_for_vm_ready`). See `web/docs/src/content/docs/gotchas/concurrent-suspend-resume.md`.

- **`shutdown_lock`**: serializes VM teardown across `handle_delete` / `handle_stop` / `handle_purge` / `handle_run`. Without it, N concurrent deletes under load starve each other of the bandwidth each `capsem-process` needs to exit cleanly within the 1s fast-path budget; past the budget the service SIGKILLs mid-checkpoint and leaves a non-empty `session.db-wal`. Held through `shutdown_vm_process` for the whole `SIGTERM` + `wait_for_process_exit` window.

When to reach for this pattern:

- Symptom is "works solo, fails under concurrency on the same host."
- Root cause is a *per-host* resource, not per-VM: Apple VZ main thread, virtiofsd, DbWriter checkpoint, APFS fsync.
- Production runs exactly one service per host per user, so an in-process tokio mutex is enough -- no need for a file-lock or distributed primitive.

When NOT to reach for it:

- If the contention is per-VM (two handlers acting on the same VM), protect the VM entry in `instances: Mutex<HashMap<...>>` instead.
- If the "contention" is really a durability race (writer thread hasn't flushed), the right fix is usually the signal-handler explicit-cleanup pattern below, not another serialization lock.

### Signal-driven explicit cleanup for background-thread owners

Any long-running Rust process that owns background threads (SQLite writer, the fs-monitor scan loop, MCP aggregator subprocess, vsock relay) and runs under a bounded SIGTERM-to-SIGKILL budget must NOT rely on `Drop` + tokio-runtime-drop ordering to finish cleanup. On SIGTERM, hand owned resources to the signal handler and drain them synchronously BEFORE letting the main run loop return.

Symptom when this is missing: under concurrent teardowns on one host, the service SIGKILLs a child mid-checkpoint or mid-flush. Visible as `session.db-wal` left non-empty, missing `fs_events` rows, dangling aggregator subprocesses. Works solo, fails under `-n 4`.

Concrete primitives in this tree:

- **`DbWriter::shutdown_blocking(&self)`** — takes the stored mpsc sender, joins the writer thread, runs the final `PRAGMA wal_checkpoint(TRUNCATE)`. Arc-safe: other `Arc<DbWriter>` clones remain valid but their writes become no-ops. Idempotent. Drop delegates to it.
- **`FsMonitor::shutdown_and_join(&self)`** — sends on the shutdown channel so the scan loop runs one final scan and emission, then joins the thread. Bounded at two scans and two emissions. Must run BEFORE DbWriter shutdown, because fs_events fan into DbWriter.
- **`CAPSEM_TEST_SLOW_CHECKPOINT_MS`** — test-only env var in `writer_loop` that inserts a sleep before the final checkpoint. Use in tests that need to distinguish explicit cleanup from implicit runtime-drop ordering.

Canonical wiring in `crates/capsem-process/src/main.rs`:

```rust
struct Shutdown {
    db: Option<Arc<DbWriter>>,
    fs_monitor: Option<FsMonitor>,
}

impl Shutdown {
    fn drain_blocking(&mut self) {
        // fs_events fan into DbWriter -- flush fs_monitor first.
        if let Some(m) = self.fs_monitor.take() { m.shutdown_and_join(); }
        if let Some(db) = self.db.take() { db.shutdown_blocking(); }
    }
}

// Populate as owners are constructed:
shutdown.lock().await.db = Some(Arc::clone(&db));
shutdown.lock().await.fs_monitor = Some(monitor);

// Signal handler drains through spawn_blocking, then stops the run loop:
rt.spawn(async move {
    /* wait on SIGTERM/SIGINT */
    let mut owned = std::mem::take(&mut *shutdown.lock().await);
    let _ = tokio::task::spawn_blocking(move || owned.drain_blocking()).await;
    unsafe { core_foundation_sys::runloop::CFRunLoopStop(...); }
});
```

Key properties:

1. **Deterministic order.** The drain order is explicit (fs_monitor -> db), not "whatever reverse-declaration-order Drop happens to give us after tokio aborts tasks."
2. **Synchronous join.** The handler waits for each background thread to finish. No "hope the task finishes before the runtime drops."
3. **Run loop stops last.** `CFRunLoopStop` (macOS) fires only after drain returns. Main returns afterwards; the remaining tokio-runtime drop is now a no-op fast path because the heavy work already completed.
4. **Arc-safe shutdown APIs.** `shutdown_blocking(&self)` works through a shared `Arc<DbWriter>` — callers don't have to chase down every clone. Use `std::sync::Mutex<Option<Sender>>` internally; the hot-path `write()` clones the sender under the lock and releases it before `.await`.

When to reach for this pattern:

- The process has `std::thread::spawn` or `tokio::task::spawn_blocking` workers that run durability-critical work on shutdown (WAL checkpoint, queue flush, child-process wait).
- A parent sends SIGTERM then SIGKILLs after a short, fixed budget.
- Today's cleanup relies on Drop running inside tokio task abort — i.e., you can't draw a line between "cleanup finished" and "run loop exited."

Call out when NOT to use it:

- One-shot CLIs that exit on natural task completion (no run loop, no signal window).
- Workers whose only side effects are in-memory (no durability to lose).

When adding a new long-running process or a new background-thread owner, wire it through `Shutdown` from day one. Don't ship a new binary that "should be fine because Drop will run" — under load, Drop won't run in time.

## Logging

- `tracing` crate with `FmtSpan::CLOSE` for timing spans
- `RUST_LOG=capsem=debug` for full boot timing breakdown
- `RUST_LOG=capsem=info` for top-level only
- Use structured fields: `tracing::info!(domain = %domain, status = %code, "request completed")`

## One rule, one function

**A rule that lives in its callers is a rule that will be wrong in some of
them.** Resolution, precedence, and lifetime rules belong in a function; callers
call it.

This is the single most expensive pattern in this codebase's history. Two
instances, both found the same day:

| Rule | Copies | Outcome |
|---|---|---|
| "read the recent log" | 4 (`support_bundle`, `/service-logs`, `triage`, service session tails) | rotation landed, one copy learned, two silently returned nothing -- a daemon logging errors reported none |
| "point Capsem at a temp root" | every fixture, by hand | `CAPSEM_RUN_DIR`/`CAPSEM_ASSETS_DIR` outrank `CAPSEM_HOME`, so setting the home alone read the caller's directories: green locally, broken in the gate |

Both look locally correct in every copy. That is what makes them expensive:
nothing is visibly wrong at any one site, and a fix applied to one never
reaches the others.

**Established wrappers -- use them, do not re-derive:**

| Need | Call | Never |
|---|---|---|
| Host log content | `telemetry::read_log_tail(stream, max)` | `File::open`/`fs::read` on a `*.log` path |
| Session console/transcript | `session::read_session_serial_log` / `read_session_transcript` | Reopening a root-level log path or resolving worker-writable parents by path |
| Enumerate a stream | `telemetry::log_stream_files(stream)` | `read_dir` + name filtering |
| Redirect paths in a fixture | `paths::CapsemPathsGuard::redirect(root)` | `set_var("CAPSEM_HOME", ...)` |
| Any `~/.capsem/...` path | a `paths::` helper | `home.join(".capsem")` |
| Checkpoint marker | `paths::checkpoint_complete_path(cp)` | rebuilding `<name>.complete` |

`build_system/tests/gate/test_path_and_log_wrappers_are_mandatory.py` enforces the first three.

**When unifying, take the better implementation, not the first one.**
`support_bundle`'s tail reader seeked to each file's end; the shared one read
whole files. Guest console output is guest-controlled, so read-whole let a
chatty VM decide how much memory `capsem support` allocated. The shared
function now seeks.

**Two copies that agree today are still a bug.** `checkpoint_complete_path`
existed in capsem-process and capsem-service, identical except that one
hardcoded the fallback name and the other used a constant. Changing that
constant would have left the process writing a resume marker the service never
looked for, with nothing wrong at either site.

## Lessons learned

1. **Content-Encoding**: Always handle response decompression generically. Gzip compressed SSE responses caused NULL telemetry because the parser got binary garbage. Never strip Accept-Encoding as a workaround.

2. **Platform type widths**: `as _` is your friend for cross-platform libc calls. Explicit casts (`as c_ulong`) will fail on the other platform.

3. **Debouncer timing**: If a VM shuts down before debounced events flush, telemetry is lost. Add `sleep 1` in test commands, or use explicit flush on shutdown.

4. **VirtioFS whiteouts**: Apple VZ's VirtioFS doesn't support `mknod`, so overlayfs can't use it directly as upper. The ext4 loopback workaround provides full POSIX.

5. **setsid for controlling terminal**: Without `setsid`, the PTY has no foreground process group and Ctrl-C (SIGINT) is not delivered. `capsem-init` uses `setsid` to fix this.

6. **serde_json::Value on LLM hot path**: Three ai_traffic struct fields (`ResponseInfo.output`, `FunctionResponse.response`, `FunctionCall.args`) used `serde_json::Value` for large payloads that were only stringified. This forced full DOM allocation on every streaming request. Fixed by removing unused fields and switching to `Box<serde_json::value::RawValue>`.

7. **Prefer syscalls over subprocesses**: `std::process::Command` costs 5-30ms per spawn (fork/exec). If a syscall does the same thing, use it. Example: `cp -c -R` for APFS clonefile was 20-30ms; a direct clone syscall is <1ms. `capsem_foundation::unix::fs::clone_file_into` uses `fclonefileat` on macOS and the `FICLONE` ioctl on Linux -- no subprocess. Always check if the OS provides a syscall before reaching for `Command`.

7. **Blocking I/O in MCP tool handlers**: Tool handlers once ran blocking I/O (clonefile subprocess, walkdir, blake3) directly on tokio worker threads while holding a `tokio::sync::Mutex`, so tool calls hung from the model's perspective. Fixed by wrapping in `spawn_blocking` everywhere; see the anti-pattern section above.

7. **Clone guest-writable trees through descriptors, never paths**: Everything below `session_dir/guest/` is guest-controlled and may change mid-copy (an entry swapped for a symlink to a host path). Fork/create-from therefore go through `capsem_core::session::clone_sandbox_state`, which composes `capsem_foundation::unix::tree_clone`: descriptor-relative `O_NOFOLLOW` opens, exclusive destination creation, symlinks recreated but never followed, special files skipped, setuid/setgid/sticky bits dropped, and a non-regular `rootfs.img` refused. File contents clone from the open descriptor via `unix::fs::clone_file_into` (shared extents, or a sparse copy when the filesystem cannot share them). Never add a path-based `std::fs::copy`/`walkdir` copy of guest state. A CoW clone is metadata-only, so flush `rootfs.img` before cloning (`flush_system_overlay`) or the clone captures stale data.

8. **Platform-gate all macOS-only APIs**: Any code using macOS-only symbols (`libc::clonefile`, Apple framework bindings, etc.) must be wrapped in `#[cfg(target_os = "macos")]` -- both the struct/impl and the tests. The Linux app build (Tauri deb/AppImage) compiles the full workspace; ungated macOS symbols cause `cannot find function` errors on Linux CI. This burned v0.14.7: a clone helper used `libc::clonefile` without a cfg gate. Rule: when adding platform-specific code, gate the definition, the impl, and the tests.

9. **Readiness gates must reflect actual state**: `handle_ipc_connection` responded to Ping with Pong the moment the UDS socket existed -- before vsock connections, boot handshake, or command handler spawn. `wait_for_vm_ready` treated Pong as "ready", so exec commands were sent to a process that couldn't handle them yet, blocking silently in a channel until `setup_vsock` finished. Tests masked this with `wait_exec_ready()` client-side retry loops, creating a double-wait: 30 client retries x 30s server wait each. Fix: `Arc<AtomicBool>` (`vm_ready`) gated by `setup_vsock` after BootReady; IPC handler only sends Pong when the flag is set. One wait, one place -- the server waits; the client calls once. When adding any new IPC readiness check, never respond "ready" based on socket existence alone; check actual process state via a shared flag or state enum.

10. **VirtioFS and FSEvents**: Apple VZ VirtioFS guest writes bypass macOS FSEvents (the kernel's file notification subsystem), so `notify::RecommendedWatcher` silently drops guest-originated events and a host directory shared into a guest has to be polled. Do **not** reach for `notify::poll::PollWatcher` to do it: it walks with `follow_links(true)` hardcoded (`notify-7.0.0/src/poll.rs`) and its `configure` is a no-op, so one guest `ln -s / workspace/evil` turns every scan into a walk of the host's entire filesystem. Own the loop instead, as `crates/capsem-core/src/fs_monitor.rs` does: a `walkdir` walk with `follow_links(false)`, a snapshot of `(kind, len, mtime, ctime, ino)` per entry, and the difference between two walks as the events. Owning it also lets the cadence track the measured scan cost rather than a constant, and lets a symlink be recorded as a symlink instead of being descended.

11. **Process sandbox: env_clear() on child spawn**: When spawning a child process (e.g., capsem-process from service), always call `env_clear()` then re-add only the minimal env vars needed (`HOME`, `PATH`, `USER`, `TMPDIR`, `RUST_LOG`). The service's shell environment may contain API keys, tokens, or secrets that the child process has no business seeing. The guest's `--env` args are a separate injection path and are already validated.

12. **UDS socket permissions must be 0600**: After `UnixListener::bind()`, immediately `set_permissions(..., 0o600)`. The default umask leaves sockets world-accessible, meaning any local user can connect to a VM's IPC or terminal WebSocket with no auth. The gateway token file already does this; per-VM sockets must match.

13. **Never process::exit() on guest-controlled I/O**: A guest can close a vsock fd at any time. If the host handler calls `process::exit(1)` on read error, the guest has an unconditional DoS. Use `break` to exit the read loop and let the process shut down through normal channels.

14. **Sensitive logs cross a worker boundary**: Terminal and serial files contain raw user data. Use `unix::fs` single-link, no-follow file helpers and bound reads; a confined writer can still plant a redirect at its own log path. Below worker-writable directories, anchor access through `ContainedDir` descriptors. Prepare `session::prepare_session_logs` under the exclusive owner lock before confinement: it retains `owner/logs/` and migrates legacy generations without overwriting conflicts. Hypervisor console and terminal output share one `SharedLogWriter` and rotation counter; never open an independent appender. Writers retain directory descriptors across rotation, which needs directory authority rather than exact-file grants. Keep session root and ledger outside that grant. Create files with mode 0600 and repair permissions before confinement or through the authorized broker; confined writers must not call `set_permissions`. Session readers use the shared console/transcript APIs, falling back only for a missing layout. Support exports redact recognized credential bytes even in mixed binary/text lines, preserving unrelated bytes without a UTF-8 bypass. The PTY, telemetry, support-export and actual owner-sandbox regressions prove retention and malicious redirects.

15. **VirtioFS share boundary -- only guest/ subtree**: The VirtioFS share must point at `session_dir/guest/`, not `session_dir` itself. Host-only state (`session.db`, `owner/logs/`, `owner/checkpoint.vzsave`, `system/rootfs.img`) stays outside the share. The system disk lives in the host-only `system/` directory; never restore a compatibility link into guest storage. Put guest-visible content under `guest/`. Use `capsem_core::guest_share_dir(session_dir)` to get the share root.

16. **VM route id is not the display name**: `SandboxInfo.id`, route path ids,
`InstanceInfo.id`, DB handle keys, session directory basenames, and
`CAPSEM_VM_ID` must be the opaque VM/session id. Persistent registry `name`
values such as `co-work1` or `code-vm1` are display aliases only. User surfaces
may accept names, but they must translate to id before calling `/vms/{id}/...`.
Do not use `registry.get(&id)` in route code unless the route id has first been
resolved to the registry key; use the persistent identity helpers instead.
When returning UI/API payloads, preserve both `id` and `name`. Regression
coverage lives in `persistent_session_routes_keep_uuid_id_separate_from_display_name`,
`resume_sandbox_requires_uuid_route_id_not_display_name`, and
`persistent_route_identity_source_guard`.

17. **Use `capsem_core::poll::poll_until` for all async polling**: All "wait until ready" patterns must use the shared `poll_until` utility in `capsem-core/src/poll.rs`. It provides deadline-based timeout, exponential backoff, and structured tracing (attempt count, elapsed time, label). Never write ad-hoc `for _ in 0..N { sleep(X) }` or `while now < deadline { sleep(fixed) }` loops -- they lack logging, use fixed intervals instead of backoff, and hardcode timeouts. For sync code (guest agent), `vsock_connect_retry` in `vsock_io.rs` has the same pattern with `eprintln` logging. Every retry loop must have a total deadline.

18. **DRY wait patterns -- one wait, one place**: When a server endpoint already waits for a subprocess to become ready (e.g., `wait_for_vm_ready` in `handle_exec`), clients must not add their own retry loop on top. The test helper `wait_exec_ready` previously polled 30 times with 1s sleep, and each poll triggered a 30s server-side wait -- a 30x30s pathological cascade. After fixing the readiness gate (lesson 9), the client calls exec once with adequate HTTP timeout and the server handles the wait. Apply this DRY principle to any client/server readiness pattern: decide which layer owns the wait and make others pass through.

19. **Companions must not outlive their parent -- `kill_on_drop` is not enough**: `tokio::process::Command::kill_on_drop(true)` only fires when the parent's `Child` handle is dropped on graceful shutdown. Under SIGKILL/OOM/test-harness timeout/ pytest-xdist worker death, Drop never runs and companion processes (gateway, tray) get re-parented to PID 1 and survive forever. Every `just test -n 4` run leaked a fresh batch of orphans; accumulated orphans caused VM-ready poll spins, UDS-port collisions, and the suspend/resume regression. Defense in depth is mandatory for any spawned companion process, enforced on the COMPANION side so the parent can't get it wrong:
    - Pass `--parent-pid <spawner_pid>` when spawning.
    - Companion calls `capsem_guard::install(parent_pid, lock_path)?` at startup:
      - Refuses to run if parent PID is missing, dead, or not our actual `getppid()` (`parent_is_expected`). Exit 0 — standalone launches become silent no-ops.
      - Acquires an `flock(2)` singleton at `lock_path` (O_CLOEXEC opened atomically; process-local registry covers the brief fork-to-exec window where the flock fd can be inherited). Second instance exits 0.
      - Spawns a 100ms-interval watcher thread that calls `std::process::exit(0)` the moment `getppid()` no longer equals the declared parent PID. `getppid()` is immune to zombie state and flips to 1 on re-parenting, which is the reliable signal across SIGKILL, SIGSEGV, and OOM.
      - The mandatory terminator runs before any best-effort tracing. Parent death commonly closes a companion's stdout/stderr pipes; logging to that dead sink can panic the watcher thread, so observability must never stand between detecting reparenting and exiting.
    - Lock paths: tray is SYSTEM-WIDE (`$HOME/.capsem/run/tray.lock`) because the macOS menu bar is a shared global resource; gateway is per-run_dir because each test's gateway bridges a distinct UDS. Regression tests in `tests/capsem-service/test_companion_lifecycle.py` cover: refuse-standalone (no parent / wrong parent), singleton (double spawn, 20-way hammer), and die-with-parent (SIGKILL the parent, companion exits within 5s). When adding any new companion process, wire it through `capsem-guard` — don't invent a new pattern.

20. **Retry loops must classify errors, not time-bound a blanket wait**: When waiting for a resource to come up, the retry closure must distinguish *retryable* errors from *permanent* ones, and the classification depends on the caller's context, not the error itself. Identical `NotFound` / `ConnectionRefused` errors mean "service is down, give up" on an initial probe but "socket not bound yet, keep waiting" one call later in the post-launch retry. Pattern:
    - **Use `capsem_core::poll::poll_until`, not a hand-rolled backoff loop.** The poll primitive already gives you deadline, exponential backoff, per-attempt logging (label + elapsed + attempts), and a typed `TimedOut` error. Every new retry site that reinvents these is a future bug -- the `capsem doctor` "Service manager started capsem but socket not ready" bug existed only because `UdsClient::connect_with_timeout` hand-rolled its own loop and fast-failed on `ENOENT` before the just-started service had bound its socket.
    - **Inside the `poll_until` closure:** return `None` on retryable errors (`poll_until` keeps polling), return `Some(Err(...))` on permanent errors (`poll_until` exits immediately).
    - **Thread a small enum, not a `patient: bool`**, so every call site documents intent: `ConnectMode::FailFast` vs `ConnectMode::AwaitStartup`, `ProbeMode::Expected` vs `ProbeMode::MustBeRunning`. `crates/capsem/src/client.rs::UdsClient::connect_with_timeout` is the canonical example in the tree.
    - **Don't `.map_err(|_| anyhow!(...))` on the timeout branch.** You erase the inner cause. Chain with `Context` so the root error lives in the error chain and `{err:#}` prints both the summary and the underlying io::Error kind.

## Async reference

Read `references/rust-async-patterns.md` for comprehensive tokio patterns (tasks, channels, streams, error handling). From the community (6.4K installs).