Tart Ci

---
name: tart-ci
description: Stand up a fast, cached, isolated, disposable macOS CI lane on Tart — layered golden VM images, ephemeral per-job GitHub Actions runners, host-mounted caches, and a reusable per-repo vm-image manifest. Use when setting up VM-based CI for Pulp or generalizing it to another repo, building/refreshing golden images, wiring ephemeral runners, or debugging the VM lane.
requires:
  - tart            # brew install cirruslabs/cli/tart
  - sshpass         # brew install hudochenkov/sshpass/sshpass (first-boot key injection only)
  - gh              # authenticated; minting JIT runner configs needs repo admin
---

# Tart golden-VM CI lane

Run every macOS build/validation in a **throwaway VM cloned from a versioned golden image** so the host stays responsive and builds are reproducible. Generalizes to any repo via one `vm-image` manifest. Born from Pulp `planning/2026-06-01-macos-ci-isolation-plan.md`; the scripts live in `tools/ci/`.

## Why (the failure modes this fixes)
- **Build-dir churn → ODR heap corruption.** One `build/` reconfigured across branches/build-types mixes object layouts → `malloc: error for object 0x3f800000` (that's `1.0f` freed as a pointer) aborting in e.g. `Theme::~Theme`. Every job in a *pristine* clone makes this impossible.
- **Host-local validation is fragile + invasive.** Validating in the editing checkout inherits churn, pops GUI keychain dialogs, and competes for CPU. VMs are headless and disposable.
- **Spotlight/`fseventsd` storms** from build churn make the Mac unusable. VMs + `.metadata_never_index` keep the host calm.

## Core pieces (all in `tools/ci/`)
| Script | Role |
|---|---|
| `setup-ci-host.sh` | **One-command host onboarding** (opinionated): install tart/sshpass, set `~/VMs` (no FDA), acquire the golden (`--copy-from` or bake), install the launchd runner agent with a `--class` label. Mirrors `docs/guides/mac-ci-host-setup.md`. |
| `tart-provision.sh` | Build/refresh layered golden images; `verify`/`tag`/`resize`/`manifest` helpers. Subcommands: `base` → `apple-xcode` → `pulp` → `runner`. |
| `tart-runner.sh` | **Ephemeral per-job GitHub Actions runner.** Mints a JIT (single-job) runner config, clones the runner golden, boots with host ccache mounted, runs one job, destroys the VM. `--loop` boots a fresh VM only when there's queued work AND a free VM slot (`running_macos_vms < cap`); `--once` for a pilot; `--cap N` overrides the per-host cap. Registers under a **static** name per (host, slot) — `pulp-<class>-<NN>`, derived from the `pulp-build-<class>` label (override with `--name` / `--name-prefix` / `--slot` or `PULP_RUNNER_NAME[_PREFIX]` / `PULP_RUNNER_SLOT`). `--print-name` echoes the derived name and exits (pure; no gh/tart — what `test_tart_runner.py` asserts). |
| `tart-run-job.sh` | **Direct** ephemeral build (no GitHub runner): clone golden → virtio-fs mount host caches → build+ctest in-guest → discard. Useful for Shipyard `backend` / manual builds. |
| `pulp-worktree.sh` | Per-branch worktrees + shared ccache (host-side dev isolation; complements the VM lane). |
| `.shipyard/vm-image.toml` | **The per-repo reuse unit** (see below). |

## The vm-image manifest (the unit of reuse)
A new repo adds one `.shipyard/vm-image.toml` and the same `tart-provision.sh manifest <path>` bakes it — no hand-provisioning. Two strategies:
- `strategy = "bake"` — pre-bake a project image (fast clones for hot repos).
- `strategy = "configure-on-boot"` — clone the bare base + apply the manifest on boot (flexible/new repos).

Fields: `base`, `disk_gb`, `auto_login`, `[toolchain].xcode` (omit → no Xcode tier), `[brew].packages`, `[pip].packages`, `[caches].ccache_max`, `[[mounts]]`. See `.shipyard/vm-image.toml` (Pulp: Xcode+Skia) and `tools/ci/examples/vm-image.rust-repo.toml` (a light Rust profile, no Xcode — proves generalization).

## Verified base/runtime specifics (cirruslabs + Tart, 2026-06-01)
- Base image `ghcr.io/cirruslabs/macos-tahoe-base:latest` = macOS 26 "Tahoe" (matches Xcode 26.5 / build 17F42). Default creds `admin`/`admin`.
- The vanilla base **already** enables auto-login (kcpassword + `loginwindow autoLoginUser`) and Remote Login (sshd) — *verify*, don't recreate. Auto-login is REQUIRED for WindowServer/Metal.
- `tart set <vm> --disk-size N` only grows, only on a **stopped** VM; the bundled tart-guest-agent grows APFS on next boot.
- Prefer rsync'ing a **host-installed Xcode** into the golden over an in-guest `xcodes install` (interactive 2FA + multi-hour re-download). Remote rsync to `/Applications` needs `--rsync-path="sudo /opt/homebrew/bin/rsync"`.

## Caching strategy (the smart split)
- **Immutable, expensive-to-build deps → baked into the golden (CoW-shared, ~free per clone).** Skia + Dawn are prebuilt static libs (`libskia.a`, `libdawn_combined.a`, ~385 MB) baked at `~/pulp-skia-build`; `SKIA_DIR` points there. They are *never* recompiled.
- **Mutable, growing caches → host-mounted via virtio-fs.** ccache (warm across clones — measured cold→warm 0.6%→88%) and FetchContent sources. Keep `CCACHE_TEMPDIR` **in-guest** (cross-fs rename onto virtio-fs breaks ccache); `CCACHE_BASEDIR` normalizes paths; guest `admin` is uid 501 == host primary user so the shared ccache is writable both ways.

## GPU works in the guest (no hybrid lane needed)
Apple Virtualization provides Metal in the guest **even with `--no-graphics`**. Verified: `pulp-screenshot --backend skia` renders a real Skia/Metal PNG, `nm pulp-ui-preview | grep MacGpuWindowHost` = present. So the full mac lane (build + GPU + tests) runs in-VM.

## Dependency updates → golden re-bake (don't get stuck on old deps)
Skia/Dawn are pinned in `tools/deps/manifest.json` (release-asset URL + sha256 per platform; `tools/scripts/fetch_skia_for_release.py` consumes it). Single source of truth = the manifest pin.
- When deps bump (new `danielraffel/skia-builder` release, e.g. `chrome/m149` → newer), the Dependency Update Workflow bumps the manifest pin.
- **Then re-bake the golden** so its baked Skia matches: re-run the `pulp`/`runner` tiers (they should fetch per the *current* pin, not a stale copy), tag a new `:<date>`, refresh `:latest`. Ephemeral jobs clone `:latest` → get the new Skia. Keep the last 1–2 dated goldens per tier; prune older.
- Tie golden re-bakes to dependency-bump PRs and toolchain bumps (Xcode). Pinning Xcode + Skia in-image keeps font/raster goldens reproducible.

## Concurrency & hosts
- **macOS caps 2 concurrent running VMs PER HOST** (kernel quota; booting a 3rd throws "number of VMs exceeds the system limit"). For ≥3 concurrent, **distribute runners across multiple Macs** (e.g. Mac Studio + MacBook Pro M5 → 2+2 = 4) — each runs `tart-runner.sh`; new hosts inherit the host-class label (`*-studio`, `*-m1`, `*-m5`) and cap=2. A dedicated Studio *can* raise the cap via the kernel-quota override (plan Appendix D; SIP off + dev kernel — last resort).
- A persistent operator VM (e.g. `pulp-vm`) on a host consumes 1 of its 2 slots.
- **Capacity-aware cloud→local queue draining is implemented and VM-slot-aware.** Two cooperating pieces share one rule — a host has free macOS capacity when `running_macos_vms < cap` (cap = 2/host, Linux/Windows guests don't count): (1) `tart-runner.sh --loop` boots a VM only when there's queued `Build and Test` work AND a free slot (`PULP_VM_CAP`, default 2; `--cap N` to override); (2) `tools/scripts/macos_reroute_watcher.py`'s `free_macos_slots(hosts)` sums free slots across hosts (`--hosts-config` JSON, `tart list` locally or over SSH) and reclaims a still-queued cloud job into a freed slot. Default (no hosts-config) = a single local bare-metal slot, i.e. the pre-#3299 single-runner behavior — safe to run before the cutover. The two never double-book a host because they evaluate the same `running_macos_vms < cap` predicate.

## Linux + Windows pool runners (join the Actions pool like macOS)

Each platform serves the GitHub Actions pool via its own ephemeral per-job
runner supervisor — the analog of `tart-runner.sh` for macOS:

| Supervisor | VM | Golden | Labels (pilot) | LaunchAgent |
|---|---|---|---|---|
| `tools/ci/tart-runner.sh` | Tart macOS | `pulp-build-runner` | `…,pulp-build` | `com.danielraffel.pulp.tart-runner` |
| `tools/ci/tart-runner-linux.sh` | Tart Linux | `pulp-linux-build` | `…,Linux,ARM64,pulp-build-linux` | `com.danielraffel.pulp.tart-runner-linux` |
| `tools/ci/qemu-runner-windows.sh` | QEMU Windows | `pulp-windows-build-*.qcow2` | `…,Windows,ARM64,pulp-build-windows` | `com.danielraffel.pulp.qemu-runner-windows` |

All three: mint a JIT (single-job) runner config → boot a throwaway clone
(Tart CoW for Linux, qcow2 overlay on a dynamic SSH port for Windows) → run the
baked `~/actions-runner` agent once → discard. The goldens carry the
`actions-runner-{linux-arm64,win-arm64}` agent (Windows install-if-missing if a
golden predates the bake). `--loop` only boots when there's queued
`Build and Test` work.

**Per-platform opt-in/out** is the Shipyard macOS GUI's "Serve CI builds from
this Mac" switch: each lane is a `CIServingLane` toggled by `launchctl
load/unload` of its LaunchAgent (the labels above). Install a lane on a host by
sed-templating its `tools/launchd/*.plist.template` into `~/Library/LaunchAgents`
(replace `$PULP_REPO`, `$HOME`, and `$TART_HOME`/`$TARTCI_GOLDENS` — launchd
doesn't expand shell vars). Pulp CI routes to these via `build.yml`'s opt-in
`PULP_LOCAL_{LINUX,WINDOWS}_RUNS_ON_JSON` repo vars (default off → github-hosted).

**Hard-won Windows-runner gotchas:**
- The multi-KB JIT blob must NEVER ride a command line — through the
  ssh→cmd.exe→powershell chain it blows cmd's 8191-char limit ("The command line
  is too long"), whether passed as a `run.cmd --jitconfig` arg OR embedded in a
  `powershell -EncodedCommand`. Stream it into a file via **ssh stdin**, then run
  `Runner.Listener.exe run --jitconfig (Get-Content jit.cfg)`.
- `git reset --hard` (+ `core.autocrlf false`) for checkout — the golden's tree
  carries autocrlf churn that aborts a plain `git checkout`.

## Shipping FROM a VM-only runner host
A VM-only host (no host-side cmake/Xcode/Skia — builds only ever run *inside*
the VMs) can serve the pool fine, but `shipyard pr` initiated **on** it needs
care: the default `[targets.mac]` is `backend = "local"` (build on the host) and
fails there ("cmake/git-lfs not found"). Don't reach for `backend = "ssh"` to a
build box either — `auval` (AU validation) needs a real login/audio session to
register the component, so it fails "didn't find the component" over a headless
`ssh host cmd` (compile + the rest of ctest pass; only the auval tests fail).
The lane where auval works is one *with* a session: the self-hosted pool's
**auto-login VMs** (`auto_login = true` in the manifest — this is why) or a
cloud macOS runner. So on a VM-only host set, in gitignored `.shipyard.local`:
```toml
[targets.mac]
backend  = "cloud"
workflow = "build"   # dispatch build.yml; resolve-provider routes mac local-first
platform = "macos-arm64"
```
`shipyard pr` then dispatches `build.yml`; its `resolve-provider` sends the mac
leg to the self-hosted pool's auto-login VM (auval green), with cloud overflow.
Flip a lane mid-flight with `shipyard cloud retarget --target mac --provider <p>`.
(SSH build hosts that you *do* want to drive headless need brew on the non-login
PATH — `eval "$(brew shellenv)"` in `~/.zshenv`, not just `~/.zprofile` — or
`ssh host cmd` won't find cmake.)

## Diagnosing a red macOS leg (read the runner's LOCAL logs — `gh` is opaque)

On a self-hosted macOS runner, `gh run view --log` / `--job` returns **nothing
useful** for the build/test step — only "Process completed with exit code N"
(exit 8 = ctest had test failures), and check-run annotations are empty too. You
**cannot** tell which test failed from GitHub's API. But the runner persists
everything locally, so **if you're on the runner host** (true for Pulp's Mac
Studio — the deps here are the self-hosted Tart/Shipyard pool), read it directly:

1. **Find the work dir** from the runner config — it is NOT `~/actions-runner-*/_work`:
   ```bash
   # workFolder is custom on Pulp's runners:
   grep workFolder ~/actions-runner-pulp-studio-01/.runner
   #   → "/Volumes/Workshop/ci/pulp/work/pulp-studio-01"
   ```
2. **Read ctest's own result logs** in the persisted build dir (`clean:false` on
   self-hosted keeps `build-<key>` warm across runs; `build-macos` is the macOS
   leg). These are the authoritative source for *which* test failed and *why*:
   ```bash
   WS=/Volumes/Workshop/ci/pulp/work/pulp-studio-01/pulp/pulp   # <workFolder>/<repo>/<repo>
   cat "$WS/build-macos/Testing/Temporary/LastTestsFailed.log"  # failed test names (+ ctest index)
   grep -aA25 '<test name>' "$WS/build-macos/Testing/Temporary/LastTest.log"  # the failing REQUIRE + expansion
   ```
   `LastTestsFailed.log` lines look like `7675:pulp-import-design reports help…`;
   `LastTest.log` has the full Catch2 output (`file:line: FAILED:` + `with
   expansion:`). Check all three runners (`pulp-studio-01/02/03`) — the leg runs
   on whichever was free; the freshest `LastTestsFailed.log` mtime is the run.
3. The **step command/env** (not its output) lives in
   `~/actions-runner-<name>/_diag/Worker_*.log` (most-recent file = most-recent
   job; match by the commit SHA, which the checkout echoes). Read step *output*
   from the ctest logs in (2), not from the Worker log.

A "red macOS leg with exit code 8" on a change that can't plausibly affect macOS
is usually a **pre-existing failure on `main`** that the (flaky/overflowing) gate
let slip — e.g. a stale exact-match CLI assertion. Read the ctest log, fix the
real failing test, don't chase your own diff. (Found 2026-06-03: #3386's
`--emit swiftui` broke two `test_import_design_tool.cpp` exact-match asserts on
main, reddening every PR's macOS gate.)

## Rollout: pilot → graduate
1. **Additive pilot (safe):** run `tart-runner.sh --once` with a **non-required** label (`pulp-build-vm`). Trigger a real job without touching required routing: `gh workflow run build.yml -f macos_runner_selector_json='["self-hosted","pulp-build-vm"]'`. Confirm green.
2. **Graduate:** add the runner to the required `pulp-build` pool (or stage replacing bare-metal), distributed across hosts. Never point a required check at an empty label; preflight that the label has online runners.

## Gotchas (hard-won)
- **NEVER run signing/keychain tests on a non-VM host.** `check_notarization`/codesign tests call `security`/`codesign`/`notarytool`, which on an interactive host pop GUI keychain dialogs and can disrupt the default keychain. Run them **only in the disposable VM**. Never click "Reset To Defaults" on a keychain prompt on a real Mac; never wipe a host keychain.
- **Gatekeeper disabled in the CI base:** the cirruslabs base ships `spctl --master-disable`, so `spctl --assess` returns 0 for *any* path (even nonexistent). `check_notarization` was hardened with an `fs::exists` short-circuit so it's correct in both environments (see the `ship` skill).
- **Clean the build dir on build-type flips.** Shipyard `backend=local` reconfiguring Debug over a Release `build/` reproduces the ODR churn → false test failures. `rm -rf build` first, or (better) validate in the VM, not the editing checkout.
- **Disk: sparse + CoW.** Each `disk.img` is a sparse 150 GB file (~45 GB real); `du`/Finder show apparent size (N×150 GB) but `df` shows the truth (CoW clones share blocks — e.g. 13 VMs ≈ 313 GB real). Don't panic at apparent size; prune redundant bare working VMs (tags retain shared blocks).
- **First key injection needs `sshpass`** (password auth once); afterward everything uses the injected `id_ed25519`. `tart ip` can take ~10–120 s after boot — poll, don't fixed-sleep.
- **A missing `--dir` mount target reads as a fake "no IP".** `tart-runner.sh` boots each VM with `--dir="ccache:$PULP_CI_CACHE/ccache"` (default `$HOME/.cache/pulp-ci/ccache`). If that host dir doesn't exist — the common case on a **fresh CI host** — `tart run` exits *immediately* with `VZErrorDomain Code=2 "directory sharing device configuration is invalid"`, so the VM never boots and the runner times out 120 s later reporting **"no IP"** — pointing you at networking when the real cause is a missing directory. `setup-ci-host.sh` now pre-creates it and `tart-runner.sh` both `mkdir -p`s it and prints the `tart run` boot log on failure. If you ever see "no IP", read the boot log it now emits before suspecting vmnet/DHCP. (Diagnosed on the blackbook/M5 bring-up, 2026-06-01: a no-`--dir` boot got an IP in 0 s while a `--dir`-to-missing-path boot died instantly — the mount, not the network.)
- **Use shipyard (its own higher-quota auth) over raw `gh`** for GitHub ops to avoid rate limits; `gh`'s token lives in the login keychain (or `~/.config/gh/hosts.yml` with config storage).
- **Persistent runner via launchd: Full Disk Access + absolute paths.** Run the supervisor as a LaunchAgent (`tools/launchd/pulp-tart-runner.plist.template`) so it survives reboot. Two traps: (1) launchd does NOT expand `$HOME`/`$PULP_REPO` in plist values — the install `sed` must write absolute paths (a literal `$HOME` log path → exit 78); (2) a LaunchAgent can't read a `/Volumes` external VM store without **Full Disk Access** (exit 126 "Operation not permitted") — grant it in System Settings → Privacy & Security → Full Disk Access. The interactive shell has this access; the agent does not.
- **Every per-job step in a `set -euo pipefail` supervisor must clean up on its own failure.** The runner supervisors (`tools/ci/qemu-runner-windows.sh`, `tart-runner-linux.sh`) boot a VM/overlay, then run several SSH/QEMU steps. Any *unguarded* command that can fail (a dropped SSH, a PowerShell error streaming the JIT blob to a file) will abort the whole script under `set -e` **before** the trailing `kill "$qpid"; rm -rf "$jobdir"` cleanup — leaking a live QEMU process + overlay dir that a launchd `--loop` runner then trips over on KeepAlive restart. Mirror the surrounding steps: append `|| { note …; kill "$qpid" 2>/dev/null||true; rm -rf "$jobdir"; return 1; }` to each fallible per-job command, including pipelines (the JIT-config stdin→file upload was the one that slipped through). Caught by Codex review on tartci#10, fixed in both the tartci port and this original.
- **A `--loop` supervisor must tear down its in-flight VM on SIGTERM, not only after each job.** Per-job cleanup runs when the agent exits normally, but `launchctl unload` (the Shipyard GUI "Serve CI builds from this Mac" toggle OFF) sends **SIGTERM** mid-job or mid-wait — and a warm JIT runner can sit with a VM up for hours waiting for a job. Without a trap, stopping reclaims RAM (launchd kills the `tart run` child) but orphans a **stopped CoW clone** on disk and leaves the runner **registered-but-offline** on GitHub. Track the live VM at script scope (`CURRENT_VM`/`CURRENT_RPID`) and `trap 'discard_current_vm; trap - EXIT; exit 143' INT TERM` + `trap discard_current_vm EXIT`, clearing the vars in each normal teardown path so the EXIT trap no-ops. **The teardown must be FAST** — launchd SIGKILLs the supervisor shortly after SIGTERM, so a graceful `tart stop` + `sleep` can be cut short and leave a *stopped* clone behind (RAM freed, but `tart delete` never runs — observed live on the M5). Hard-`kill -9` the `tart run` host PID (ends the VM at once), then `tart delete` immediately — no `sleep`. `tart-runner.sh` does this; mirror it in the Linux/Windows pool runners when they grow a GUI toggle.

- **Runner names are STATIC per (host, slot) — reclaim before reuse.** The supervisor registers as `pulp-<class>-<NN>` (e.g. `pulp-m5-01`), not the old `ephr-<pid>-<counter>` churn, so the same physical Mac is recognizable in the Settings → Actions → Runners list and matches the bare-metal `pulp-studio-01` convention. A static name is only reusable if you clear the prior identity first: a SIGKILL'd supervisor / errored job / crashed clone can leave a stale GitHub registration (shows **Offline**) *and/or* a stopped Tart clone of the same name — and `generate-jitconfig` rejects a duplicate name while `tart clone` rejects an existing VM. `reclaim_runner_name` (runs before every mint) deletes both, best-effort — the JIT-lane equivalent of bare-metal `config.sh --replace`. The class comes from the `pulp-build-<class>` label `setup-ci-host.sh --class <name>` writes into the plist, so no plist edit is needed for the name. **Two supervisors on one host** (to use the 2-VM cap) must run with **distinct `--slot`** (→ `-01`/`-02`) or their static names collide. Seeing a lone **Offline** `pulp-<class>-NN` row alongside an Idle one is normal — it's a leftover the next reclaim clears.

## Store & hygiene
`TART_HOME=/Volumes/Workshop/VMs` (Spotlight-excluded via `.metadata_never_index`). Tag goldens `:<date>` + roll `:latest`. Ephemeral job VMs are deleted after use; confirm cleanup (`tart delete` fails silently on a *running* VM — stop → delete → verify). Reclaim with `tart-provision.sh list` + prune.