Manual E2e Qa

---
name: manual-e2e-qa
description: Manual end-to-end QA playbook for decode against a real provider — what to type at each surface (chat, gated read/write, bash, todo, web_fetch, ask_user, lsp, agent subagents, docker/modal sandboxing, headless `decode run`, model override, a benchmark trial, the local Kitaru server, session recording, worker replay, mid-turn steer/abort, persistence/memory, /ship, sandbox git token) and what "working" looks like. Use when manually testing or verifying a decode surface end-to-end.
---

# Testing E2E

Automated M1 proof: capstone [`tests/integration/test_milestone1_capstone.py`](../../../tests/integration/test_milestone1_capstone.py) — six-step conversation (read → gated write approve → gated write deny → todo_write → ask_user → web_fetch) through **real** `build_agent()` + `Runner` + `render_event` + session log + memory write-back; only network boundary swapped (`FunctionModel` for model, `httpx.MockTransport` for web tool). Run: `make integration-tests` or `make ci`. No API key, no network.

Below: **manual** e2e vs real Gemini — exercise each surface, then try to break it (adversarial half = Tester's job).

**Launch** (one env var, no service):

```bash
export GEMINI_API_KEY=…        # the only required secret (see .env.example); or put it in .env
uv run decode                  # the REPL: a "> " prompt + a footer hint render
```

No `GEMINI_API_KEY` → one friendly stderr line — `decode: set GEMINI_API_KEY in your environment or .env to start (see .env.example).` — exit non-zero, no traceback (task-004 guard in `cli.py`).

Per surface: what to type + what "working" looks like.

| Surface | Type this | Working looks like |
|---|---|---|
| **Plain chat** | `what can you do?` | answer **streams** token-by-token above prompt; prompt pinned at bottom. |
| **Read (gated)** | `read pyproject.toml` | `permission? read …` prompt; `y` → panel with numbered file contents; `n` → model told denied, adapts. |
| **Write (gated, approve)** | `create a file hello.txt that says hi` | `permission? write …` → `y` → file **appears** on disk (`cat hello.txt`), model confirms. |
| **Write (gated, deny)** | repeat write, answer `n` | file **not** created (`ls hello.txt` → absent); model told denied (doesn't pretend it wrote). |
| **Bash** | `run the tests with make unit-tests` | `permission? bash …` → `y` → panel with stdout/stderr (truncated past cap); runaway command bounded by `bash_timeout_s`. |
| **Todo checklist** | `make a 3-step plan to add a CLI flag and track it` | blue **tasks** panel renders checklist (`[ ]` / `[~]` / `[x]`), re-renders as statuses update. |
| **web_fetch** | `fetch https://example.com and summarize it` | `permission? web_fetch …` → `y` → page as Markdown (HTML stripped), model summarizes. |
| **ask_user** | `deploy my app` (underspecified) | model calls `ask_user`; `ask: …` question + `type your answer:` cue; next typed line **is** the answer, turn resumes. |
| **lsp (code intelligence)** | `where is build_agent defined?` | model calls `lsp` (`definition`); **auto-allows** (read-only, no prompt); answer cites `src/decode/agent/factory.py:68:5`. Then `write a broken bad.py with a syntax error` → approved write's result carries appended `LSP diagnostics (ty) — fix these:` block; model corrects. |
| **agent (Explore subagents)** | `explore how permissions and sandboxing each work across the repo, in parallel` | **ONE** `agent(prompts=[…])` call — one prompt per angle — **auto-allows** (READ_ONLY — "can only cause reads"); result panel = ONE **labelled aggregate**, a `## Subagent i — "<prompt>"` section per child, in prompt order (no synthesis LLM call — the parent model synthesizes). Child = read-only **Explore subagent** (`read`/`glob`/`grep`/`lsp` only — no `write`/`edit`/`bash`/`web_fetch`). The **harness** guarantees the parallelism (`asyncio.gather` inside the tool, not the model volunteering N calls): width cap **6 prompts** per call (`ModelRetry` → "consolidate" above it; empty list → `ModelRetry`), concurrency capped `subagent_max_parallel` (default 4 — a 6-wide fan-out runs 4, then 2). Child bounded `subagent_max_requests` (default 25); reports **share** the fold budget — each truncated to `subagent_result_max_bytes // len(prompts)` (default 16000 total, so the fold costs the same at any width). **Input guarded** pre-spawn: a prompt under **8 words** (`explore the repo`) nags the model back (`ModelRetry` naming `Prompt i ("…")` as *too terse*) and spawns NOTHING — rewrite and it fans out (a nag costs one retry leg; `agent` registers `retries=3`, so a stubborn model is coached, not aborted). **Output validated**: a report that is empty or backed by **zero tool calls** (child answered from memory) buys that child **exactly one** re-spawn with a nudge — two attempts, never three — then its section carries `The subagent returned no usable report.`; a child that *raises* gets `This subagent failed before producing a report.` Either way the siblings still fold. Every result ends with the **Synthesis Footer** — the harness telling the parent to compile the sections into ONE answer: prose **+ a text diagram** (ASCII/box-drawing; Mermaid only for a genuine graph), so a good fan-out answer *shows you a diagram* in the terminal. Children **silent-until-done** (no-op event sink), transcripts ephemeral — `--resume` carries only the one spawn call + the folded aggregate (ADR-0013, ADR-0017). |
| **Opik tracing (observability)** | `export OPIK_API_KEY=<comet-key>` (free at comet.com), relaunch `uv run decode`, run any turn; open it in Opik UI | on launch `Decode - Opik tracing on (project 'decode-local').` prints **once, before banner** — the project is DERIVED `decode-<DECODE_ENV>` unless `OPIK_PROJECT_NAME` is set (ADR-0015 §8), so `DECODE_ENV=prod` files traces under `decode-prod`; unset key → **byte-identical** (zero spans, no network, no line). One REPL **turn** = one **Trace** (root **Span** `chat_turn`); session's traces = one **Thread** keyed on session id — gated tool's approve/resume leg + follow-ups ride the **same** trace (latency honestly includes gate wait). Every LLM + tool call = **Span**: inputs/outputs, latency, tokens (`gen_ai.usage.*`), cost for models Opik prices (Gemini yes; OpenRouter/Modal may be tokens-only). Memory write-back + compaction ride the one global `instrument_pydantic_ai()`; subagent `agent(...)` child **nests inside** parent turn's trace, child tokens visible (closes ADR-0013 §9). `--resume` mints fresh session id → **new** Thread. **Headless:** a `decode run` = one Trace (root span `decode_run`), Thread = the run's session id; activation surfaces only in LOG — **stdout stays exactly the answer** (pipe-clean), stderr carries only guards/notices. Opik and Kitaru are independent instrumentation (OTLP vs the recording adapter) and compose — recording a run changes no span (ADR-0019 §6). Evals/experiments = M13 (ADR-0014). |
| **decode run (headless)** | `decode run "list the python files"` (separate command, not in REPL) | agent tool-loops **headlessly** on a plain `asyncio.run` around the same `build_agent()` — every tool inline under bypass, no prompt, `ask_user` a no-op — prints the answer on **stdout** (pipe-clean: `\| pbcopy` is safe), exit `0`. **No durability, no pause, no wait** — a crash is a re-run (ADR-0019 §1). `RUNTIME_ENABLED=false` → one friendly stderr line, non-zero exit, nothing built. A missing TASK argument (and no Worker Task env) → one line naming both ways to supply one. |
| **Recording Seam (Kitaru session)** | `export KITARU_API_URL=https://<workspace>.cloudinfra.zenml.io; export KITARU_AGENT_ID=<agent-uuid>`, then `decode run "explain this repo"` — A/B: re-run with `env -u KITARU_AGENT_ID` | recorded run → LOG line `[kitaru] recording this run on <url> (agent_id=…, session_name=<run session id>)`, and `uv run kitaru session get <id>` shows every LLM + tool call as nodes; the REPL records the same way with `session_name` = decode session id, so a conversation's turns group. **B side (unset):** byte-identical run, **no kitaru module imported at all**. `KITARU_API_URL` only in `.env` (never exported) = NOT configured — decode never reads it, the adapter's client does (`set -a && . .env && set +a`). **Degrade:** point `KITARU_API_URL` at a closed port → ONE stderr line `[kitaru] not recording this run: … continuing on the bare agent`, the run still answers and exits `0` (recording is an observer, never an availability dependency). **Hard-fail A/B:** the same with `KITARU_TASK_ID=<uuid>` in the env (a Kitaru Worker Task) → non-zero exit instead, because an unrecorded replay is a lying experiment (ADR-0019 §3). |
| **decode run --model (model override)** | `decode run --model gemini-2.5-pro "list the python files"` | same bypass run; **Model Override** swaps only the active provider's model id for this run (provider stays `LLM_PROVIDER`-selected — no cross-provider swap). Answer → **stdout**, stderr carries only guards/notices. Presence, not correctness — a wrong model id for the provider fails at the first model request. A replay-time model swap is Kitaru's own `--override`, NOT this flag. |
| **Worker replay (baseline)** | `uv run python scripts/bootstrap_kitaru.py --dry-run` then without it; `set -a && . .env && set +a && kitaru worker start` in a second shell; then `kitaru replay create <SESSION_ID> --agent decode@<VERSION> --tool-policy '{"default":{"type":"history","scope":"baseline","on_miss":"error_result"}}'` | `--dry-run` prints the exact `kitaru agent version register` argv and registers **nothing**; a real run adds a **version** to the EXISTING `decode` agent (never a second agent). `kitaru worker list` → one `live` Worker. `kitaru replay get <REPLAY_ID>` → `status: completed` + `result_session_id`; `kitaru session get <RESULT>` → `origin: replay`, the node counts, and `inputs` = the baseline's own recorded payload (`{"input": …}` or a bare prompt string — all three shapes resolve to the prompt, anything else hard-fails; ADR-0019 Amendments §1). Isolation is the point: the spawn's cwd is Harness Home `~/.decode-kitaru-worker` (sessions, logs, `.decode/sandbox` clone all land there — `docker ps` during the run, `tail -f ~/.decode-kitaru-worker/.decode/logs/decode.log`), and `git status` in YOUR checkout stays clean. Provider keys come from the **Worker's shell**, not the run spec (`secret_ids: []`) — a Worker started without them fails at the first model request. Full runbook: [`06_evals_replays.md`](../../../running_the_code/06_evals_replays.md). |

**Sandboxing** (ADR-0012 — see invariants above for the seam / Workspace / fresh-exec / Harness Home model). Default `SANDBOX_MODE=none` keeps every row above byte-unchanged. In `docker` / `modal` the **whole tool scope** — file/search tools **and** `bash` — is the one Workspace; decode's artifacts stay at Harness Home. `--repo`/`SANDBOX_REPO` needs a sandbox mode; with `SANDBOX_MODE=none` → `Decode: --repo/SANDBOX_REPO clones a repo into the isolated sandbox Workspace, which only exists in a sandbox mode …`, non-zero, no traceback.

| Surface | Type this | Working looks like |
|---|---|---|
| **file tools + bash (docker Sandbox)** | launch `SANDBOX_MODE=docker decode --repo <url-or-local-path>` (needs running Docker daemon; `--local` = fast local-path clone), then `glob **/*.py, read one of them, write a NOTES.md, then run 'echo hi > side.txt && cat side.txt'` | launch: `Decode - cloning <repo> into the workspace…` (only when repo given AND Workspace still empty), then `Decode - starting docker sandbox (ghcr.io/astral-sh/uv:python3.12-bookworm-slim)…` + `sandbox:docker` banner segment — `docker ps` shows a `sleep infinity` container before any bash (clone failure degrades to empty Workspace + one friendly line). File/search tools + `bash` share the one `/workspace` tree — **live bind mount**, always truthful: bash-written file → `read`; bash `rm` → `glob`. `bash` still gates; fresh-exec (chain: `cd /workspace/app && …`); timeout kills only that `docker exec` — container + filesystem survive, reply says timed out. `lsp` + post-edit diagnostics **run** (host `ty` reads bind mount). **`git` installed into container at startup** (slim base ships none), identity preconfigured (`SANDBOX_GIT_USER_*`, default `decode`/`decode@localhost`) → model `git commit` in `/workspace` works; push/PR creds stay host-side (hand-back) unless you opt into `SANDBOX_GIT_TOKEN` (own row). `web_fetch` stays **gated** (host network). Exit or `/ship` → hand-back (see `/ship` row). **QA peek:** `docker exec -it <id> bash` (filter `ancestor=ghcr.io/astral-sh/uv:python3.12-bookworm-slim`) → `/workspace` = host `.decode/sandbox`. **Guard:** daemon stopped → `Decode: SANDBOX_MODE=docker but the Docker daemon is not reachable — start Docker and retry (see .env.example).`, non-zero, no traceback. |
| **file tools + bash (modal Sandbox)** | launch `SANDBOX_MODE=modal decode --repo <url>` (needs `modal token set` creds), then `glob **/*.py, read one, then run 'ls -a /workspace'` | after clone line: `Decode - starting modal sandbox …` **and** `Decode - uploading the workspace to the modal sandbox…` — Workspace **bootstrap-uploaded** (ONE tar, host → remote; NOT `add_local_dir`-seeded, NOT mtime-synced) → `/workspace` holds the cloned repo + seeded `.decode/skills`, not empty scratch. File tools go directly against remote `SandboxFilesystem` (`read`/`write`/`edit`); `glob`/`grep` = remote `find`/`grep` — truthful, no host mirror. `bash` gates; fresh-exec. Sandbox outliving `SANDBOX_TIMEOUT_S` recreated on next op, **re-bootstrapped from last local `.decode/sandbox` state**; reply notes in-sandbox changes since last export may be lost. `lsp` + post-edit diagnostics **best-effort-disabled** (host `ty` can't reach remote fs — ADR-0012 §7). **`git` baked into image** (`apt_install("git")` + `git config` layer for `SANDBOX_GIT_USER_*` identity, both cached) → `git commit` works out of box. `SANDBOX_GIT_TOKEN` set → token injected INTO modal sandbox (`GITHUB_TOKEN` via `modal.Secret` + credential-helper layer) → model can `git push` / open PR from inside — **one** `SANDBOX_GIT_TOKEN`, **one** mechanism, both backends (docker direct-injects the identical way, ADR-0016 §2; use a scoped, revocable PAT — the sandbox holds it and can read it). `web_fetch` gated. Session end or `/ship` → `/workspace` **exported** to host `.decode/sandbox` for hand-back. **Guard:** no Modal creds → ``Decode: SANDBOX_MODE=modal but Modal credentials are missing — run `modal token set …` (see .env.example).``, non-zero, presence only (no `modal` import, no network call). |
| **decode run in a Sandbox** | `SANDBOX_MODE=docker decode run --repo <url> "run 'uname -a', then read README.md and summarize it"` | headless **bypass** run executes `bash` + file tools inside the Workspace, no prompt, prints the answer, exit `0`. Completion → Workspace **auto-shipped** as a `decode/<session-id>` branch, pushed **host-side from this very process** (outcome line → **stderr**, stdout pipe-clean; the "hand-back runs inside the flow" constraint died with the flow, ADR-0019 §1). Same backend guard in the headless pre-flight (daemon down / modal creds absent / `--repo` in `none` mode → one stderr line, non-zero, **nothing built**). A Kitaru Worker replay is this same path, with `SANDBOX_MODE=docker` + the repo clone pinned by the Agent Version. |
| **/ship (git hand-back)** | in a `SANDBOX_MODE=docker decode --repo <url>` session, do work, type `/ship` (idle-only, like `/compact` / `/clear`) | Workspace secured onto `decode/<session-id>` branch, pushed host-side: `Decode - handed the workspace back on branch decode/<id> (pushed to origin).` (modal exports `/workspace` first; docker's mount already live). Failed push never loses work — `Decode - could not push decode/<id> to origin; the results are safe on the local branch decode/<id> in .decode/sandbox — push it yourself when ready.` Unchanged/non-git → `Decode - the workspace is unchanged from the cloned HEAD, so there is nothing to hand back.` `none` mode / no `--repo` → `Decode - no sandbox workspace to ship.` Same hand-back runs automatically on REPL exit (silent no-op on skip). |
| **sandbox git token (`SANDBOX_GIT_TOKEN`, both backends)** | `rm -rf .decode/sandbox` (**required** — see below), then `SANDBOX_GIT_TOKEN=<fine-grained-PAT> SANDBOX_MODE=docker decode run --repo <url> "create NOTES.md with one line, commit it, push the branch, then open a PR against main"`. A/B: re-run with `env -u SANDBOX_GIT_TOKEN` | branch pushed + PR opened **from inside** the sandbox. ONE mechanism, BOTH backends (ADR-0016 §2): the token is **direct-injected into the Worker env as `GITHUB_TOKEN`** — docker via a **value-less** `-e GITHUB_TOKEN` (value rides the docker *client's* env → never in a host argv: `ps aux \| grep -F "<PAT>"` is EMPTY), modal via a `modal.Secret` — plus the shared credential-helper (`x-access-token:$GITHUB_TOKEN`) so `git push` over HTTPS works; `gh` reads `GITHUB_TOKEN` natively. **Expected (and the honest cost): `docker exec <worker-id> env \| grep GITHUB_TOKEN` PRINTS the token** — a sandboxed process CAN read it, so use a **scoped, revocable** PAT. **B side (unset):** no `GITHUB_TOKEN`, no `git config --global credential.helper`, `docker run` argv byte-identical to the no-token case — the sandbox holds **no credential at all**, and host-side hand-back still ships the branch (ADR-0016 §4). The token rides the one config surface — a laptop's `.env`, a deployment's Modal Secret (ADR-0021 §1) — no second lookup. `rm -rf .decode/sandbox` before ANY `--repo` run — a populated Workspace is **reused** and `--repo` **silently ignored** (no `origin` → push dies with `'origin' does not appear to be a git repository`; check `git -C .decode/sandbox remote -v`). Explicit-empty `SANDBOX_GIT_TOKEN=` injects nothing (gated on the *value*); mode `none` = no-op (it IS the host). Retired `SANDBOX_CREDENTIAL_PROXY_ENABLED` / `SANDBOX_PROXY_IMAGE` in a `.env` are **silently ignored** — the Credential Proxy is deleted (ADR-0016 §1), no proxy container exists. Full tutorial: [`03_sandboxing.md`](../../../running_the_code/03_sandboxing.md#2-optional-push-code-to-github-from-the-sandbox-sandbox_git_token). |

**Evals + replay** (operator surfaces — [`05_evals.md`](../../../running_the_code/05_evals.md), [`06_evals_replays.md`](../../../running_the_code/06_evals_replays.md)). Both cost real model calls; without keys every command **skips friendly** (one line, exit 0).

| Surface | Type this | Working looks like |
|---|---|---|
| **Benchmark trial** | `export OPIK_API_KEY=…` (+ your provider key), Docker running, then `make eval-benchmark ARGS='--task 001-find-and-replace'` | ONE Trial of one task through the **shipped runtime**: a Rich table `decode benchmark — 1 task(s) x 1 trial(s)` with `pass@1 1.00` and an empty `infra` column, then `evals benchmark: experiment bench-<stamp> logged under decode-evals; trial dirs in .decode/evals/runs/bench-<stamp>`. The **Trial Dir** is the evidence — `ls .decode/evals/runs/<job>/001-find-and-replace__<short-id>/` shows `seed/ home/ pristine/ agent/ verifier/ result.json`: `agent/summary.json` has `exit_reason: completed` and a `handback.branch` of `decode/<short-id>`, `git -C seed branch` lists that branch, `cat verifier/reward.txt` → `1.0`, `result.json` says `"status": "agent_ok"`. Grading is **host-side**: `pristine/` is a clone of the handed-back branch with `tests/` overlaid last, so the agent could not have touched its grader. **Break it on purpose:** `--task 018-git-bisect-revert --trials 2` (hard tier) — a lost trial must read `agent_fail` with a reason naming the ceiling/timeout, NOT `infra_error`; an `infra_error` means the harness failed (Docker down, seed failed) and is excluded from both numerator and denominator (ADR-0022 §4). Keyless: `make eval-benchmark` → `evals benchmark: skipped — set OPIK_API_KEY, …`, exit 0. |
| **Local Kitaru (record → cohort → replay)** | `make kitaru-local`, export the two lines it prints, then `uv run decode run "say hi in exactly three words"` | `make kitaru-local` starts the OSS server under docker compose (`docker ps`: the kitaru image + `postgres:16-alpine` on `localhost:8000`), then `scripts/bootstrap_kitaru.py` prints one row per resource (agent `decode`, version `decode@1` docker, importer `opik`, every `evaluators/*.py`) and the `export KITARU_API_URL=…` / `export KITARU_AGENT_ID=…` pair — **run it twice: the second run registers nothing** (idempotent). With both exported, the run logs `[kitaru] recording this run on http://localhost:8000 (agent_id=…, session_name=<session id>)` and `uv run kitaru session list --agent decode --origin recorded --size 3` shows it; `session get <ID>` shows every model + tool call as nodes. Join to Opik: a benchmark job run with `KITARU_AGENT_ID` exported freezes its losers with `uv run python -m evals kitaru cohort from-experiment <JOB>` (refuses with ONE line when the experiment recorded no `kitaru_agent_id`), and `uv run python -m evals kitaru import <TRACE_ID>` backfills a trace decode never recorded (needs a live Worker, else it waits and times out). Replay: in a second shell `set -a && . ./.env && set +a`, `unset KITARU_AGENT_ID`, `uv run kitaru worker start`, then `uv run kitaru replay create <SESSION_ID> --agent decode@1 --evaluator 'decode-request-limit@1' --tool-policy '{"default":{"type":"history","scope":"baseline","on_miss":"error_result"}}'` → `kitaru replay get <REPLAY_ID>` reaches a terminal state with a `result_session_id`. `uv run kitaru logout` stops the server; the same steps against the managed workspace are ONE `KITARU_API_URL` switch. |

**Mid-turn interaction** (while a turn streams — ADR-0002 §4-5):

- **Steer** — type a line + plain **Enter**: injected at next model-request boundary (never mid-stream/mid-tool).
- **Follow-up** — **Alt+Enter**: queued, drained only when turn would otherwise stop; continues as new turn.
- **Abort** — **Esc**: turn stops at next boundary, keeps work done, REPL idle (`[aborted]` marker).

**Persistence + memory across sessions:**

- `decode --resume` (or `decode --resume <session-id>`) replays latest (or named) session log from `.decode/sessions/*.jsonl`; prior conversation seeded, continue it.
- On quit (`/quit` or `Ctrl-D`), one cheap Gemini call appends a dated one-line summary (`- YYYY-MM-DD: …`) to `.decode/MEMORY.md`; relaunch injects it into agent instructions.