Test Expert

---
name: test-expert
description: Authoritative reference for the cogni-template test pyramid and production-like e2e test topology — enforcement vs test layers, vitest configs, infra prereqs, GitHub/DoltHub test org boundaries, the cogni-operator-test app, what agents can/can't run locally, coverage tracking, and the non-obvious gotchas that bite every time. Use this skill whenever the user is writing a new test, asking "which layer does this belong in", debugging a flaky test, hitting CWD/env-loading issues, deciding between unit/component/stack/external, running any `pnpm test:*` or `pnpm arch:check` / `pnpm lint` command, working with testcontainers, touching fake adapters or `APP_ENV=test`, troubleshooting skip-gates, seeing errors from `.env.test` not loading, wondering about coverage or whether a test will run in CI, or trying to decide whether to run stack/e2e locally vs defer to CI. Also trigger when the user mentions mocking the database, Privy/GitHub App construction errors in tests, the smee proxy, `pnpm test:smee`, validating the agent API, the candidate/test GitHub App, `cogni-test-org`, `cogni-test-nodes`, node-formation flight testing, or anything that smells like test-environment setup. Short-circuits the usual "spelunk through docs + configs" lookup.
---

# test-expert

Reference desk for writing and debugging tests in this monorepo. Leads with the matrix; gotchas follow because half of test failures here trace back to one of eight repeated mistakes.

There are two distinct things people lump together as "tests":

- **Enforcement** — static checks run in CI that fail builds when rules are violated (typecheck, lint, dep-cruiser, format, doc invariants).
- **Test layers** — vitest (+ Playwright) suites that exercise code behavior.

The matrix below separates them because their tradeoffs, speeds, and fix patterns are different.

## Enforcement matrix (not vitest — static checks)

| Check           | Command                  | What it enforces                                         | Fix when it fails                           |
| --------------- | ------------------------ | -------------------------------------------------------- | ------------------------------------------- |
| **Typecheck**   | `pnpm typecheck`         | TS types across workspace                                | Fix the type, don't `any`-cast              |
| **Lint**        | `pnpm lint`              | Biome + ESLint rules                                     | `pnpm lint:fix` auto-fixes most             |
| **Format**      | `pnpm format:check`      | Prettier + Biome format                                  | `pnpm format` auto-fixes                    |
| **Arch**        | `pnpm arch:check`        | `.dependency-cruiser.cjs` layer / entry-point boundaries | Refactor the import, don't disable the rule |
| **Docs**        | `pnpm check:docs`        | AGENTS.md headers, metadata, work-item index             | Fix the frontmatter                         |
| **Root layout** | `pnpm check:root-layout` | Project root structure invariants                        | Move the misplaced file                     |

All of the above run in `ci.yaml`. Local bundle: `pnpm check:fast:fix` (iterate with auto-fix) → `pnpm check:fast` (strict, verify-only — what pre-push runs) → `pnpm check` (once, pre-commit).

## Test layer matrix (vitest + Playwright)

| Layer              | Config                             | Tests live in                   | Proves                                                                                                                                                     | Infra needed                                       | Command                    | In PR CI?                                                                                                                                                                                                        |
| ------------------ | ---------------------------------- | ------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Unit**           | `vitest.config.mts`                | `tests/unit/`                   | Pure logic, no I/O                                                                                                                                         | None                                               | `pnpm test`                | ✅ (via `test:ci`)                                                                                                                                                                                               |
| **Meta**           | same                               | `tests/meta/`                   | Doc / spec invariants                                                                                                                                      | None                                               | `pnpm test:meta`           | ✅                                                                                                                                                                                                               |
| **Contract**       | same                               | `tests/contract/`               | Zod shapes vs route handlers                                                                                                                               | None (in-memory)                                   | `pnpm test:contract`       | ✅                                                                                                                                                                                                               |
| **Ports**          | same                               | `tests/ports/`                  | Every adapter implements its port                                                                                                                          | None                                               | (in `pnpm test`)           | ✅                                                                                                                                                                                                               |
| **Security**       | same                               | `tests/security/`               | Auth, RLS, injection guards                                                                                                                                | None                                               | (in `pnpm test`)           | ✅                                                                                                                                                                                                               |
| **Arch (meta)**    | same                               | `tests/arch/`                   | _Enforcement itself hasn't been weakened_ — spawns `depcruise` against arch-probe fixtures to prove rules still catch violations                           | None (subprocess)                                  | `pnpm test:arch`           | ✅                                                                                                                                                                                                               |
| **Lint (meta)**    | same                               | `tests/lint/`                   | Verifies lint-rule config hasn't been weakened — same meta-test pattern as `tests/arch/`. Catches an LLM disabling an ESLint/Biome rule to make a PR pass. | None                                               | `pnpm test:lint`           | ✅                                                                                                                                                                                                               |
| **Component**      | `vitest.component.config.mts`      | `tests/component/*.int.test.ts` | Adapter ↔ real Postgres                                                                                                                                   | Testcontainers (Docker)                            | `pnpm test:component`      | ✅                                                                                                                                                                                                               |
| **Stack (single)** | `vitest.stack.config.mts`          | `tests/stack/`                  | Full HTTP through one node                                                                                                                                 | `dev:stack:test` or `docker:test:stack`            | `pnpm test:stack:dev`      | ✅                                                                                                                                                                                                               |
| **Stack (multi)**  | `vitest.stack-multi.config.mts`    | `tests/stack/`                  | Cross-node isolation / routing                                                                                                                             | `dev:stack:full:test`                              | `pnpm test:stack:multi`    | ✅                                                                                                                                                                                                               |
| **External**       | `vitest.external.config.mts`       | `tests/external/` (non-money)   | Real 3rd-party APIs                                                                                                                                        | GH App creds, Ollama, optional `pnpm test:smee`    | `pnpm test:external`       | ❌                                                                                                                                                                                                               |
| **External money** | `vitest.external-money.config.mts` | `tests/external/money/`         | Real on-chain + real OpenRouter spend                                                                                                                      | Funded wallet, `dev:stack` running, OpenRouter key | `pnpm test:external:money` | ❌                                                                                                                                                                                                               |
| **E2E**            | Playwright                         | `nodes/operator/app/e2e/`       | Browser black box against a running stack                                                                                                                  | `docker:stack`                                     | `pnpm e2e`                 | ❌ — **currently not triggered anywhere** (known gap; the old `staging-preview.yml` runner was deleted during the flighting/CI-CD refactor). Tests exist and are runnable locally; automated invocation is TODO. |
| **Env e2e**        | GitHub App + GH Actions + Argo     | deployed candidate/test env     | Production-like pipeline: node publish → child repo image → parent pin/flight → `/version`                                                                 | `cogni-operator-test`, test GH/DoltHub orgs, VM    | operator API + GH checks   | 🟡 Partially wired. This is the missing repeatable flight lane, not a vitest suite. See "Environment e2e test lane" below.                                                                                       |

### Meta note on `tests/arch/` + `tests/lint/`

These are intentionally separate from the enforcement commands. `pnpm arch:check` validates the codebase _right now_. `tests/arch/` validates that the enforcement itself still works (i.e., that someone — including an LLM — hasn't quietly weakened the dep-cruiser rules to make a failing PR pass). If `arch:check` passes but `tests/arch/` would fail, that's the signal rules have been neutered. `tests/lint/` is the same pattern, currently unimplemented.

## Tokenomics distribution e2e — the "walk" fork harness (scripts/e2e/)

The full token-distribution loop has a **proven, product-code e2e** most agents don't know exists because it lives in `scripts/e2e/`, not a vitest lane. It is the regression proof for the whole finalize→fold→publish→claim mechanism — reach for it before hand-testing distributions.

| Harness                                             | Proves                                                                                                                                                                                                                                                                                | How                                                                                                                                                                                                                                              | Real $ / prod risk                 |
| --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------- |
| `scripts/e2e/finalize-mint-claim.ts`                | **sign → finalize → fold → mint → claim, conservation holds** — drives the REAL `finalizeEpoch` + `buildAndPersistCumulativeDistribution` fold, the persisted manifest, and the vendored 1inch `CumulativeMerkleDrop`. Only the admin EIP-712 signature + orchestration are scripted. | `pnpm test:walk:dev` (anvil Base-fork + real host ledger worker via Temporal). Prereqs: foundry on PATH, `pnpm dev:infra`, a review epoch pinned to the anvil approver (`SEED_APPROVERS=0xf39F…2266 pnpm db:seed`). See `scripts/e2e/README.md`. | None — anvil fork + local DB only. |
| `scripts/e2e/verify-publish-condition-on-fork.ts`   | The scoped `DistributionPublishCondition` answers ERC-165 so `DAO.grantWithCondition` doesn't revert, and `isGranted` allows ONLY the publish shape (positive + negative execute checks).                                                                                             | `pnpm tsx scripts/e2e/verify-publish-condition-on-fork.ts` against a running `scripts/e2e/start-fork.sh`.                                                                                                                                        | None — fork.                       |
| `scripts/e2e/seed-toks2.mts` (+ `SEED_INCREMENT=1`) | Seeds a toks2 review epoch (or an increment folded onto the prior manifest) for the local-UI walk rig.                                                                                                                                                                                | `DATABASE_SERVICE_URL=… pnpm tsx scripts/e2e/seed-toks2.mts`.                                                                                                                                                                                    | None — local DB.                   |

**GAP (known):** these are manual harnesses — `finalize-mint-claim.ts` is now `pnpm test:walk:dev` but NOT yet in CI (`ci.yaml`), and the fork-condition + toks2 rig have no `pnpm test:*` alias. Wiring `test:walk` into a CI lane (needs anvil + temporal + postgres) is the missing repeatable regression gate for tokenomics — same shape as the `e2e`/`env-e2e` gaps above. Spec: `docs/spec/tokenomics-distribution.md` (§ "What is proven", § "Local rig vs production").

## What agents can actually run locally

For ~90% of agent sessions, infra-gated lanes are out of reach. Use this split:

**Always available (no infra):**

- `pnpm test`, `test:meta`, `test:contract`, `test:arch`, `test:ci` (coverage)
- `pnpm typecheck`, `pnpm lint`, `pnpm arch:check`, `pnpm check:docs`, `pnpm check:fast`, `pnpm check`

**Needs Docker running:**

- `pnpm test:component` (spins up testcontainers-postgres per run)
- `pnpm test:external` (testcontainers + external APIs)

**Needs Docker + full stack + secrets — usually defer to CI:**

- `pnpm test:stack:dev` / `:docker` / `:multi` (requires `dev:stack:test` or `docker:test:stack` running, plus `.env.test` populated)
- `pnpm test:external:money` (funded wallet, real $)
- `pnpm e2e` (full docker stack, browser)

**Default agent pattern:** run what you can locally; for stack/e2e/money, push the branch and **defer to CI** (`ci.yaml` runs component + stack:docker), or ask the human to run `pnpm dev:stack:test` locally and then run the stack test against it.

## Coverage — the ignored dial

`pnpm test:ci` runs the unit suite with coverage (lcov + json-summary + text reporters). It's wired in `ci.yaml:128`. Nobody's been tracking the output lately, but the infrastructure is live — a PR adding a coverage report comment or a coverage-diff gate is a small change, not a new project.

If the user asks "is this covered?" or "what's our coverage look like?", the answer is: run `pnpm test:ci` locally, then open `coverage/lcov-report/index.html`. Or (future) wire a CI step to comment coverage deltas on PRs.

## Picking the right layer

Use the lightest layer that can prove the assertion — heavier layers cost minutes, not seconds, and a misplaced test burns budget every CI run.

- Pure logic, no I/O → **Unit**.
- Shape of an HTTP request or response → **Contract** (Zod round-trip, no server).
- Adapter ↔ real Postgres/Drizzle behavior → **Component** (testcontainers).
- Full HTTP request going through middleware, auth, services, DB → **Stack (single)**.
- How nodes behave when cross-calling each other → **Stack (multi)**.
- Real GitHub API / Ollama / OpenRouter behavior → **External**.
- Real on-chain transaction or real OpenRouter spend → **External money**.
- Production-like browser-driven black box → **E2E** (and know it only runs in deploy flows).

If the user is about to mock the database, push back — use **Component** with testcontainers instead. Mocked DBs have previously masked broken migrations here; the convention exists for a reason.

## Related but distinct: agent API validation

When the user's question is "does the machine-agent API actually work end-to-end against canary or a local stack?" — that's **validation**, not testing. See `docs/guides/agent-api-validation.md`: a curl-based checklist for discover → register → execute graph → list runs → stream events. It's a human/agent-driven probe, not a CI suite. Point at that guide when the user is validating the agent API surface rather than writing a unit/component/stack test.

## Environment e2e test lane — GitHub App + DoltHub + flight

This is the production-like lane for questions such as "can the operator really birth a node and flight it?" It is not covered by local vitest, stack tests, or Playwright. Treat it as a disposable mini-prod environment driven by the same deploy code.

> **Status 2026-07-09 — `cogni-test-org/cogni-monorepo` is now production-shaped** (PR #47). It previously **404'd the env-membership verb at catalog fetch** because `infra/k8s/argocd/appsets/<env>/` never existed there, its `render-node-{overlays,appset}.sh` had drifted from `cogni-dao/cogni` (pre-#1934, not envs-aware), and the preview/production `node-template` overlay templates were absent. PR #47 synced the #1934 node-deploy tooling + 3-env node-template overlays, added `envs:` to the deployable rows, and generated the appsets tree (6 ApplicationSets) + 15 overlays — all drift gates green. **`anotha` + `brother` are now fully shaped (overlay + appset in all 3 envs) → verb-targetable** for the story.5020 add→merge→reconcile e2e. Keeping the test-org tooling in sync with `cogni-dao/cogni` is an ongoing requirement — the verb emits `cogni-dao/cogni`-shaped output, so the test-org drift gates must match or the verb's PR won't merge.

Current test topology:

| Boundary              | Value / rule                                                                                                                                                        | Source                                                                                                                  |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| GitHub App            | `cogni-operator-test` for candidate/test. It must be an env-scoped App, installed on the disposable mint org as **all repositories**, with `workflows:write`.       | `.claude/skills/git-app-expert/SKILL.md`, `docs/guides/github-app-webhook-setup.md`, `docs/spec/node-ci-cd-contract.md` |
| GitHub mint org       | `cogni-test-org` via `NODE_MINT_OWNER` and `NODE_TEMPLATE_OWNER`. Candidate/preview operators must not mint into `Cogni-DAO`.                                       | `infra/k8s/overlays/{candidate-a,preview}/operator/kustomization.yaml`                                                  |
| Parent pin repo       | `cogni-test-org/cogni-monorepo` via `NODE_SUBMODULE_PARENT_OWNER` / `NODE_SUBMODULE_PARENT_REPO` for test runs.                                                     | `infra/k8s/overlays/candidate-a/operator/kustomization.yaml`                                                            |
| DoltHub knowledge org | `cogni-test-nodes` via explicit `DOLTHUB_OWNER`. Production uses `cogni-dao`; non-prod must fail closed rather than silently creating prod knowledge repos.         | `.claude/skills/database-expert/SKILL.md`, `docs/runbooks/dolthub-remote-bootstrap.md`                                  |
| Control endpoint      | Candidate operator remains `https://test.cognidao.org`. This is where the test App calls operator APIs; it is not a node-workload DNS suffix.                       | `.claude/skills/dns-ops/SKILL.md`                                                                                       |
| Flight target         | **New as-will-be design, still being refined:** test-org Akash nodes serve at `https://<node>.cogni-testing.org`; the flight must verify `/version.buildSha` there. | `.claude/skills/dns-ops/SKILL.md`, `scripts/ci/verify-buildsha.sh`                                                      |
| DNS credential        | The writer token is restricted to the `cogni-testing.org` zone and held by candidate-a OpenBao/ESO, never by `cogni-test-org`.                                      | `.claude/skills/dns-ops/SKILL.md`, `.claude/skills/cicd-secrets-expert/SKILL.md`                                        |

Recent PR context to know before changing this lane:

- #1521 taught candidate flight to detect added submodule catalog rows, resolve child images by `sha-<gitlinkSha>`, and promote per-target source SHAs.
- #1527 added DoltHub knowledge remote bootstrapping and proved `cogni-test-nodes/knowledge-e2e-*` creation through the DoltHub REST/SQL API.
- #1542 made the catalog's `image_repository` authoritative for externally built child node images (`ghcr.io/<owner>/<repo>-node`), removing GHCR inference.
- #1544 hardened publish retries when a child repo already exists from a partial wizard run.
- #1546 is the current prototype for a node-ref flight endpoint: `POST /api/v1/nodes/[id]/flight { sourceSha, environment }`. Review concerns are real; do not treat it as landed architecture until merged.

Pareto path for repeatable env e2e:

1. Make the **test org/app contract** explicit and audited first: `cogni-operator-test` installed all-repositories on `cogni-test-org`, `workflows:write`, `administration:write`, webhook to `https://test.cognidao.org/api/internal/webhooks/github`, creds in OpenBao for candidate/test.
2. Keep candidate/test disposable but production-shaped **without copying production identity or credentials**. Workflow/generator behavior should mirror production; owners, Apps, domains, secrets, wallets, and paid-provider accounts remain test-scoped. The test App is wild-west authority only inside the test org.
3. Bind public routing correctly: `DOMAIN=cogni-testing.org` for test workloads, `FORK_DOMAIN_ROOT=cognidao.org` for the candidate-a shared substrate, the new zone id in the materializer input, and a DNS writer token restricted to `cogni-testing.org` in candidate-a OpenBao. The candidate operator itself remains at `test.cognidao.org`.
4. **Match the current production GHCR contract: deployable child images are public.** Parent CI and Akash providers pull them anonymously; the current SDL renders no registry credentials. [GitHub makes a newly published container package private by default](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry), and the organization `Package Creation · Public` control only permits public packages—it does not change that default. Therefore node birth currently includes an explicit package-visibility flip to public, and the test org mirrors that same operation for every deployable child package. For the first flight, `spawny-boi` being public is sufficient. Private runtime pulls are a future hardening path tracked by `task.5143`, not a prerequisite or description of today's production behavior.
5. Run one synthetic-node smoke path: publish via the operator API, wait for `sha-<childSha>`, wait for the parent pin PR, merge/flight through the operator route, assert `https://<slug>.cogni-testing.org/version` reports the old source SHA, then promote the new SHA under a bounded probe and require zero DNS/TLS/HTTP failures.
6. Only after that exact path is repeatable should preview/prod consume it. Do not call static CI, a merged parent sync, a rendered `XComputeWorkload`, or a successful dispatch “E2E green”; the public workload must actually serve and update.

### Test-environment parity means behavior, not shared authority

Use this scorecard when someone says the test repo “mirrors production”:

| Must match production                                                                                                                            | Must be isolated                                                                                                                            |
| ------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------- |
| workflow logic, generators, schemas, required gates, operator verbs, Crossplane/Actuator path, image naming, promotion and verification sequence | GitHub App, org/repositories, DNS zone + token, provider/wallet account, DoltHub org, runtime secrets, databases, and destructive authority |

If a test credential can mutate `cognidao.org`, production GitHub repos, production VMs, or a production wallet, the test topology is not isolated. Do not “seed” production credentials to make it green; fix the missing test substrate or authority boundary.

### How test code reaches the test parent — deliver it directly, never merge-first

The parity table above says _what_ must match; this says _how the code gets there_. Getting these backwards produced a multi-PR detour through `Cogni-DAO/cogni` main during subtask.5007 — slow, and it exercised a forked path instead of the real one. Classify every change into one of two lanes:

| Change                                                                                                                                | Where it's authored                                                    | Delivery path                                                                                                                                                    | Speed                                              |
| ------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- |
| **Test-owned** — identity, org/domain/zone config, roster filters, generated appset/overlay state, fixtures, obsolete-fixture removal | Directly in `cogni-test-org/cogni-monorepo` (as `cogni-operator-test`) | Commit straight to the test-parent PR. **Never** route it through a `Cogni-DAO/cogni` main merge.                                                                | Seconds-to-minutes; no canonical dependency        |
| **Shared** — workflow logic, generators, operator verbs, schemas, required gates (destined for production)                            | Originates as a `Cogni-DAO/cogni` PR                                   | Stage the **PR head / exact ref** into the test parent → prove E2E on candidate → **then** merge canonical main → post-merge auto-sync converges the test parent | Bounded by one candidate E2E, not by a merge queue |

**The invariant:** you merge canonical main _because_ you proved the change on candidate — never _in order to_ prove it. Auto-sync (`Cogni-DAO/cogni` → `cogni-test-org/cogni-monorepo`) is a **post-merge convergence** mechanism; it is not a delivery channel for unproven code.

**Anti-pattern that bit subtask.5007** — `canonical main merge → auto-sync → candidate test`. This reverses the candidate-before-merge lifecycle: it puts the irreversible step (main merge) before the proof, and it validates a synced copy rather than the PR head you'll actually ship. Test-only generated state, roster filters, and dead fixtures should have stayed 100% in the test-parent PR.

**Missing platform primitive:** exact-ref/commit staging of a canonical PR head into the test parent. Until it lands, hand-stage the shared PR's head into a test-parent branch (cherry-pick / subtree the exact commit); do **not** merge canonical prerequisites just to make them appear in the test repo. File the gap, don't route around it with a merge.

## Gotchas — these bite repeatedly

1. **`APP_ENV=test` swaps fakes via the DI container.** Fake adapters live in `src/adapters/test/*/fake-*.adapter.ts` and are wired in `src/bootstrap/container.ts` via `serverEnv.isTestMode`. LLM is the exception — it's always real LiteLLM, routed to `mock-openai-api` via `litellm.test.config.yaml`. If a stack/component test is calling a real external service, it's almost always a missing fake wiring in the DI container, not a test bug.

2. **dotenv path-CWD trap.** A vitest config that does `config({ path: ".env.test" })` resolves **relative to CWD**, not the config file. It works when the script runs from repo root. It silently **fails to load** if invoked via `pnpm -F <node> ...` or `turbo run ...` because CWD changes to the node's directory. Symptom: skip-gates think creds are missing, tests blow up at provider construction. Fix pattern:

   ```ts
   const env = config({ path: path.resolve(__dirname, "../../../.env.test") });
   ```

3. **Skip-gate must precede provider construction.** External tests that build Privy/GitHub-App/EVM clients must do so _inside_ `describe.skipIf(!hasCreds)` or a gated `beforeAll`, never at module scope. Module-scope construction throws on missing env → the skip never runs → red test instead of a clean skip. Reference: `work/items/bug.0314` documents four real failures from this pattern.

4. **Testcontainers globalSetup uses `pnpm -w db:migrate:direct`.** The `-w` flag (workspace root) matters. Without it, the migrate script isn't found when globalSetup runs under `pnpm -F <node>`. See `nodes/*/app/tests/component/setup/testcontainers-postgres.global.ts`.

5. **Never mock the database.** In component/stack/external tests, use testcontainers or the real DB. Mocked DBs have concealed production migration bugs here before — the testcontainer overhead is cheap insurance.

6. **Sequence non-parallelism for stateful lanes.** Component + external configs use `sequence: { concurrent: false }` + `pool: forks`, `singleFork: true`. Stateful tests (shared GitHub test repo, single testcontainer DB epoch) race catastrophically in parallel. Don't remove this in a new config.

7. **Check discipline.** `pnpm check:fast:fix` during iteration (auto-fixes format/lint, runs unit tests). `pnpm check:fast` (strict, verify-only) is what `.husky/pre-push` runs — if it fails with drift, run the `:fix` variant, commit, retry. `pnpm check` once as the pre-commit gate; never run `pnpm check` more than once per session — it's the heavyweight pipeline.

8. **Time budgets.** Unit test files <1s. Component 5–30s. Stack 30–90s. External up to 3min (`testTimeout: 30_000` per-test; totals add up). If a test is exceeding these, first suspect missing env (see #2), not genuine slowness.

## When the test is already failing

Triage in this order:

1. **Env loaded?** Check the vitest output header for `[dotenv] injecting env (N)` — if `N=0`, `.env.test` didn't load. See gotcha #2.
2. **Creds asserted before construction?** If the stack trace shows the failure inside a provider constructor, not an assertion, see gotcha #3.
3. **Testcontainer started?** If `db:migrate:direct` errors, see gotcha #4. If migrations ran but DB state is surprising, the globalSetup's test-container epoch may not match expectations — check `testcontainers-postgres.global.ts`.
4. **Shared-state flakiness?** If tests pass solo but fail in the suite, see gotcha #6.
5. **External service reachable?** For external lane: is `pnpm test:smee` running for webhook-dependent tests? Is `OLLAMA_URL` reachable? Is the funded wallet still funded for money tests?
6. **Am I the wrong runner?** If the user is an agent hitting a stack test locally without `dev:stack:test` running, the answer may be "push and let CI run it."

## References

- `docs/guides/testing.md` — APP_ENV=test pattern + fake adapter conventions
- `docs/guides/full-stack-testing.md` — stack-test specifics
- `docs/guides/agent-api-validation.md` — machine-agent API validation checklist (validation, not testing)
- `docs/guides/github-app-webhook-setup.md` — per-env GitHub App setup; `cogni-operator-test` is the candidate/test app
- `docs/spec/node-ci-cd-contract.md` — submodule-pinned node CI/CD, test org identity, node-ref flight constraints
- `docs/runbooks/dolthub-remote-bootstrap.md` — `DOLTHUB_OWNER`, DoltHub PAT, and Dolt push credential split
- `work/items/bug.0314.external-tests-require-more-than-env-test.md` — latest skip-gate + RPC/webhook setup bug
- `work/projects/proj.system-test-architecture.md` — mock-LLM + FakeLlmAdapter strategy
- `nodes/operator/app/tests/external/AGENTS.md` — per-lane invariants for external tests
- `.github/workflows/ci.yaml` — exact CI step list; compare against this matrix if anything above looks wrong
- CLAUDE.md — check-discipline + format-before-commit rules

## Adding a new test — fast recipe

1. Decide layer from the "picking the right layer" list. If unsure, default to the lightest that proves the behavior.
2. Drop the file into that layer's directory with the matching filename pattern (`*.test.ts`, `*.spec.ts`, or `*.int.test.ts` for component).
3. Run the layer's specific command first (`pnpm test:component`, `pnpm test:contract`, etc.), not `pnpm check`. Fastest feedback.
4. If the test needs env or infra, read the relevant config's header TSDoc — every config documents its invariants at the top.
5. If the layer is stack/e2e/money and you're an agent without infra access, open the PR and let CI run it rather than burning a session on local setup.
6. Once the new test passes in isolation, run `pnpm check:fast:fix` (auto-fix) then `pnpm check:fast` (strict). Only run `pnpm check` when ready to commit.

## Driving the TEST operator as an agent (RBAC + credentials)

The **test operator** (`test.cognidao.org`, the `cogni-operator-test` GitHub App) is a **separate principal store** from the **prod operator** (`cognidao.org`). "operator" vs "test operator" is load-bearing — a credential valid on one is NOT valid on the other.

- flock-leader's `COGNI_*_API_KEY` (in `.env.cogni`) authenticates ONLY to the **prod** operator. Verify an agent key with `GET /api/v1/cognition` (200) — NEVER `/users/me` (agent keys always 401 there, which is not proof the key is dead). On `test.cognidao.org` the same key returns **401** (no principal there).
- On the **test** operator, flock-leader's identity is a **wallet session**, not the API key: `~/dev/cogni-template/.local-auth/candidate-a-operator.storageState.json` (Playwright storageState — build a `Cookie:` header from its `cognidao.org` cookies). The `personal-test-users-me.curl` capture **expires** — always confirm a session with `GET /api/v1/users/me` (200) before using it.
- That wallet session authenticates but has **no node grants by default** → `POST /api/v1/vcs/{merge,flight}` return **403 `authz_denied`** (NOT 401). Fix by requesting the grant, never by assuming a key gap:
  `POST https://test.cognidao.org/api/v1/nodes/<slug>/access-requests {"role":"developer"}` → `201 {status:"pending"}`; the node **owner approves** in the test-operator UI → `developer` grants `can_flight` → `vcs/merge` + `vcs/flight` then succeed.
- Diagnose by status: **403 `authz_denied`** = authenticated but ungranted → request access; **401** = wrong/absent principal for THIS operator; **400** validation_error = authorized, bad body (400 ≠ 403 — test before claiming a perm gap).
- `GET /nodes` is **empty** on the test operator (the in-repo operator resolves via `NODE_SUBMODULE_PARENT_*`, not the DB registry); `GET /nodes/<slug>` **500s** — use the node UUID from the catalog.