Test Claude Locally

---
name: test-claude-locally
description: Run the Weave router locally in docker compose and drive it with `claude -p` to reproduce and verify routing/translation behavior for a specific upstream model (e.g. GLM-5.1, DeepSeek, Qwen). Use when verifying a router fix end-to-end, reproducing a prod routing bug, confirming a model's streaming behavior (nudges, tool-call suppression, loop/no-progress breaks), or testing a `/force-model` route — without touching the user's global Claude Code config.
---

# Testing the router locally

> For an **automated** pre-merge regression net (fixture-driven, asserts caching/streaming/decision-headers against real Anthropic), run `make smoke` — see [docs/SMOKE.md](../../../docs/SMOKE.md). This skill is the interactive counterpart: stand the stack up by hand and drive it with `claude -p` to reproduce or verify a one-off bug.

Stand up the router in docker compose, point a one-off `claude -p` session at it via `--settings`, and read the local server logs to confirm behavior. Two upstream modes: the **real** provider API (needs a working key + credits) or a **mock** upstream that emits an exact SSE shape (deterministic, no credits).

## Critical gotchas (read first)

- **`~/.claude/settings.json` `env` overrides inherited env vars.** Setting `ANTHROPIC_BASE_URL` in the shell does NOT redirect `claude` — settings.json wins and the request silently goes to prod. Always redirect with `claude --settings <file>` (see `scripts/local-settings.json`). Never edit the user's global `~/.claude.json` or `~/.claude/settings.json` — that breaks their live session.
- **`claude -p` is stateless across invocations.** A standalone `/force-model` call does not persist to the next `claude -p`. Put `/force-model <model>` as the first line of the SAME prompt that contains the task.
- **The router ignores the request's `model` field** and routes via the cluster scorer. The ONLY way to pin a specific model is `/force-model` through a Claude Code session (raw curl cannot).
- **Port 8085 conflict.** The monorepo's pubsub emulator may already own host port 8085. Drop the router's host binding with a `docker-compose.override.yml` (see workflow). The server still reaches the emulator over the compose network.
- **No credits / no key = no reproduction.** If the real upstream returns an error (e.g. OpenRouter "Insufficient credits"), use the mock-upstream path instead.

## Workflow

```
- [ ] 1. Bring up the stack (handle port 8085)
- [ ] 2. Seed an API key
- [ ] 3. Choose upstream: real provider OR mock
- [ ] 4. Write a one-off local-settings.json
- [ ] 5. Drive with `claude -p --settings`, forcing the target model
- [ ] 6. Read local logs to verify behavior
- [ ] 7. Clean up
```

### 1. Bring up the stack

```bash
cd <router-repo>
# Drop the pubsub host-port binding to avoid an 8085 conflict:
cat > docker-compose.override.yml <<'EOF'
services:
  pubsub-emulator:
    ports: !reset []
EOF
docker compose up -d --build server   # --build picks up code changes
until curl -sf http://localhost:8080/health >/dev/null; do sleep 2; done
```

The override file is gitignored-by-intent scaffolding — delete it in cleanup.

### 2. Seed an API key

```bash
docker compose run --rm seed
```

Copy the `rk_...` key it prints.

### 3. Choose the upstream

**Real provider** — set the provider key in `.env.local` (e.g. `FIREWORKS_API_KEY=...`) and restart `docker compose up -d server`. Confirm the boot log shows `<Provider> provider enabled` with the real base_url. Use this to confirm a model genuinely produces the behavior.

**Mock upstream** — for a deterministic, credit-free repro of a precise SSE shape. Point the provider's base URL at a local mock and restart:

```bash
python3 scripts/mock_openai_upstream.py >/tmp/mock.log 2>&1 &   # serves :8099
# In docker-compose.override.yml under `server:`, add:
#   environment:
#     FIREWORKS_BASE_URL: http://host.docker.internal:8099/v1
#     FIREWORKS_API_KEY: sk-mock
#   extra_hosts: ["host.docker.internal:host-gateway"]
docker compose up -d server
```

Edit the mock's emitted chunks to match the upstream shape you're reproducing. Provider→env-var names live in `internal/providers/provider.go`; base-URL overrides are read in `cmd/router/main.go` (`<PROVIDER>_BASE_URL`).

### 4. One-off local settings

```bash
cat > /tmp/local-settings.json <<EOF
{ "env": {
  "ANTHROPIC_BASE_URL": "http://localhost:8080",
  "ANTHROPIC_CUSTOM_HEADERS": "X-Weave-Router-Key: rk_REPLACE_ME"
}}
EOF
```

### 5. Drive it

```bash
cd <scratch-dir-with-files-to-act-on>
env -u CLAUDE_CODE_SESSION_ID -u ANTHROPIC_BASE_URL -u ANTHROPIC_CUSTOM_HEADERS \
  claude -p 'First send exactly: /force-model z-ai/glm-5.1
Then <task that requires tool use>, then stop.' \
  --settings /tmp/local-settings.json --max-turns 10 --verbose
```

### 6. Verify via logs

The decision + completion log per action (one `ProxyMessages complete`) carries the signals you need. Strip ANSI first:

```bash
docker compose logs server --since=3m 2>&1 | sed -E 's/\x1b\[[0-9;]*m//g' \
  | grep 'ProxyMessages complete' | grep 'decision_model=z-ai/glm-5.1'
```

Useful fields: `decision_model`, `decision_provider`, `upstream_finish_reason`, `suppressed_tool_calls`, `text_only_turn_nudged`, `tool_use_blocks`, `resp_stop_reason`, `stop_reason_demoted`. Also grep for `recovery nudge`, `tool-call loop`, `no-progress`.

**Confirm the model was actually served** before drawing conclusions:
```bash
docker compose logs server --since=3m 2>&1 | sed -E 's/\x1b\[[0-9;]*m//g' \
  | grep 'ProxyMessages complete' | grep -oE 'decision_model=[^ ]+' | sort | uniq -c
```
If you only see other models, `/force-model` didn't take — recheck step 5.

### 7. Clean up

```bash
pkill -f mock_openai_upstream.py 2>/dev/null
rm -f docker-compose.override.yml /tmp/local-settings.json
# `docker compose down` if you want to stop the stack
```

## Testing the balance gate / subscription usage-bypass (managed billing)

For fixes to `internal/billing` + `internal/server/middleware/balance_check.go` + the usage-bypass path (`internal/proxy/usage_bypass.go`), the setup differs from the model-behavior workflow above:

- **The balance gate only attaches in managed mode with billing enabled.** Set `ROUTER_DEPLOYMENT_MODE=managed` on the `server` service. Billing auto-enables when the `router` schema's billing tables exist (they do after `migrate`), logged as `Router billing enabled`. In default selfhosted mode `WithBalanceCheck` is never wired and none of this triggers.
- **The Anthropic provider has no `ANTHROPIC_BASE_URL` override** (unlike Fireworks/Together/etc.) — `cmd/router/main.go` hardcodes `anthropic.DefaultBaseURL`. To point Anthropic at a mock, temporarily edit that line to `config.GetOr("ANTHROPIC_BASE_URL", anthropic.DefaultBaseURL)`, rebuild, and **revert it after testing** (don't ship it in the fix PR). The boot log line still prints `DefaultBaseURL` cosmetically — verify the mock is actually hit via the mock's own logs, not the boot log.
- **Requests must be MainLoop-shaped to reach the usage-bypass decision.** Short prompts (small `max_tokens`, no tools) classify as `Probe`/`Classifier`/`TitleGen` — all **hard-pinned**, short-circuiting before the usage-bypass and scorer branches. A trivial `"say hi"` will wrongly hit the refusal path. Use a realistic turn: `tools` present + `max_tokens>=4096` + a normal user message. Confirm `turn_type=main_loop` in the `turnloop classified` log before trusting the result.
- **Drive with raw `curl`** on the production router-key path — no `claude` binary needed, and it lets you pin the requested model (the usage-bypass path serves the *requested* model verbatim, so no `/force-model` needed here):
  ```bash
  curl -sS -N -D /tmp/h.txt -X POST http://localhost:8080/v1/messages \
    -H "Authorization: Bearer $RK" \
    -H "X-Weave-Anthropic-Subscription: sk-ant-oat01-anything" \
    -H "anthropic-version: 2023-06-01" -H "Content-Type: application/json" \
    -d '{"model":"claude-sonnet-4-5","max_tokens":4096,"stream":true,"tools":[{"name":"Bash","input_schema":{"type":"object"}}],"messages":[{"role":"user","content":"<realistic multi-sentence task>"}]}'
  ```
  A fake `sk-ant-oat...` token is enough for router-side classification; the mock doesn't validate it.
- **Set DB state directly** (host `psql -h localhost -p 5433 -U router -d router`, password `router`, `SET search_path TO router`): `UPDATE model_router_installations SET usage_bypass_enabled=true WHERE external_id='__router_admin__';` and upsert `organization_credit_balance` with `balance_usd_micros` below `SubscriptionOverdraftFloorMicros` (−5_000_000). Assert no debit via `SELECT count(*) FROM organization_credit_ledger WHERE organization_id='__router_admin__';` before/after.
- **Deterministic mock that branches on the requested model** is the cleanest way to drive both the serve path and the refusal path from one server: return 200 SSE for one model and 429 for another (the usage-bypass path forwards the requested model verbatim, so branching on `model` in the request body is reliable).
- **What good looks like:** header `X-Router-Decision: usage_bypass`, the depleted-credits warning injected as content-block index 0 (real content re-indexed to 1), log `Balance past subscription overdraft floor: serving subscription-only`. Retryable upstream error → log `Subscription-only bypass hit retryable error; refusing instead of paid reroute` → HTTP 402. No subscription header below floor → `insufficient_credits` 402 that never logs `ProxyMessages start`.

## Notes

- Local cluster version comes from `ROUTER_CLUSTER_VERSION` in `.env.local`; it may differ from prod, which is why `/force-model` (not the scorer) is the reliable way to hit one model.
- GLM-5.1's primary binding is Together (then Fireworks, then OpenRouter) — see `internal/router/catalog/catalog.go`.
- To confirm a deploy contains a given router commit: the prod Cloud Run revision name maps to a monorepo commit; `git ls-tree <monorepo-commit> router-internal/router` shows the pinned router submodule SHA.