Deploy Verify

---
name: deploy-verify
description: Before/after gate for a deploy's telemetry — snapshot before, then check that every service still logs with environment=prod, traces carry the deployed release, scrape jobs are up and counters have series, no GlitchTip issue appeared since the deploy (each new one listed with status), Langfuse app traces carry the release, Umami still receives events, and (orrery) every live JS file's debugId matches an uploaded source-map bundle. Ends with DEPLOY-VERIFY PASS/FAIL. Use when asked to "verify the deploy", "check telemetry after the release", "did anything break since the deploy", or "do source maps resolve".
---

# deploy-verify

After every deploy on 2026-10-16/17, someone wrote a fresh verify script per service by hand.
This is that script, once, for both projects. It is a **gate**: expected against live,
PASS/FAIL. `o11y-review` is a **review**: patterns and counts over a window. Use both:
- the gate first;
- then `o11y-review` for anything the gate flags.

## Run (on the mini, through host-run)

```bash
V=~/.claude/skills/deploy-verify/scripts/verify.py
R=~/.claude/skills/host-run/scripts/run.sh

$R homelab $V snapshot --project closelistening --out /tmp/cl-base.json     # before the deploy
# ...deploy, then let it serve traffic for a while (10+ minutes)...
$R homelab $V compare --project closelistening --baseline /tmp/cl-base.json --release sha-ded80e0
$R homelab $V compare --project orrery --since 2026-10-17T08:21:00Z --sourcemaps
```

- **Projects:** `closelistening`, `orrery` and `flotilla`. The `PROFILES` table in the script lists
  each one's services, scrape jobs, counters, GlitchTip projects, Langfuse key file, Umami sites,
  live URL, and the `environment` its live lines carry (default `prod`; flotilla's is `operations`).
  Add a project there.
- **`--since`:** use it when there is no baseline, i.e. the time the new version went live.
- **`--release`:** the image tag the deploy shipped (`sha-<7>`, ADR-0010).
  - For podcast, run from the deploy's **image tag**, not the workflow commit. On 2026-10-16
    deploy-all ran from b0edbb9 and deployed `sha-ded80e0`.
  - orrery's web release is `gha-<short sha>` and isn't checked by this flag.

| Check | PASS means |
|---|---|
| logs | each service has prod lines since the deploy. Non-prod lines (dev runs) are INFO; lines with no environment label FAIL. |
| traces | each traced service has traces since the deploy, and `service.version` includes `--release` |
| probe | (orrery) one `GET` to `mcp.orrerylearn.com/mcp` with a `traceparent` whose trace id the script just generated: it must answer the pinned status (401, no token) and **that** trace id must reach VictoriaTraces within 60 s (polled every 5 s). Another status FAILs with it shown; a timeout FAILs with the trace id. Needs a sampler that keeps the span (orrery-mcp: `AlwaysOnSampler`). |
| metrics | every scrape job `up=1`; every listed counter has series over 2 days (`count_over_time`, not an instant query) |
| errors | **no** GlitchTip issue first seen since the deploy. Each new one FAILs the gate on purpose: someone has to look. Its status is shown. |
| langfuse | app traces since the deploy all carry `--release`. LiteLLM's own `litellm-*` traces are ignored. |
| umami | each site got events since the deploy, with the latest event's time. Zero events is INCONCLUSIVE while the time since the deploy is shorter than the quiet gap before it, from the site's last event to the deploy (sparse traffic: re-run `compare` later), and WARN once it is older. A failed Umami query is a FAIL, not "0 events" |
| sourcemaps | every live `_app/immutable` JS with a debugId has a stored GlitchTip bundle. Unmatched stubs under 1 KB are INFO. |

## Verified

**2026-10-17, podcast deploy `sha-ded80e0`** (since 2026-10-16 19:31):
- logs, traces, metrics and Langfuse passed;
- it FAILed on four new GlitchTip issues, three already known (the smoke alert, two reprocess
  issues);
- one **was not known**: `PLAYER-1I`, an Android `ForegroundServiceStartNotAllowedException`.
  The manual checks had missed it, and it was handed to the mobile session.

**orrery** (since 08:21):
- PASS, with 59 of 61 live JS files matched to bundles, the two map-less stubs as INFO, and
  dev lines from omarchy as INFO.

**Fixes made while testing:**
- Cloudflare 403s Python's default User-Agent, so the script uses a browser UA;
- `litellm-acompletion` traces are LiteLLM's own, so they're ignored;
- dev lines are INFO, not FAIL.

**2026-11-02, orrery v0.9.3 (`sha-b4ddf1e`):** PASS (58 of 58 live JS with a debugId, 56 matched,
2 stubs as INFO; `up{job=orrery}` 1 on both targets). Two gaps fixed first:
- served orrery JS carries the id as `sentry-dbid-<uuid>` (sentryVitePlugin; only the artifact
  copy has `//# debugId=`), so the check found 0 ids and still passed. It now reads both forms
  and fails when no live file has an id;
- the orrery profile had no scrape job, so metrics were skipped; it now checks `orrery`.
- A service with no traffic since the deploy shows 0 traces (mcp, 20 min after this deploy):
  send it one request and re-run before calling tracing broken.

**2026-11-08, MCP trace probe (from the orrery session):** no probe credential exists for
`mcp.orrerylearn.com`, so the probe sends none — the 401 is still a traced request.
- `compare --project orrery`: `[PASS] orrery-mcp probe: 401 from https://mcp.orrerylearn.com/mcp,
  trace a5af2f688d8e7fc371d22f58a91b78cf in VictoriaTraces after 21 s`.
- Failure paths, run from a throwaway copy: expected status 200 → `[FAIL] … answered 401, expected
  200 … — trace d3bfd59ed91d86173ec61ed260ce5022`; a service that never appears → `[FAIL] … trace
  a3260bf1c3b79e1959e91aaaf9094b47 NOT in VictoriaTraces after 20 s` (limit lowered for the test).

**2026-11-08, flotilla `sha-75a7491`** (since 18:45Z, its first deploy): `DEPLOY-VERIFY: PASS`. 21
flotillad, 142 worker and 272 egress lines (environment=operations), 6 flotillad traces with
service.version sha-75a7491, the three counters present. No GlitchTip project, Langfuse keys or
site yet, so those sections have nothing to check.

## Not covered

- **App behaviour** (user-facing smoke tests): the deploy workflows' own smoke steps do that.
- **The DGX services** (vLLM, pyannote): use `dgx-status` and `infra-drift`.
- **Release names:** GlitchTip release tags aren't checked, because orrery and podcast name
  them differently.
- **An unknown regression with no error:** the gate sees signals, not intent. Read the
  `o11y-review` output for warnings.