Chat Quality

---
name: chat-quality
description: "Biweekly loop that turns real chat feedback into eval coverage. Read the thumbs verdicts from the live Redis, join them to the logged exchanges, grow the golden set from actual failures, run the answer-quality eval, and land the fixes on a local branch for approval. Use when asked to run the chat quality loop. Never pushes, merges, or deploys."
icon: reviews
displayDescription: "Reads the site chat's own report card: every thumbs verdict a visitor left, joined back to the logged exchange it rates. Disliked answers become golden-set regression cases, the answer-quality eval runs against the updated set, and the fixes land on a branch with a plain-English report for approval. One of the loops described on the Loops page."
invoke: ["run the chat quality loop","/chat-quality"]
---

# chat-quality

Biweekly loop closing the site chat's feedback cycle. The widget's thumbs verdicts land in Redis and, until this loop, nothing ever read them. Each run joins the verdicts to the exchange log, turns real failures into golden-set regression cases (the standing rule in `evals/chat/README.md`), runs the eval, and hands the fixes over on a local branch. **Never push, merge, or deploy — the user approves every change.**

## When invoked

Run when asked to "run the chat quality loop" (`/chat-quality`), or by a scheduled task once one is created.

## Scope guardrails (read first)

- **The KV store is live.** An Upstash Redis has been connected to the site since 2026-09-26, so the loop has real verdicts to read. The stop-and-ask rule in step 1 still stands as the safe failure mode if the credentials are ever missing.
- **Read-only on the live Redis.** `SCAN`, `GET`, `LRANGE`, `LLEN` against `chat:*` keys only — never `SET`, `DEL`, `EXPIRE`, or anything that writes. The logs are production evidence with a 30-day TTL; this loop observes them.
- **Visitor privacy.** Logged questions are visitors' own words and may carry personal details. They may appear verbatim in the local report, never in the committed diff: a golden-set case gets a paraphrase that preserves the failure, not the visitor's sentence. Visitor hashes never leave the report.
- **One eval run per loop.** `npm run eval:chat` spends real API budget (golden set × 3 repeats). Run it once, after the golden-set changes, not iteratively.
- **Fixes stay inside the chat's answer pipeline**: `evals/chat/golden-set.json`, corpus sources (page prose, `corpus-facts()` blocks, the generator's exclusions), the persona in `website/src/app/api/chat/persona.ts`, and the chat's lookup-tool layer — the `CHAT_TOOLS` definitions in `website/src/app/api/chat/route.ts` and their shared implementations in `website/src/lib/site-tools.ts` (a disliked prop or token answer can be a tool-description or lookup bug, and `site-tools.ts` also serves `/api/mcp`, so a fix there changes both surfaces). The engine is the package's, so a cause can also sit there: the follow-ups prompt, the handlers' notices and the `ask_user` guidance (`DEFAULT_ASK_USER_DESCRIPTION`, the authority for SPEC rules A1 and A2: a reply that asks when it should answer, or answers when it should ask) in `src/chat/server.ts`, the guardrail notices in `src/chat/upstash.ts`, the fallback follow-up pool in `src/chat/followups.ts` (and the site's own in `website/src/lib/chat-followups.ts`). Those are in scope, but a package edit changes every consumer's chat, not only this site's — say so in the finding, and prefer the site-side fix when one exists (for the asking guidance that is `askUser: { description }` in `website/src/app/api/chat/route.ts`). No UI, no components, no guardrail-cap changes (spend caps are the user's call — propose, don't edit).
- **Local branch only.** Never push or touch the user's working tree — do the work in a temporary git worktree on a fresh branch named `chat/YYYY-MM-DD-<slug>`, and remove the worktree when done (the branch survives). Skip the worktree entirely when there is nothing to encode.

## The loop

### 0. Close the previous loop

Read the newest report in `evals/chat/loop-reports/` (git-ignored, local-only). Note whether its branch was merged; don't re-propose a case a previous run already encoded. First run ever: say so and move on.

### 1. Pull the verdicts and the log

Credentials are `KV_REST_API_URL` / `KV_REST_API_TOKEN` in `website/.env.local`. They are often **commented out** there so local dev fails open — read the values from the file either way; never uncomment them or export them into a dev server. If the file has no values at all, stop and ask the user for read access; there is no loop without the data.

Query the Upstash REST API directly (plain `curl`): `SCAN` for `chat:feedback:*` and `GET` each verdict; `LRANGE` the day lists for the window since the last report (the key builders are owned by `src/chat/upstash.ts` and the entry shape by `ExchangeLog` in `src/chat/server.ts` — the site's `website/src/app/api/chat/guardrails.ts` only configures them — so read them there rather than trusting a remembered shape). Join verdicts to log lines by exchange `id`.

### 2. Triage

Every **down** verdict is a golden-set candidate: read the question, the answer, and the page it was asked from. The exchange's `path` field holds the request's whole context as the route checked it, so for the site chat it is a JSON string of the page and, when the visitor had highlighted something, the highlight with what it sits in (`website/src/lib/site-chat-context.ts`); a bare path is an older entry. A down verdict on a highlighted question is often about the highlight, so read it with the answer. Also skim the unrated exchanges for silent failures — invented paths, guardrail notices where a real answer was possible, questions the corpus plainly couldn't answer. The log does not record a question set the reply offered: an answer tapped on a question card arrives as the next exchange, its "question" being the plain-text answer ("Next.js", or one prompt per line for a set), so read those against the exchange before them rather than as visitor questions. For an asking failure, the cheapest assertion is the `[question]` line `evals/chat/transform-response.js` appends (`contains` when the reply should ask, `not-contains` when it should not). **An empty or all-up window is a valid outcome**: write the short report and stop. Don't manufacture cases from answers that were actually fine.

### 3. Encode the failures

Apply the standing rule from `evals/chat/README.md`: each real failure becomes a golden-set case **before** it is fixed — paraphrased question, its `requiredFacts`, and the cheapest assertion that would have caught it, with the case's `description` citing the rule ids it covers per `evals/chat/SPEC.md` (a failure no rule covers means the spec gains the rule in the same change). If a required fact is missing from the corpus, that's the actual bug: fix the source (page prose or a `corpus-facts()` block), and `scripts/validate-chat-coverage.mjs` will hold the new case to the regenerated corpus. The one exception: a failure whose facts live in the generated prop or token data is answered by the chat's lookup tools, not the corpus — its case is marked `source: tools` and carries empty `requiredFacts` (the standing-rule section in `evals/chat/README.md` and rule T4 in `evals/chat/SPEC.md` own the convention); never fix one by stuffing a prop fact into page prose.

### 4. Run the eval

Follow `evals/chat/README.md` exactly — it owns the procedure (the `website-eval` launch entry / `npm run dev:eval -w website`, the port-3000 gotcha, `npm run eval:chat`) and how to read the transcripts (seat by seat; the aggregate score is noise). The question being answered: do the new cases fail before the fix and pass after, and did nothing that previously passed regress?

### 5. Fix on a branch

Work in the temporary worktree on branch `chat/YYYY-MM-DD-<slug>`. One coherent batch: the new golden-set cases plus the corpus/persona fixes they demanded. A persona or tool-definition edit updates its matching rule row in `evals/chat/SPEC.md` in the same commit, and a new rule ships with either a tripwire or an explicit unenforced entry there. Do not build in the worktree: a fresh one has no `node_modules` of its own, so the prebuild chain fails wherever module lookup cannot climb to the main checkout's install (the corpus generator needs `typescript`), and a worktree that happens to sit inside the checkout only borrows that install, and installing there is what `land` forbids. The build check happens when the branch is landed, whose combined `npm run verify` builds it; when page prose changed, the corpus that build regenerates is committed with the landing.

### 6. Report and hand off

Save to `evals/chat/loop-reports/YYYY-MM-DD.md` **and** repeat in full in the final message: the window, verdict counts (up/down/total exchanges), each disliked answer and the diagnosis, what was encoded, the eval result, and the branch name. Plain English. End with the approval step: merging the branch (or saying `ship`) approves it; deleting it rejects it. Nothing is pushed or deployed.