Analyze

---
name: analyze
description: >-
  Understand failures and improve the eval assets: cluster failures, label
  judge-calibration cases (15-minute flow), edit rubrics while grading, mine
  traces for new cases, and manage the sealed holdout. Use with --transcripts
  when the user has exported chats, support logs or saved traces and wants to
  know what is going wrong — that branch needs NO eval setup: no profile, no
  cases, no run. Otherwise use after runs to triage results, to calibrate the
  judge, or to promote real conversations into the dataset.
argument-hint: "[--cluster] [--transcripts <path>] [--label] [--mine] [--unseal]"
allowed-tools: >-
  Read Grep Glob Bash(python3 ${CLAUDE_PLUGIN_ROOT}/scripts/*) Bash(python3 "${CLAUDE_PLUGIN_ROOT}/scripts/*)
---

# Analyze — Failures, Labels, and the Living Dataset

> **Plugin root:** `${CLAUDE_PLUGIN_ROOT}`. Reference files and docs write that
> placeholder literally (it is only substituted here), so read every
> `${CLAUDE_PLUGIN_ROOT}/…` path you meet in them as this absolute path, and
> quote it in shell commands.
>
> **Calling the plugin.** Open its files with Read — never `cd` into the
> plugin, and never `ls`, `grep` or `cat` it from the shell: the plugin is not
> a working directory, so those prompt, and headless a prompt is a denial.
> Run each script as its own Bash call, spelled
> `python3 "<that path>/scripts/<name>.py" …` with the path written out: no
> `cd`, no `&&` or `; echo $?` tail, no shell variable holding the path. The pre-approval matches
> that literal form only, and it lapses when the user next replies — a prompt
> after that is expected, not a fault. If a call is **denied**, stop and tell
> the user which permission is missing; never work around it by hand.
>
> **Every shell call, not only script calls.** One plain command per Bash
> call — no `cd`, `&&`, `|`, heredoc, `$( )` or `/tmp` — so an allow rule can
> match it. A denial of **any** call is a stop, never a retry in another form.

Every branch here reads a run and edits the dataset. Four documents own what
they name, and this file never restates their rules:
[references/annotation-ux.md](references/annotation-ux.md) (the viewer, the
annotation JSONL, open→axial coding and its stopping rule),
`${CLAUDE_PLUGIN_ROOT}/docs/rubric-format.md` (rubric shape and the calibration
numbers), `${CLAUDE_PLUGIN_ROOT}/agents/judge.md` (judge runtime, model family)
and `${CLAUDE_PLUGIN_ROOT}/skills/generate/references/case-format.md` (what a
valid case is).

Arguments, when the user typed any: `$ARGUMENTS`

## `--transcripts <path>` — real conversations, before any dataset exists
The error-analysis-first entry, and the one branch that needs **no profile, no
cases and no run**: a user with exported chats, support logs or a folder of
saved traces can learn what is broken today. Read them, and write one viewer
record per conversation to a scratch JSON file — `{"case_id": "t-0001",
"trace_id": ..., "answer": {"actual": <the app's reply>}, "tool_calls": [...]}`
plus the user message; the record shape is loose by design
(`build_review_viewer.py`'s docstring) and no field is required. Redact first
when the data may hold PII, and show the redacted form. Build the viewer on
that file and run annotation-ux.md's open→axial pass with the user: ~20–50
conversations, free-text critique first, categories only once they repeat.
No reviewer in the session: code them yourself, record the reviewer as
yourself (not a human) in the annotation log, and call the taxonomy
provisional. Exported chats rarely carry tool calls; when the app has traces,
they can be joined by conversation id or time to supply them — say which you
did. End with the failure taxonomy by count and an offer to promote the
clearest failures into cases (origin `real-trace:<id>`, expectation = what
SHOULD have happened, `review.status: pending`, `gating: false` until a human
accepts it) — which is `generate`'s "real seeds first" done literally.
Bump `dataset_version` in `dataset.yaml` when you promote; the pinned
baseline then stops being comparable (run/SKILL.md §4): say so, and pin a new
one after. This is not `--mine`: that queries a live trace
backend and needs `stage: traffic`; this reads files the user already has.

## First run — no prior data
Check before clustering, labeling or mining an empty set, and name which empty
state you found. Same discipline as `run/SKILL.md` §4's first-run branch. Rule
out the state that only *looks* empty before calling it a start.
- **No completed run** — no `reports/<run-id>/`, or the latest is all
  `infra_error`. Point at `/evalup:run`, or `/evalup:start` when you
  can't tell which of discover/generate/run is missing.
- **No rubric or judge** — `judge.status` absent, no case carries `rubric:`.
  `--label` has nothing to calibrate; point at rubric-format.md rather than
  launching the judge on zero cases.
- **`--mine` finds nothing.** Say which: no trace backend wired, stage is
  pre-`traffic`, or wired and genuinely nothing new.

## `--cluster` (default after a run)
Read the latest run's failed cases from `reports/<run-id>/`, never the holdout
details. Pick the latest by the **timestamp segment** of the run id
(`<mode>-<YYYYMMDDTHHMMSSZ>`, as `make_plan.py` builds it). A whole-string sort ranks by
mode and hands you a stale `smoke-*`. Use the timestamp over mtime, which
drifts when a report is regenerated or a directory copied.

**Trace-less run** (`manifest.yaml` `traces.collected: false`): there is no
trajectory to read, so cluster mechanically first — by (expected status,
observed status, the routing or answer check that failed), all of which
`verdict.json` states — and launch the agent only for answer-layer failures,
where the response text needs reading. A pure status mismatch (expected 200,
got 400) is already diagnosed by its verdict row; two agent launches to
re-read it is what the first field test spent.

Launch the `trace-analyzer` agent (registered as `evalup:trace-analyzer` when
the plugin is installed) per failed case, on its normalized trajectory
(raw spans when that view is insufficient) and its violated expectations.
Name every file by absolute path in the assignment: the case directory
`reports/<run-id>/cases/<id>/`, and the state dir's `adapter.yaml`,
`profile.yaml` and `findings.md` — the agent may not guess an answer path or
a SHOULD, and it treats the profile's `verified: true` as observed, not
correct (its brief says so; findings.md is the SHOULD source). Group
its outputs by failure mode, not by metric: same wrong-route pair, same tool
confusion, same rule violation, same missing clarification. Per cluster: count,
2 exemplar traces (expected vs actual), implicated surface, effort tag
(text-edit / code-change / architecture). Order by frequency × severity.
`optimize` consumes this output. The user just fixing the top cluster by hand
is success too. End by asking which clusters are real and which mean the case
label is wrong. Label corrections feed back to the dataset (bump
`dataset_version`).

For a hands-on look, build the viewer:
`python3 ${CLAUDE_PLUGIN_ROOT}/scripts/build_review_viewer.py reports/<run-id> [-a <annotations.jsonl>] -o <out.html>`
— a run directory is recognized and its `cases/*/verdict.json` records are
read without a flag (`--glob` is for any other layout). Pointed at a run
directory the viewer leaves every sealed holdout case off the page itself, so
never stage a filtered copy by hand — and never pass a `--glob` that matches
files other than `verdict.json` under a `--holdout` run's `cases/` (e.g.
`cases/*/*.json`): the seal filter reads only `verdict.json`, so any other
per-case file would put sealed material on the page. Both, and why: annotation-ux.md
§"Which files it reads", worth reading before the first invocation. Its
§"Theoretical-saturation stopping rule" owns the cadence — ≥100 traces, and
stop a sitting after ~20 that add nothing. `--label` runs on that same rule.

Zero gating failures → produce insight, not silence:
- what the passing suite CANNOT detect: unjudged dimensions, layers skipped
  this run, coverage gaps named in the dataset metadata;
- the non-gating / tracked-only results worth a look;
- the next investment: labeling, cases in a named gap, or a skipped layer;
- an offer to open-code a sample of PASSING traces in the viewer. A passing
  suite is not the same as a good one. That is how the issues a green suite
  cannot see get caught.

### Across runs, not just this one
`python3 ${CLAUDE_PLUGIN_ROOT}/scripts/run_history.py reports/ [-o reports/history.json]`
— every other comparison here is pairwise, so five consecutive "within noise"
diffs can hide a ten-point erosion. It groups `reports/` into comparable series
(`run/SKILL.md` §4's diff rules, plus the split and the enabled layers), flags a
monotone drift per metric, and journals each case's verdict history. Quote
`drift_flagged` and the journal's transitions, never a "better"/"worse" — that
is `stats.py`'s call on a specific pair, and this script deliberately makes
none. Read `excluded_runs` first: a run without `summary.status: ok` is not in
the series, which is usually why a series looks short.

### What it cost, beside whether it worked
`python3 ${CLAUDE_PLUGIN_ROOT}/scripts/score_cost.py <run> [--baseline <run>]
[--prices <state>/prices.json] [-o reports/cost.json]` — `stats.py` compares
binary verdicts only, so "2% better but 3× more expensive" reads there exactly
like a free win. This prices the tokens the trace already recorded and
summarizes latency as order statistics; with `--baseline` it adds the paired
diff (an exact sign test, an exact distribution-free interval for the median
difference, and an exact permutation test on the total — no bootstrap, per
`stats.py`'s no-approximation rule).

**It is not a second gate.** There is one definition of "better" in this
package and it is `stats.py`'s keep rule; cost is not in it and there is no
budget anywhere. Report the two side by side and let the user make the trade.
Read `by_verdict_change.unchanged` — that is the spend that bought nothing
measurable.

**The price table is declared, never guessed.** Prices go stale silently, so
nothing is bundled: write `<state>/prices.json` (shape in the script's
docstring; `as_of` and `source` are required) and match `gen_ai.request.model`
exactly, adding an `aliases` entry rather than hoping a prefix matches. A model
the table does not name withholds the **dollars** and keeps the **tokens**.
A **trace-less app cannot be priced at all** — `cases/<id>/trajectory.json` is
the only token source — and its latency block is still complete. Quote
`declared_limits` with any figure: cached tokens are not accounted for, TTFT is
not measured, and at `k > 1` the cost is one representative repeat's.

## `--label` — judge calibration, 15 minutes at a time
The 15 minutes is the **sitting**, not the total: labeling runs as repeated
short sessions until the numbers hold, so optimize for low friction per case.
Run rubric-format.md's two passes: discovery, ~30 cases, then a stratified
validation pass of ~100–200. Take every number from there — TPR and TNR and
Cohen's κ, never raw accuracy, ≥90% on a slice the judge prompt never saw.
Launch the `judge` agent (`evalup:judge` when installed) with profile.yaml's
`judge.model` as the model override (agents/judge.md), and record that model
in the calibration notes: calibration must measure the model a judged run would
use, not the agent's default.

The judge executes one rubric node per call, so label **per node**. A
case-level agree/disagree cannot say which criterion drifted. Per case:
1. Show: user message → app answer → the tool results it drew on (compact).
2. Show that node's provisional verdict, its quoted evidence and its one-line
   reason. `unknown` is signal, not a skip — that criterion is usually
   ambiguous for the case.
3. Ask: agree / disagree / "the rubric is wrong here".
4. On "rubric is wrong": edit the rubric NOW, bump its version, mark prior
   labels for a relabel-check. Split the validation labels **20/40/40** —
   few-shot examples, judge-prompt iteration, and one touched once at the end
   for the final number. That split is honest only because the prompt was never
   tuned against it. Every disagreement you resolve on it silently converts it
   into a training split.

Do not compute the numbers yourself. Append one JSONL line per labelled node —
`{case_id, rubric_id, node_id, label, judge_label, critique, reviewer, ts}`
(annotation-ux.md §"The calibration fields") — then run
`python3 ${CLAUDE_PLUGIN_ROOT}/scripts/score_agreement.py <annotations.jsonl> --write
<state>/judge/calibration.json`. That sidecar is what the runner reads; a
`calibrated` profile with no sidecar behind it does not open the judged gate.

Record the result **per rubric first**: `status.calibrated` and that rubric's
own numbers **as the scorer computed them**, in its own file (rubric-format.md
§File shape). Until it flips its
verdicts stay watermarked `PROVISIONAL`, and `optimize` refuses to target it.
Then roll the aggregate into profile.yaml's `judge.calibration`. Never set
`judge.status` by hand — it is derived, and one uncalibrated active rubric
holds the whole harness at `uncalibrated`
(`${CLAUDE_PLUGIN_ROOT}/skills/discover/references/profile-schema.md`
§`judge:`). Same-family judging is accepted consciously or not at all
(`judge.same_family_accepted: true`). When you record that flag, say that the
required cross-family spot-check audit comes with it (agents/judge.md). It is
not a surprise obligation to discover later.

Canaries run in every batch, and one scoring wrong voids that batch's numbers
(rubric-format.md §"The canary guard"). Recalibrate on that doc's cadence, with
10–20 fresh outlier cases. An asynchronous pass — a domain arbiter without
Claude Code open — takes the viewer instead, built with
`-a <calibration.jsonl>`. Its export panel produces exactly the JSONL this
flow consumes.

## `--mine` — real traffic → dataset (stage: traffic+)
Query the trace backend for recent conversations (adapter `traces` config).
Surface: failures users hit, routing flip-flops, tool loops, latency outliers,
and ordinary conversations unlike anything in the dataset (embedding distance).
Per candidate: show it, then propose the case with expected values from what
SHOULD have happened. When the adapter declares `may_contain_pii`, redact
before anything reaches a dataset file and show the redacted version for
approval. Origin: `real-trace:<id>`, `review.status: pending`,
`gating: false` until a human accepts the case. Real-seeded cases progressively retire
synthetic-only ones. A trace already annotated through the viewer is most of
this review already done — reuse it (annotation-ux.md, "the flywheel").

## `--unseal` — holdout access, deliberately inconvenient
Warn that looking spends the holdout's validity, append one JSON line —
`{"date", "mode": "unseal", "reason": "<why you looked>"}` — to the ledger,
then show what was asked. After 5 recorded looks, require a reseal. Rewrite
each viewed case's `split` from `holdout` to `full`, flip fresh unviewed cases
the other way, and bump `dataset_version`.

The ledger is a **`.jsonl` sidecar** beside the dataset
(`datasets/holdout-looks.jsonl`, from the run plan's `paths.holdout_ledger`),
**not** the dataset YAML — `${CLAUDE_PLUGIN_ROOT}/docs/runner-contract.md` §6
owns why. N=5 counts **looks**, not unseals: `run --holdout` and `run --full`
append to this same file (`${CLAUDE_PLUGIN_ROOT}/skills/run/SKILL.md` §4). Take
the running total by **counting its lines**, not the entries this skill wrote,
or the optimizer's repeated holdout gates spend the seal invisibly.

## Re-validate after any case edit
All three branches edit case files. `--cluster` ends in label corrections,
`--mine` writes new cases, and `--unseal` rewrites `split` on both the cases it
reseals and the ones it flips in. Those are exactly the edits `generate` §4
validates before handing a dataset over — done here later, by hand, with no
validation step in sight. So run the same check, after the edit and before the
next run:

```
python3 ${CLAUDE_PLUGIN_ROOT}/scripts/validate_cases.py --cases <suite.json> \
  --capabilities <profile.capability_matrix.json> --adapter <adapter.json> \
  [--manifest <dataset-fields.json>]
```

`python3 ${CLAUDE_PLUGIN_ROOT}/scripts/convert_suite.py <state-dir> --split-dir <work-dir>`
(run/SKILL.md §1 defines `<work-dir>`) writes the JSON inputs from the YAML — never transcribe them yourself
(run/SKILL.md §1). The linter takes JSON on purpose (its docstring says why),
and case-format.md owns the rules it enforces. `--manifest` matters most here
(pass it whenever `datasets/dataset.yaml` exists — convert_suite writes
`manifest.json` only then): it catches the `dataset.yaml` counts that a
reclassification, a deleted case or a reseal leaves stale. A reseal must
also come back clean on `holdout_not_sealed` and `missing_split`, the only
mechanical evidence the seal still holds.

## Rubric editing (any time)
Rubrics live in `rubrics/*.md` under the state location (default
`.evalup/`, or the adapter's `state_location`), versioned. Editing one
while grading is the expected workflow, not a detour. Criteria drift is what
grading real cases discovers. Bump the version and record the motivating
finding in `calibration_notes`. Runs record rubric versions, and a diff across
two of them is refused. Node kinds, ordering, the canary guard, the authoring
procedure and the file shape: rubric-format.md.