Loupe Analysis

---
name: loupe-analysis
description: "Use when a work request asks for an analysis of a Loupe project, such as a cost report or an experiment comparison, or when a prompt names loupe-analysis or gives an analysis id."
---

# Running a Loupe analysis

An analysis reads the worker runs of one project, or the comparison of one experiment. It ends with a report document and a short list of proposals. The owner starts it on the Reports tab of the Analytics page. A bridge then runs you with the analysis id. The owner reads the report and creates or dismisses each proposal.

Change no code and no card. Read through the `loupe` MCP, write one document, and report. An analysis saves no graph to the Dashboard. The report may name the settings of a graph worth saving, and the owner decides whether to ask for it with `graph_save`.

## Procedure

1. Call `analysis_get` with the analysis id. Read `topic`, `scope.range`, `scope.experiment` and `state`.
2. When `state` is `done` or `failed`, stop. The analysis takes no second report.
3. When `topic` is `cost`, do the steps in [Find where the cost goes](#find-where-the-cost-goes).
4. When `topic` is `time`, do the steps in [Find where the time goes](#find-where-the-time-goes).
5. When `topic` is `experiment`, do the steps in [Compare the variants of an experiment](#compare-the-variants-of-an-experiment).
6. When `topic` is `host`, do the steps in [Find what the host did](#find-what-the-host-did).
7. For any other topic, do the steps in [Another topic](#another-topic), and stop.
8. Load the `loupe-documents` skill, then write the report as [The report](#the-report) says.
9. Call `analysis_report` with `analysisId`, the `documentId` from `document_create`, and your proposals.
10. Read the response. Its `state` must be `done`.

## Find where the cost goes

Pass `scope.range` as the `range` of each `metric_query`. Call `metric_list` first, and use only the units, statistics and groups that it gives.

1. Query `cost` with unit `run`, statistic `sum`, and group `stage`. Do it again with group `model`, then with no group. Note the `total` of each series and its `rows`.
2. Query `cost` with unit `card`, statistic `median` and `sum`, and group `card-type`.
3. Query `cost` with bucket `week` and no group, to see the trend.
4. Query `merge-rate` and `fix-rounds` with unit `card`, with the same groups as step 2.
5. Take the costliest series. Sort its `rows` by `value` yourself, because `rows` holds the newest 100 rows and no sort by value.
6. Call `worker_run_get` on the costliest runs, about five to ten. When the rows are not enough, call `worker_run_list` with `workKind` and read `metrics.costUsd`.
7. For each run, find why it cost much. Use the signs below.
8. Join cost to outcome. A cheap run on a card that did not merge, or that needed many fix rounds, is no saving.

| Sign | Where to read it |
|---|---|
| Many resumes | `runs` of `worker_run_get` holds the whole series |
| Long polling or waiting | `metrics.idleGapMs` against `metrics.durationMs` |
| Big subagent work | `metrics.subagentMs` |
| Repeated gate runs | `worker_run_tool_calls`, the same `signatures` many times, such as a test command |
| A large model on easy work | `model` and `usage` against the work kind |

## Find where the time goes

Time here is the time of the tool calls of the main session, and the time the worker spends thinking. Assume nothing about the stack of the project. Read the command signatures as they are.

Pass `scope.range` as the `range` of each `metric_query`. Call `metric_list` first, and use only the units, statistics and groups that it gives.

1. In `metric_list`, find the `bucket-time:<name>` entries. Each one is a bucket that has time on a run of the project. The bucket `other` holds the calls that no rule takes. `analytics_settings_get` does not list the rules, so infer the buckets from `metric_list`.
2. Query `duration` with unit `run`, statistic `median`, `p90` and `sum`, and group `stage`. Note the `total` of each series and its `rows`.
3. Query `duration` again with unit `run`, statistic `sum` and no group. A stage group leaves out the runs that belong to no card, and a bucket query counts them. Query each `bucket-time:<name>` with unit `run` and statistic `sum`. Compare each sum with this ungrouped `duration` sum.
   - A bucket with a large share is a cost of time.
   - A run with no bucket data has an unknown value, so note how many rows each series holds.
4. Call `worker_run_list` with `workKind` for the stage with the most time. Read `metrics`: `durationMs`, `toolTimeMs`, `modelTimeMs`, `idleGapMs`, `longestCallMs` and `toolCalls`. Sort by the value you study, because the list is newest first.
5. Call `worker_run_get` and `worker_run_tool_calls` on the longest runs, about five to ten. Page through each run while `hasMore` is true.
6. Find the slowest calls. Sort the calls by `durationMs`, and read their `signatures`.
7. Find repeated steps. Count the same `signatures` in one run, and across runs. The same step many times is a candidate for a cache, a smaller scope or a skip.
8. Find polling loops. A polling loop is a run of calls that repeat in a short time, with signatures such as `for`, `grep` or `sleep`, and `waitsOn` null. A call with `waitsOn` set reads a background task that it started earlier, so the wait is a known step. A loop with `waitsOn` null polls with no link to its task, so its time is pure waiting.
9. Separate the outliers that are not the fault of the worker. Name each one in the report, and leave it out of the proposals:
   - a call far longer than the other calls of its run, with no sign of work in it;
   - a gap that ends the run, such as an `idleGapMs` before the last call;
   - a machine that lost power or network, shown by a long gap with no call.
10. When the bridge has host data, compare the time of a run with the concurrency at that time. Slow runs that overlap many other runs point at the host. Do not require this data, and say so when it is missing.
11. When one signature hides many different steps, such as `git` for all subcommands, say so. The list of subcommand programs groups the second word of a command into the signature. Propose a change to `subcommandPrograms` as a card, and name the programs. The owner sets it with `analytics_settings_update`. Do not call that tool yourself, because an analysis changes nothing.
12. Find the signatures in `other` that take real time. Compare the sum of `bucket-time:other` with the `duration` sum. A signature that repeats in `other` and takes more than a few percent of the run time needs a rule.

### Proposals for time

1. Propose a rule of kind `bucket-rule` for a signature that falls in `other` and takes real time. Give `payload` as `{"pattern": "<glob>", "bucket": "<name>"}`.
   - The pattern is a glob over a call signature, such as `git *`, `grep` or `Agent`. In it, `*` matches any run of characters and `?` matches one. The pattern is at most 120 characters.
   - The bucket name matches `a-z`, `0-9`, `_` and `-`, at most 64 characters. Prefer a bucket name that already exists.
2. Propose a `card` for each fix, such as a cache, a faster step, or a polling loop that waits on the task. Cite the runs and the calls that show the time.
3. Give `estimatedSaving` as a short text in time, such as `about 3 minutes a run`, based on the numbers in the body.
4. State the sample size and your confidence in the report, as [The report](#the-report) says. A claim from fewer than ten runs is low.

## Compare the variants of an experiment

`scope.experiment` names the experiment. The comparison reads every card of the experiment, so `scope.range` is `all`. An experiment runs one kind of work with two or more variants, such as two models, and compares the cards each variant worked. Each experiment names one variant as its baseline, and the comparison reads every other variant against it. No comparison reads two non-baseline variants against each other.

### Read the comparison

1. Call `experiment_get` with `experiment` set to `scope.experiment`.
2. Call it again with `page` 2, 3 and on while `hasMore` is true. Keep every row of `cards`.
3. Read `baseline`. It is null when the experiment names no baseline. Then `headline.comparisons` is empty and the experiment has no verdict. Say so in the report, and go on to the confounders.
4. Read `confidenceLevel`. The ranges are 95% for one comparison, and widen to 1 minus 5%/N for N comparisons, such as 97.5% for two. Say the level in the report. A wider range needs more cards before a variant clears the baseline.
5. Read `variants`: the model, the weight, `cards`, `finishedCards`, `runs` and `costUsd` of each.
6. Read `metrics`. For each metric, note the range of each variant in `byVariant` and its `clear` flag. `clear` is true when every variant is clear against the baseline, false when one is not, and null when there is no baseline. Read the `parts` of `fix-rounds` too.
7. Read `headline`. `headline.goal` keeps only the metric, the direction and a `problem`. The results are in `headline.comparisons`, one for each variant other than the baseline. Read the `state` of each: `not-enough-cards`, `clear`, `not-clear` or `held`.
   - Read its `change`, its `cardsBehind` and the `guards` changes. Each change reads the variant against the baseline. Read `headline.baselineArm` for the baseline figure, and `headline.controls`.
8. Read `minFinishedCards`.
9. Note `unknownMetrics`. The rule declares these keys, and the server has no such metric. Say so in the report.

### Read the verdicts

1. For each comparison, state the variant, the baseline, the change and its range, and the `state`.
2. A `held` comparison is held. It is neither lost nor clear. Read `holdReason` and `heldBy`, and name the controls that hold it. A held comparison has no change, so give no verdict for that variant. A control on the baseline holds every comparison.
3. A `not-clear` comparison only leans. Say that it is not a result.
4. A `not-enough-cards` comparison needs more cards. Use `cardsBehind` to say how many.
5. Check `headline.goal.problem`. A value of `unknown-metric` or `no-direction` means the goal gives no comparison. Say so in the report.

### Check the sample

1. Compare `finishedCards` of each variant with `minFinishedCards`.
2. When a variant has fewer finished cards than the minimum, say that the sample is too small to call a result.
3. Say the same when no metric is clear, or when no comparison is clear.
4. In both cases, recommend more runs. Name how many more finished cards each variant needs to reach `minFinishedCards`.
5. Keep reading. The confounders still help the next analysis.

### Find the confounders

A confounder is a cause other than the variant that changes a figure.

1. Read the left-out cards, the rows of `cards` with a `leftOut` list. Count them by reason, and say what each reason removes from the figures.
2. In each variant, take the kept cards with the highest `costUsd`, about three to five. Take the cards with the most `runs` as well.
3. Call `worker_run_list` with the `cardNumber` of each card. Call `worker_run_get` on its costliest or longest runs.
4. Look for the signs in the table below. Name each confounder and the cards it touches.
5. Compare the card mix of the variants. Call `card_get` on a few cards of each variant, and compare their types and sizes. A variant that got the harder cards looks worse for that reason alone.

| Sign | Where to read it |
|---|---|
| A usage limit or a stop | `reason` and `state` of the run, such as `stopped` |
| Long waits | `metrics.idleGapMs` against `metrics.durationMs` |
| Resumes | `runs` of `worker_run_get` holds the whole series |
| Failed tool calls | `metrics.failedCalls` |
| Slow repeated tool calls | `worker_run_tool_calls`, the same `signatures` many times with a long `durationMs` |
| A shared machine under load | runs on one `bridgeId` that overlap in time |
| A power loss or a lost bridge | a run that ends `lost` |

### Recommend

1. Recommend a variant over the baseline, keep the baseline, or ask for more runs. Give the reasons. Base a recommendation on a `clear` comparison alone. A held comparison supports no recommendation.
2. Join cost with quality. A cheaper variant with a lower merge rate, or with more fix rounds, is no clear win.
3. Give the sample size of each claim, and say how sure you are: high, medium or low.
4. Say which confounders weaken the result, and how much.

### Proposals for an experiment

1. Propose at most three changes. Use kind `card` for each one.
2. Propose a change such as "Keep variant X for implement, and remove the experiment", or "Run 12 more cards before a decision".
3. Base `estimatedSaving` on the numbers, such as the cost per merged card of each variant and the cards a week.
4. In the `body`, cite the metric ranges, the sample sizes and the cards that show the result.

## Find what the host did

The host metrics come from samples that a bridge takes of its machine. Read them in `metrics` of `worker_run_list` and `worker_run_get`: `meanCpuPct`, `peakMemBytes`, `peakSwapBytes`, `concurrentRuns` and `onBattery`. `concurrentRuns` counts the runs on the same bridge whose times overlap, the run itself included. Loupe computes it from the start and end times of the runs, so an ended run has a value with no sample. The other four metrics come from samples. A null value in them means that no sample is known.

1. Call `worker_run_list` with `endedAfter` set to the start of `scope.range`, such as 30 or 90 days before today. Pass no `endedAfter` for `all`. Walk every page while `hasMore` is true, with `perPage` 100.
2. Count the runs whose `meanCpuPct`, `peakMemBytes`, `peakSwapBytes` and `onBattery` are all null. These runs have no sample. Say in the report that sampling can be off for these reasons. The instance flag `bridge.host_sampling_enabled` is off by default, and the `collect` key in the bridge `rules.yaml` can be `false`. A run that is shorter than the sample interval also has no sample.
3. When no run has a sample, skip steps 5 to 7. Do step 4, because `concurrentRuns` stays available with no sample. The report then says that no sample is known, and gives only the concurrency findings.
4. For each `workKind` and each `bridgeId`, compare runs with `concurrentRuns` 1 against runs with a higher count. Compare `metrics.durationMs`, and the share of runs whose `state` is `failed`, `timed-out` or `lost`.
5. Find the runs with `onBattery` true. Also find the runs that ended `stopped`, `failed` or `lost` on a bridge that was on battery near that time.
6. Call `bridge_host_samples` with the `runId` of each of those runs. A falling `batteryPct` with `onAc` false shows a drain. A gap in `sampledAt` before the end of the run can show that the machine slept or lost power. A bridge restart, a failed host read or sampling that went off also leave a gap. Name the cause as likely only when the battery fell before the gap, and say what else can explain it.
7. Find the runs with a high `meanCpuPct`, and the runs whose `peakMemBytes` is near `memTotal` of their samples. A `peakSwapBytes` above zero shows memory pressure. Group them by `workKind` and `bridgeId`.
8. Find a change for each problem. Examples are a smaller worker pool on a bridge, or a laptop bridge that stays on AC power. A heavy work kind that moves to another bridge is a third example.

| Sign | Where to read it |
|---|---|
| Too many runs at one time | `metrics.concurrentRuns` against `metrics.durationMs` and `state` |
| A machine on battery | `metrics.onBattery`, then `batteryPct` and `onAc` of `bridge_host_samples` |
| A machine that slept or lost power | a gap in `sampledAt` of `bridge_host_samples` before the run ended, after a falling `batteryPct` |
| CPU pressure | `metrics.meanCpuPct`, and `cpuPct` per core of `bridge_host_samples` |
| Memory pressure | `metrics.peakMemBytes` against `memTotal`, and `metrics.peakSwapBytes` |

## The report

1. Call `document_create` with the tag `analysis`. Use a title such as `Cost report: last 90 days`, `Time report: last 30 days`, `Experiment report: implement` or `Host report: last 30 days`.
2. Open with a summary of three to five lines. Give the main findings and the best proposal. A cost report gives the total cost and the main cost drivers. A time report gives the total time and the main time drivers. An experiment report gives the recommended variant or the runs still needed, and the main confounder.
3. Cite the numbers for each claim. Link the runs and the cards you read.
4. Give the sample size of each claim, and say how sure you are: high, medium or low. A claim from fewer than ten rows is low.
5. Keep advice concrete. Name the work kind, the model, the step or the command to change.
6. Say what the data cannot show, such as runs with an unknown cost or with no host samples.

## Proposals

1. Propose at most five changes. An empty list is valid when the data supports no change.
2. Use kind `card` for each one. A `time` analysis also uses kind `bucket-rule`, as in [Proposals for time](#proposals-for-time). An experiment takes at most three, as [Proposals for an experiment](#proposals-for-an-experiment) says.
3. Write a `title` of one line that says the change.
4. In the `body`, cite the run ids and card numbers that show the problem. Add the outcome data: merge rate and fix rounds for cost, or duration and failures for host.
5. Give `estimatedSaving` as a short text, such as `about $4 a week` or `about 2 failed runs a week`. Base it on the numbers in the body.

## Another topic

1. Call `document_create` with the tag `analysis`. Say in a few lines that this skill does not support the topic yet.
2. Call `analysis_report` with the `documentId` and an empty `proposals` list.