Post Launch Refine Loop

---
name: post-launch-refine-loop
description: >-
  Sets up the post-launch improvement loop for a shipped AI product: ship behind human sign-off,
  track how much customers edit outputs, re-run the full eval set on every new model, and raise
  autonomy only on eval scores. Use when the user asks what to do after launch, whether to upgrade
  to a new model, whether to drop human sign-off, or how to set up the improvement loop. Not for
  choosing which metrics to track (use metrics-that-matter).
type: generator
supersedes: none
---

## What it does

Turns a shipped product into a loop that compounds. It takes a v1 that is live behind a human sign-off gate and stands up four stations: ship (the current autonomy level and its gate), observe (instrument tweak-time — the edit distance between what the system proposed and what the human committed), learn (re-run the entire golden set from `eval-first-spec` on every model release and on a fixed floor cadence), and refine (move autonomy up or down strictly on the scored eval result). The artefact is the filled `template.md`: a cadence doc that names what runs, how often, who owns it, and the exact numbers that move autonomy. A loop that raises autonomy on a hunch is not this skill; it is the failure this skill exists to stop.

## The Icarus reframe

The generic post-launch loop is build-measure-learn with an engagement dashboard: usage goes up and to the right, everyone nods, nothing about the product's trustworthiness is known. Icarus replaces the vanity metric with **tweak-time** — a behaviour signal (0.7 on the ladder), read from production traces, that measures how much a customer edits an output before accepting it; tweak-time falling to zero is the product earning the right to act. It replaces "ship it and move on" with a standing regression harness: a new model release is a **regression risk until the eval set re-scores it**, not a free upgrade, so the whole golden set re-runs before any model change reaches production. And it replaces the autonomy promotion-by-confidence with a **ratchet driven by evidence in both directions** — the L0–L4 level from `eval-first-spec` climbs one rung only when the eval re-run passes that rung's derived failure rates over N real cycles, and drops when a model release regresses them. Tweak-time nominates; the eval confirms; feelings never vote.

## When to use / When NOT

Use it once a product is shipped or in a paid pilot, behind a sign-off gate, WITH an eval set and an autonomy ladder already produced by `eval-first-spec`. Trigger phrases: "what happens after launch", "improve the product", "set up the loop", "a new model came out — should we upgrade", "can we raise the autonomy level / drop the sign-off yet".

Hard precondition: no eval set, no flywheel. The learn station has nothing to re-run without the 20-case golden set, and the refine station has no ladder to climb without the L0–L4 levels. If those do not exist, stop and route to `eval-first-spec`. Do not invent an eval set here.

Do not use it for:

| Request | Belongs to |
|---|---|
| "Which metrics / what's our North Star / does retention flatten / cost-per-outcome dashboard" | `metrics-that-matter` (08, sibling). It picks the metric set; this builds the ship→observe→learn→refine cadence and the autonomy ratchet. This skill *reads* tweak-time and cost-per-outcome; that skill decides the wider scorecard. Route metric selection there. |
| "Turn these production traces into interviews / what unmet need do the logs reveal" | `trace-to-interview` (08, sibling). It mines the qualitative discovery question from one trace; this reads traces as a quantitative behaviour signal (tweak-time) and as the eval-regression check. Same raw feed, different question. |
| "Write the golden cases / set the autonomy level / define what working means" | `eval-first-spec` (07). It *creates* the eval set and ladder before build; this *re-runs* them after launch. If there is no eval set, you are here too early. |
| "Keep the opportunity tree alive / set up continuous discovery" | `continuous-discovery-engine` (02). That is the discovery-side loop; this is the delivery/quality-side loop. They run beside each other (dual-track) and share the trace feed; neither owns the other. |

## Method

Fill in `template.md`. Check the precondition, then the four stations, then the compounding check and the kill line.

### Station 0 — Precondition gate

Before anything, confirm three things exist. If any is missing, the flywheel cannot turn; name the gap and route.

| Precondition | If missing |
|---|---|
| A golden set (the 20+ pass/fail cases from `eval-first-spec`) | Route to `eval-first-spec`. There is nothing to re-run. Stop. |
| A chosen autonomy level L0–L4 with per-mode acceptable rates | Route to `eval-first-spec`. There is no ladder to climb. Stop. |
| The product is live behind a human sign-off gate | If it ships with no sign-off, that is a `guardrail-design` gap, not a flywheel. |

### Station 1 — Ship (behind sign-off)

State the starting position, not the destination. Record the current autonomy level (usually L1 Drafts or L2 Acts-on-approval — the level `eval-first-spec` shipped) and the sign-off gate: every output is reviewed and committed by a named human before it acts. The flywheel's whole job is to earn the right to loosen this gate, one measured rung at a time. Autonomy starts low on purpose.

### Station 2 — Observe (tweak-time)

Instrument one behaviour: **tweak-time** — how much the human edits the system's proposed output before accepting it. Read it from the production trace, not a survey. Track three numbers per output type:

- clean-accept rate: share of outputs committed with **zero** edits `[Fact, from traces]`
- median edit fraction: how much of the accepted output was changed
- the trend over weeks: falling, flat, or rising

Read the bands like this:

| Clean-accept rate | Trend | Reading and action |
|---|---|---|
| any | falling over weeks | The product is regressing or drifting. Freeze autonomy, run the learn station now, find the cause before anything else. |
| < 30% | flat | Still a draft tool. Hold autonomy. Cluster the edits — a systematic edit is a missing feature or a wrong default, not noise. |
| 30–70% | rising | Earning trust. Hold the level. Watch which failure modes sit behind the remaining edits. |
| > 70% | sustained over the level's N cycles | **Nominates** a level-up. It does not grant one. Carry the nomination to Station 4; the eval decides. |

Tweak-time is a nomination, never a promotion. Everything else (retention, cost-per-outcome, North Star) is `metrics-that-matter`'s scorecard — reference it, do not rebuild it here.

### Station 3 — Learn (re-run the WHOLE eval set)

This is the station the kill line guards. Re-run every golden case, not a sample, on any of these triggers:

| Trigger | Why it is mandatory |
|---|---|
| Any model version change (provider release, model swap) | A new model can silently regress a case that passed last week. "Obviously better" is not a measurement. Re-run **before** it reaches production. |
| Any change to the core prompt / system message | Same input, changed reasoning, unknown result. |
| Any change to retrieval or the corpus | The evidence the model reasons over moved. |
| A fixed floor cadence (monthly) with no change at all | Catches drift, data-distribution shift, and provider-side silent updates. |

On each re-run, log to a standing eval ledger and compare per-mode pass rates against two baselines: the last run, and `eval-first-spec`'s derived `acceptable_rate(mode)`.

| Date | Model / change | Miss | False alarm | Confidently wrong | ... | Δ vs last | Action |
|---|---|---|---|---|---|---|---|
| _[date]_ | _[what changed]_ | _[rate]_ | _[rate]_ | _[rate]_ | | _[better/worse]_ | _[ship / hold / demote]_ |

The regression rule overrides aggregate improvement: a regression on any safety-critical mode (a Miss in gas safety, a Confidently-wrong in a decision that moves money) blocks the release or demotes autonomy **even if the overall pass rate went up**. Net-better does not buy a safety regression.

### Station 4 — Refine (promote autonomy by eval, never by feel)

Move autonomy along the `eval-first-spec` ladder using the scored eval, gated by tweak-time. One rung at a time; no skipping.

| Condition | Autonomy action |
|---|---|
| Tweak-time nominates (>70% clean-accept sustained N cycles) AND an eval re-run **at the next level's** acceptable rates passes over N real cycles, no safety-mode regression | Promote one rung |
| Eval re-run holds the current level's rates | Hold |
| Eval re-run regresses below the current level's `acceptable_rate` (common after a model release) | Demote / re-gate behind sign-off until a passing re-run |
| Tweak-time nominates but no eval was re-run at the next level | **Hold.** The nomination is 0.7 evidence for a hypothesis; the promotion needs the scored eval. Feel is not a vote. |

N is the same "gate to earn the next level" that `eval-first-spec` Part 3 already set (e.g. Miss ≤ 3% over 40 real shifts). Do not re-derive it; carry it forward.

### Compounding check and kill line

A loop is only a flywheel if each turn lowers `cost_per_outcome` (from `eval-first-spec` Part 4) — fewer human review minutes as autonomy rises, or fewer retries as quality rises — freeing margin for the next discovery cycle. If cost-per-outcome is flat turn over turn while usage climbs, it is a loop spinning in place. Name it.

**Kill line.** The cadence doc ships only if (1) the learn station re-runs the whole eval set on every model release, and (2) every autonomy change is tied to a scored eval result. A loop that promotes autonomy by feel, or that upgrades a model with no re-run, is auto-fail. Report the gap; do not dress a vanity loop as a flywheel.

## Evidence standard

This skill weights behaviour and money over opinion. The ladder: money moved 1.0 → behaviour observed 0.7 → artefact shown 0.5 → verbal commitment 0.3 → opinion 0.1.

| Signal in this loop | Rung | Use |
|---|---|---|
| Cost-per-outcome fell / customer expanded spend | 0.7–1.0 | Proves the flywheel compounds. |
| Tweak-time (clean-accept rate from traces) | 0.7 | Nominates an autonomy level-up. Never grants it. |
| Eval re-run pass rate on the real golden set | 0.5–0.7 | The only signal that moves autonomy. |
| "The team feels it's ready" / "users like it" | 0.1 | Cannot move autonomy. Named as opinion, parked. |

An autonomy promotion must rest on the scored eval (≥ 0.5). No promotion is ever justified by the 0.1 "feels reliable now." Tag every rate `[Fact]` (measured on real traces or the real golden set) or `[Assumption]` (projected until a real re-run confirms it).

## Gotchas

Promoting on tweak-time alone. A 90% clean-accept rate feels like proof, but it is a proxy. Until the golden set is re-scored at the higher level's acceptable rates, the level is not earned. Tweak-time nominates; the eval confirms. Skipping the eval is promoting by feel with a number attached.

Treating a model upgrade as a free win. Skipping the re-run because the new model is "obviously better" is the exact failure the kill line names. A silent regression on one mode can outweigh every aggregate gain, and you will not see it without re-running all 20 cases. Newer is unknown, not better, until scored.

Tweak-time survivorship at higher autonomy. You only observe edits on outputs a human reviewed. At L3+ the un-reviewed outputs generate no tweak-time signal — so the metric blinds you exactly where cost-of-error is highest. Above L2, supplement tweak-time with the flagged-exception review rate and periodic audits; do not trust the clean-accept rate alone.

Clean-accept gamed by emptier output. If the model learns to hedge — vaguer, safer outputs a human cannot disagree with — tweak-time falls for the wrong reason. Guard: the golden set's must-refuse and adversarial bands must still pass. A rising clean-accept rate with a falling golden pass rate is drift, not trust.

Vanity loop. Usage up, edits still 100%, is a product people are forced to use, not one they trust. If tweak-time and cost-per-outcome are flat, "engagement is growing" is noise.

## Examples

`examples/sample.md` — the post-launch flywheel for Mentix's shift-handover machine-risk digest, picking up exactly where `eval-first-spec` shipped it at L1: tweak-time instrumented on the digests (clean-accept climbing 41% → 78%), a model-release re-run that regresses the Miss rate and correctly holds autonomy despite a higher aggregate score, and an L1→L2 promotion granted only after the re-scored eval clears the derived gate — with cost-per-outcome falling $3.55 → $2.90/shift as the compounding proof.

## Related skills

`eval-first-spec` (07) — creates the golden set and the L0–L4 ladder that this skill re-runs and climbs. Upstream, once, before build. If it has not run, this skill has no inputs. They are two ends of one asset: that skill writes the eval, this one keeps it alive.

`metrics-that-matter` (08, sibling) — owns the metric scorecard (AARRR, the retention curve that must flatten, the North Star, cost-per-outcome to the cent). This skill consumes cost-per-outcome and tweak-time; it does not choose the scorecard. Reference it for retention and North Star; do not restate the M9 retention pack.

`trace-to-interview` (08, sibling) — turns one production trace into a discovery interview. Same raw feed as this skill's observe station, opposite question: it asks "what unmet need does this trace reveal", this asks "is the product earning autonomy". Route the qualitative discovery question there.

`continuous-discovery-engine` (02) — the discovery-side loop (the living opportunity tree). This is the delivery-side loop. They run dual-track and share the trace feed; keep the boundary.

Supersedes: none. New skill; no prior post-launch flywheel exists in the pack to replace.