Checkpoint Promotion

---
name: checkpoint-promotion
description: Gate fine-tuned checkpoints with drift budgets, paired comparison, and forgetting checks before promotion. Use after a training run produces a checkpoint, when deciding whether a tuned model ships, or when a promoted model needs re-gating against updated goldens.
---

# Checkpoint Promotion

The Phase 5 gate for the whole
plugin: a checkpoint that trains
cleanly and beats its task metric
still doesn't ship without
clearing all four stages below.
`eval-harness-first` built the
suite re-run here — this skill is
where that suite's baseline
decides something.

**Input:** a trained checkpoint,
`eval/baseline-<model>.json` from
`eval-harness-first`, and the
frozen `eval/drift-suite.yaml`.
**Output format:**
`promotion-report.md` — the
four-stage evidence plus a
terminal `PROMOTE` or `REJECT`
verdict that `/finetune` Phase 5
and `/promote-checkpoint` consume
directly.

## The Four-Stage Gate

Each stage gates the next — a
failure at stage 2 means stage 3
doesn't run. Stages 2 and 3 share
one expensive inference pass, so
running them concurrently and
applying gate order at verdict
time is licensed on a
**deterministic** arena (nothing
saved by serializing); a
judge-based arena should still
wait for stage 2 first — that's
where the real savings are.

1. **Data-quality gate.** Before
   any eval touches the
   checkpoint: dedup the training
   set, check for eval-goldens
   leakage (the exact failure
   `trace-to-training-data`'s
   Hygiene section exists to
   prevent), and scan for label
   noise. A checkpoint trained on
   leaked goldens invalidates
   every later stage.
2. **Held-out + frozen
   capability-drift suite.**
   Re-run `eval-harness-first`'s
   `eval/drift-suite.yaml` —
   MMLU/GSM8K/IFEval plus 200–500
   domain-adjacent items — against
   the checkpoint and diff against
   `baseline-<model>.json` per
   benchmark against the Drift
   Budget table below.
3. **Paired arena vs. base.**
   Position-randomized judge,
   checkpoint vs. base model, same
   prompts — or the deterministic
   paired-comparison variant in
   `references/gate-templates.md`
   when every grader in the
   harness is deterministic (no
   LLM-judge; position
   randomization N/A there).
   **A holdout win that
   loses the live arena does not
   ship** — stage-2 numbers and
   stage-3 judgments must agree; a
   win on frozen goldens and a
   loss in paired comparison is a
   real signal, not a discrepancy
   to explain away.
4. **Canary.** 5–10% stratified
   rollout with auto-rollback for
   any checkpoint reaching
   production traffic. **Local-only
   users stop at stage 3** —
   skipping stage 4 for a local
   deployment is the correct
   stopping point, not a shortcut.

### Drift Budget

| Drift (pts) | Verdict |
|---|---|
| ≤1 | Noise — proceed |
| 2–5 | Rerun with seed variation before deciding |
| >5 | **HARD FAIL** — no exception for task gains |

The >5pt row governs regardless
of the others: a checkpoint that
gained 8 points on the target
task and lost 6 points of general
capability still fails here —
task improvement never buys back
a drift-budget breach.

**Item count derives from the
budget, not convenience:** the
strict n for a half-width under
half the 5pt hard-fail threshold
is ~1,300 at typical accuracy
(p≈0.7); n=200 is a pragmatic
floor (±6pt half-width at that
same p, n=50 ±13pt) — report the
half-width with every verdict,
and treat a margin smaller than
it as `REJECT (uncertain)`, not
PASS/HARD FAIL. Full math and a
5-run cautionary example:
`references/gate-templates.md`.

**RERUN is not a verdict.** A
2–5pt drift only ever produces a
`PROMOTE` or `REJECT` after the
seed-variation rerun completes —
`PROMOTE` requires landing back
at ≤1pt (noise); any rerun still
>1pt — 2–5pt band or >5pt breach
alike — resolves stage 2 to a
hard `REJECT`. No report may
reach the Verdict section with
stage 2 still showing `RERUN`.

## Catastrophic Forgetting

Unmanaged LoRA fine-tuning loses
real general capability, and
stage 2 is what catches it:

- **~43% knowledge loss
  unmanaged** — no replay, no
  regularization.
- **~10% with basic management**
  — some replay or a conservative
  LR.
- **~3% with replay + EWC** — the
  disciplined case.
- **10–30% general-data replay
  mix is the standard
  mitigation** — blend general-
  domain data into training
  rather than target-task data
  alone.

If a checkpoint hits the >5pt
hard fail in stage 2, work this
escalation ladder in order — the
one canonical order this skill
and `references/gate-templates.md`
both point to:

1. **Adjust the replay-mix
   fraction — swap rows, don't
   add them** (adding confounds
   fraction with total optimizer
   steps). Dose is not monotonic
   at small-run scale (<~100
   steps) — re-check drift after
   any swap.
2. **Lower the learning rate.**
3. **Fewer epochs.**
4. **A smaller LoRA rank** — the
   same rank/LR levers
   `lora-qlora-recipes` and
   `preference-optimization` tune
   for the training run, applied
   here in reverse.

This order is a default, not a
law: **remediation guidance from
a single before/after run pair
is a hypothesis** — label it
low-confidence once any lever
produces a reversal, and prefer
a seed-variation repeat over
trusting the next rung blindly.
A lever that clears the drift
breach but drops a
success-criterion metric below
target is a two-sided tradeoff
for a human, not a reason to
keep descending the ladder. Full
reasoning and the 5-run
trajectory behind both caveats:
`references/gate-templates.md`.

**Disclose drift-suite
instruction reuse.** A replay row
copying the drift harness's exact
instruction phrasing (not just
disjoint source items) makes that
benchmark's post-replay score an
upper bound — flag it
instruction-familiar, or re-probe
with a paraphrase, before
treating a near-budget pass as
clean.

## The Verdict

`promotion-report.md` covers all
four stages as sections and
**must end with a terminal
verdict: `PROMOTE` or `REJECT`**,
the evidence that produced it,
and exactly one top remediation
when the verdict is `REJECT`.
Template: `references/gate-templates.md`.
The terminal contract other
skills parse:

```
## Verdict

REJECT

Evidence: domain-adjacent drift
suite dropped 6.2pt (threshold:
>5pt hard fail) despite +8pt on
the target task.

Top remediation: swap the
replay-mix fraction from 10%
toward 20%, holding step count
constant.
```

- **REJECT is a result, not an
  error.** A checkpoint that
  fails stage 2's drift budget or
  stage 3's arena comparison did
  its job. Don't treat a REJECT
  as a failed run needing a rerun
  of this skill; it's the correct
  output of a working gate.
- **One remediation, not a
  menu.** Evidence sections may
  list everything observed; the
  verdict section names the
  single highest-leverage fix per
  the escalation ladder above. A
  report that hedges across three
  possible fixes hasn't done the
  prioritization this skill
  exists to do.
- **No auto-retraining.** This
  skill produces a verdict and a
  report, not a re-triggered
  training run. A `REJECT` hands
  the remediation back to a human
  decision at
  `finetuning-method-selection` or
  the relevant training skill.

## Related Skills

- `eval-harness-first` — owns the
  drift suite and baseline this
  skill re-runs and diffs
  against; no `baseline-<model>.json`
  means nothing to gate against.
- `quantized-export` — the only
  valid next step after a
  `PROMOTE` verdict.
- `preference-optimization` and
  `lora-qlora-recipes` — own the
  LR and rank levers in the
  Catastrophic Forgetting
  escalation path; this skill
  diagnoses the breach, those
  skills own the config that
  caused it.
- `dataset-curation` — owns the
  replay-mix construction recipe
  the escalation ladder's first
  rung applies.

Complete `promotion-report.md`
template with all four stages,
the drift-suite scoring table,
the paired-arena protocol (item
count, position randomization,
win-rate threshold), and a
replay-mix configuration example:
`references/gate-templates.md`.