Harness Catalog Retrospective

# Harness Catalog Retrospective

> Monthly retrospective over skill-adoption telemetry. Reads `.harness/metrics/adoption.jsonl`, ranks the most-invoked, most-failing, and abandoned-mid-workflow skills, flags ever-invoked skills that have gone quiet, and reports how much of the catalog emits any telemetry at all. Produces a dated Markdown report and surfaces the highlights that warrant follow-up. Compounding-via-learning at the catalog grain.

## When to Use

- On a regular cadence (monthly is the intended grain) to review how the skill catalog is actually being used.
- After a milestone, to see which skills carried the work and which never fired.
- When deciding what to prune, fix, or promote in the catalog — the report feeds catalog-rationalization work.
- NOT as a real-time dashboard — for point-in-time lookups use `harness adoption skills` / `harness adoption recent`.
- NOT when `.harness/metrics/adoption.jsonl` is absent or empty (the report will be honest but empty; there is nothing to retrospect on yet).

## Process

### Iron Law

**Separate real signal from telemetry gaps before you recommend anything.** The single most common misread of this data is treating "no telemetry" as "abandoned." Adoption records are emitted only from instrumented entry points, so a skill with zero records is usually uninstrumented, not unused. The report's coverage line and the stale-skills section exist to keep that distinction visible — never collapse it.

---

### Phase 1: SCAN — Derive the Report

1. Run the retrospective command from the project root:

   ```bash
   harness adoption retrospective            # writes docs/retrospectives/<date>.md
   harness adoption retrospective --no-write # print to stdout without writing a file
   harness adoption retrospective --json     # structured output for further processing
   ```

   Optional flags: `--inactive-days <n>` (stale threshold, default 90), `--top <n>` (rows per section, default 10), `--out <path>` (override the output file).

2. The command reads `.harness/metrics/adoption.jsonl` via `readAdoptionRecords`, derives the report via `getCatalogRetrospectiveReport`, and renders it via `renderRetrospectiveMarkdown` (all in `@harness-engineering/core`). Coverage context is computed against the skills discovered under `agents/skills/claude-code/`; when that directory is absent (a consumer project), the coverage line is omitted rather than reporting a false zero.

3. The report is followed by a **Bayesian skill-effectiveness** section (`## Bayesian skill effectiveness`), computed by the skill scorer in `@harness-engineering/intelligence`. This applies the same Laplace-smoothed success rate (α = 1) the persona scorer uses, so a skill invoked once cannot claim 0% or 100% certainty. It surfaces the least-effective skills, failing skills, and abandoned-mid-workflow skills ranked sample-aware — the view you should act on when deciding what to fix or prune. In `--json` mode the same data appears under the `skillEffectiveness` key (`scores`, `failing`, `abandoned`).

4. If there are no records, say so plainly and stop: "No adoption telemetry found at `.harness/metrics/adoption.jsonl`. Nothing to retrospect on yet."

### Phase 2: INTERPRET — Signal vs. Noise

Read the report with these discriminations:

- **Coverage line first.** "N/total catalog skills have emitted telemetry" frames everything below it. A low ratio means the rankings describe the _instrumented_ slice, not the whole catalog. State the ratio explicitly in your summary.
- **Failing skills are not automatically broken.** Some commands fail _by design_ (a gate like `ci.check` returning non-zero on a real violation is the gate working). Before flagging a high failure rate as a problem, note whether the skill is a checker/gate whose failures are expected. Call out only failures that look like defects (crashes, unexpected non-zero on clean input).
- **Abandoned mid-workflow** uses the broadened definition: an explicit `abandoned` outcome, or a non-completed run that had already reached ≥1 phase. Small counts here are meaningful — a workflow skill people start and bail out of mid-way is a UX signal even at n=1.
- **Stale skills** are drawn only from _ever-invoked_ skills quiet ≥ the threshold. When the record window is shorter than the threshold, the section says so and reports nothing — do not present that emptiness as "everything is healthy."
- **Bayesian ranking over raw counts.** The top-failing and abandoned-mid-workflow tables rank by raw count; the Bayesian skill-effectiveness section ranks the same signals by Laplace-smoothed success rate, so a skill that failed 1/1 does not outrank one that failed 30/50. When the two disagree, trust the Bayesian view for prioritization — it discounts low-volume noise. A skill needs at least a couple of failures (or abandonments) before it appears in the Bayesian failing/abandoned tables at all.

### Phase 3: REPORT — Surface and Recommend

1. Confirm where the report was written (`docs/retrospectives/<date>.md`) or present the rendered Markdown.
2. Summarize the 3–5 findings that actually warrant action, each tied to a section and a number. Prefer specifics ("`cli/review-ci` failed 23/23 — likely a real defect, worth a look") over restating the tables.
3. When the coverage ratio is the dominant finding (e.g. most of the catalog is uninstrumented), name that as the top follow-up: instrumentation, not pruning, is the lever.
4. Do not delete, prune, or edit skills from this skill — it reports. Hand any pruning decision to the human or to catalog-rationalization work.

---

## Harness Integration

- **`harness adoption retrospective`** — the CLI entry point (a subcommand of the existing `adoption` command group). Writes `docs/retrospectives/<date>.md` by default.
- **`getCatalogRetrospectiveReport` / `renderRetrospectiveMarkdown` / `isAbandonedMidWorkflow`** — the pure aggregation + render functions in `@harness-engineering/core` (`packages/core/src/adoption/retrospective.ts`). Callers can supply a fixed `now` and `catalogSkills` for deterministic output.
- **`readAdoptionRecords`** — reads and parses `.harness/metrics/adoption.jsonl` (read-only; the file is appended by the adoption-tracker hook, never by this skill).
- **`computeSkillEffectiveness` / `detectFailingSkills` / `detectAbandonedSkills`** — the Bayesian skill scorer in `@harness-engineering/intelligence` (`packages/intelligence/src/effectiveness/skill-scorer.ts`). The skill-grain counterpart to the persona scorer: same Laplace smoothing and threshold-based detectors, applied to adoption records instead of graph outcome nodes. The retrospective command consumes these to render the Bayesian skill-effectiveness section and the `skillEffectiveness` JSON key.
- **`harness adoption skills` / `recent` / `skill <name>`** — the point-in-time siblings; this skill is the periodic, persisted counterpart.

## Success Criteria

1. The retrospective is derived from `.harness/metrics/adoption.jsonl` without mutating it.
2. The report ranks top-invoked, top-failing, and abandoned-mid-workflow skills, and flags ever-invoked stale skills.
3. The report includes a Bayesian skill-effectiveness section that ranks failing and abandoned-mid-workflow skills by Laplace-smoothed success rate, so low-volume skills do not dominate the recommendation.
4. The report states telemetry coverage (how much of the catalog emits any signal) when the catalog is discoverable.
5. The stale-skills section notes when the record window is shorter than the inactivity threshold instead of silently reporting zero.
6. The summary separates real signal (defect-shaped failures, mid-workflow abandonment) from telemetry gaps (uninstrumented skills) and from expected gate failures.
7. A dated report is written to `docs/retrospectives/<date>.md` (unless `--no-write`/`--json` is requested).
8. The skill recommends but never prunes, edits, or deletes catalog skills.

## Rationalizations to Reject

| Rationalization                                                                        | Reality                                                                                                                                                             |
| -------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| "756 skills have no telemetry, so 756 skills are abandoned — recommend deleting them." | Zero telemetry almost always means _uninstrumented_, not unused. The coverage line exists to prevent exactly this misread; the fix is instrumentation, not pruning. |
| "`ci.check` fails 97% of the time — it's broken."                                      | A gate returning non-zero on a real violation is the gate working. Check whether the skill is a checker before flagging its failures as defects.                    |
| "The stale section is empty, so the catalog is healthy."                               | If the record window is shorter than the threshold, the signal simply cannot fire yet. Empty ≠ healthy; report the window caveat.                                   |
| "n=1 abandonment is noise, ignore it."                                                 | A workflow skill users start and bail out of mid-way is a UX signal even at low n. Surface it; let the human weigh it.                                              |
| "I'll prune the low-usage skills while I'm here."                                      | This skill reports; it does not prune. Pruning is a human/catalog-rationalization decision.                                                                         |

## Examples

### Example: Monthly retrospective on the dogfood repo

**Context:** ~67 days of telemetry (3.2k records) across 50 invoked skills, run at month end.

**Phase 1 — SCAN:**

```bash
harness adoption retrospective
# → Catalog retrospective written to docs/retrospectives/2026-07-06.md
#   3246 records · 50 skills · window 67d
```

**Phase 2 — INTERPRET:** The report's overview leads with `Telemetry coverage: 5/761 catalog skills have emitted telemetry (756 never invoked)`. Top-invoked is dominated by `cli/*` commands; `cli/ci.check` shows a 97% failure rate; the abandoned-mid-workflow section lists four workflow skills at n=1; the stale section is empty with the note that the 67-day window is shorter than the 90-day threshold.

**Phase 3 — REPORT:**

```
Retrospective written to docs/retrospectives/2026-07-06.md. Findings that warrant action:

1. Coverage is the headline: only 5/761 catalog skills emit any telemetry. The rankings
   describe the instrumented slice, not the catalog. Top follow-up is instrumentation, not pruning.
2. `cli/ci.check` failed 97% (721/741) — but it is a gate; non-zero on a real violation is
   expected. Not flagged as a defect.
3. `cli/review-ci` failed 23/23 (100%) — defect-shaped, worth a look. Routed to debugging.
4. Four workflow skills were abandoned mid-run (n=1 each) — low volume, a UX signal to watch.
5. Stale skills: none — the 67-day window can't satisfy the 90-day threshold yet.
```

No skills were pruned or edited — the pruning decision is left to the human.

## Gates

- **No mutation of `adoption.jsonl`.** The telemetry file is read-only input.
- **No catalog edits.** This skill never adds, edits, or deletes skills — it only reports on them.
- **Coverage must be stated when discoverable.** If the catalog can be scanned, the summary must include the telemetry-coverage ratio so rankings are read in context.

## Escalation

- **When there are no records:** State that telemetry is absent/empty and stop. Suggest confirming the adoption-tracker hook is installed if telemetry is unexpectedly missing.
- **When coverage is very low:** Name instrumentation as the top follow-up and note that the rankings describe only the instrumented slice.
- **When a failing skill looks like a real defect:** Flag it specifically (skill, failure count/rate, why it looks defect-shaped) and route it to debugging or an issue, rather than burying it in the table.