Prompt Miner

---
name: prompt-miner
argument-hint: "[--harness all|claude|pi] [--hours <N>] [--since <YYYY-MM-DD>] [--until <YYYY-MM-DD>] [--last-n <N>] [--min-turns <N>] [--top <N>] [--attribution first|all] [--include-prompt-text] [--no-git] [--weights <json>] [--out <dir>] [--report-only] [--dry-run]"
disable-model-invocation: true
allowed-tools: Read, Grep, Bash, Edit
description: |
  Rank past prompts by session outcome and mine the markers that produce the
  best sessions. Runs the deterministic mine-traces.mjs engine over Claude + Pi
  JSONL traces, scores each session by a friction + ground-truth outcome proxy,
  ranks the initiating prompts, then synthesizes falsifiable prompt markers
  STRATIFIED by session type and proposes harness identity improvements
  behind a /retro-style propose-then-confirm gate. A cross-session, data-driven
  cousin of /retro. TRIGGER when: /prompt-miner invoked, or asked to "mine
  prompts", "rank prompts by outcome", "what prompt patterns work best", "mine
  session traces", "find good prompt markers", "analyze prompt quality".
---

# prompt-miner

Mine the harness's own session history to learn which prompt traits produce the
best sessions, and feed those learnings back into identity. This skill is
the **judgment layer** on top of the deterministic `mine-traces.mjs` engine: the
engine collects, scores, and ranks; this skill correlates prompt features against
outcome, mines **falsifiable markers**, and proposes durable lessons for approval.

It is a cross-session, data-driven cousin of `/retro`. Where `/retro` reflects on
the *current* conversation, `/prompt-miner` reflects on the *corpus* of past
sessions across both harnesses.

> `disable-model-invocation: true` suppresses **auto**-invocation only — the model
> will not fire this skill on its own. A user-typed `/prompt-miner` still runs the
> full body below, including the Step-3 LLM marker synthesis. There is no conflict:
> the deterministic engine produces an objective dataset; the LLM step interprets
> it.

## Privacy contract

This skill reads real session transcripts, which can contain secrets and private
content. The contract is non-negotiable:

- **Default output is feature vectors + metadata only — never raw prompt text.**
  The engine omits `promptText` unless `--include-prompt-text` is passed.
- `--include-prompt-text` applies a redaction pass (line-level token patterns +
  block-level key bodies) and prints a `WARNING` banner. Use it only when you must
  read the prompt wording, and never commit the result.
- All artifacts land in ephemeral scratch under `$TMPDIR`, outside the repo. Never
  stage, commit, or paste a transcript or an `--include-prompt-text` report.
- The engine never writes into the repository. Its only outputs are the scratch
  artifacts under `$TMPDIR`.

## When to use

- `/prompt-miner` invoked to mine the corpus for prompt-quality markers.
- Periodically (the daily `crons/prompt-miner.md` cron drives the unattended path;
  this skill is the interactive path).

## When NOT to use

- **`/retro`** — reflects on the current conversation, not the historical corpus.
  Use `/retro` to close a session; use `/prompt-miner` to learn across sessions.
- **`/audit context` / `/audit skills` / `/wiki lint`** — those score harness
  artifacts (context budget, skills, wiki). `/prompt-miner` scores *prompts*.

## Result tag

Announce exactly one human result tag at the end of the run:

```
RESULT: MINING-COMPLETE | DRY-RUN | NO-SESSIONS | NO-CORPUS
```

| Tag | Meaning |
|-----|---------|
| `MINING-COMPLETE` | The engine ran, markers were mined, and the propose-then-confirm gate ran. |
| `DRY-RUN` | `--dry-run` was passed: dataset computed and printed; nothing written; no gate. |
| `NO-SESSIONS` | The engine found zero in-window sessions (`sessionsScanned == 0`). Stop after Step 1. |
| `NO-CORPUS` | Sessions exist, but no session-type stratum reaches the `sessions_supporting ≥ 10` floor (see `references/markers.md`). Report and stop before proposing markers. |

`NO-CORPUS` is distinct from "no markers crossed the bar" (`NO-CANDIDATE`): the
former means the corpus is too small to mine *anything* reliably.

## Steps

### Step 1 — Run the engine

Run the deterministic engine, forwarding the user's arguments verbatim. Use the
**array form** so a `--weights '{...}'` JSON argument stays a single token rather
than being word-split:

```bash
args=($ARGUMENTS)
node "${CLAUDE_SKILL_DIR}/scripts/mine-traces.mjs" "${args[@]}"
```

The engine writes `prompt-miner-<UTC-date>.json` + `.md` to `--out`
(default `$TMPDIR/oh-prompt-miner/<UTC-date>/`), unless `--dry-run` was passed (it prints the JSON
dataset to stdout and writes nothing). The flag surface (defaults in parens):

- `--harness all|claude|pi` (all), `--since`/`--until` (YYYY-MM-DD),
  `--hours N` (precedence over `--since`), `--last-n N`, `--min-turns N` (2),
  `--top N` (15), `--attribution first|all` (first).
- `--include-prompt-text` (off — see Privacy contract), `--no-git` (stub
  ground-truth bonus to 0), `--weights '<json>'`, `--out <dir>`,
  `--report-only`, `--dry-run`, `--max-file-mb N` (50).

If `manifest.sessionsScanned == 0`: announce `RESULT: NO-SESSIONS` and stop. If
`--dry-run` was passed: read the printed dataset, optionally summarize the
top/bottom ranked sessions, announce `RESULT: DRY-RUN`, and stop (no marker
proposals, no identity writes).

### Step 2 — Read the dataset

Read the emitted `prompt-miner-<UTC-date>.json` (or the `--dry-run` stdout). The
shape is documented in `references/report-schema.md`:

- `manifest` — `sessionsScanned`, `sessionsRanked`, `toolErrorsTotal`,
  `malformedLines`, `skippedFiles`, `weights`, `window`, `scoreModel`,
  `ceilingSaturation` (per-stratum `{ atCeiling, total }` census over the rankable
  population — how much of each session type sits on the clamp ceiling).
- `sessions[]` — ranked (`score` desc), each with `score`, `scoreUncapped`
  (the same value before the 0..100 clamp — the correlation scale for Step 3),
  `scoreBreakdown`, `sessionType`, and a `features` vector (the 13
  `markerFeatureKeys`).
- `unranked[]` — `noHumanPrompt` / below-`minTurns` sessions (kept, not ranked).
- `weaknesses[]` — metadata-only `WH-<NNN>` harness-weakness records clustering
  repeated failure signals across the corpus; deterministic, and **never** carries
  prompt text (`supporting_traces` = session-id metadata only). See
  `references/report-schema.md`.

Verify every score is reconstructable from its `scoreBreakdown` before trusting
the ranking — the score is a **heuristic proxy**, not a verdict (see
`references/scoring.md`; `correctionDensity` is the highest-variance signal).

### Step 3 — Mine markers (stratified by session type)

For each **session-type stratum** (`impl`, `retro`, `query`, `audit`, `cron`,
`other`) — never pooled across types (pooling manufactures Simpson's-paradox
artifacts; see `references/markers.md`) — correlate each feature in
`markerFeatureKeys` against the session `scoreUncapped` — **not** `score`, which is
censored above 100 (see `references/markers.md`). Emit each marker in the exact
falsifiable schema from `references/markers.md`:

```json
{
  "feature": "hasAcceptanceCriteria",
  "direction": "positive",
  "threshold": true,
  "sessions_supporting": 14,
  "sessions_contradicting": 3,
  "effect_size": 0.41,
  "effect_size_capped": 0.36
}
```

A marker is **reportable** only when, within a single stratum,
`sessions_supporting ≥ 10` **and** `effect_size ≥ 0.3`.

**Stability guard.** Compute `effect_size_capped` — the same statistic against the
clamped `score` — for every marker. A marker whose two scales disagree on sign, or
where exactly one of `|effect_size| ≥ 0.3` and `|effect_size_capped| ≥ 0.3` holds,
is `UNSTABLE`: **report** it with both values and the reason, but do **not** carry
it into Step 4 — it is not promotable on either scale, earns no identity proposal,
and files no issue. See `references/markers.md` § Stability guard.

**Corpus-size gate.** If **no** session type reaches the `sessions_supporting ≥ 10`
floor: announce `RESULT: NO-CORPUS`, report that the corpus is too small to mine
reliably, and stop — do **not** propose markers from a thin corpus. If strata are
large enough but nothing clears both thresholds this run, report `NO-CANDIDATE`
and stop (no identity proposals).

### Step 4 — Propose-then-confirm (mirrors `/retro`)

Only run this step when reportable markers exist and `--report-only` / `--dry-run`
were **not** passed. Translate each reportable marker into a candidate lesson,
then gate it exactly like `/retro` (`.claude/skills/retro/SKILL.md` § 6):

1. **Qualify filter.** Drop any candidate that is a secret, raw command output, a
   step-by-step plan, or anything re-derivable in under a minute.
2. **Dedup against existing probes.** For each surviving candidate, grep
   `.oh/evals/probes/` for a probe that already asserts the same invariant; if it
   is already captured, link or skip — never double-write (this is the same dedup
   `/retro` performs in its qualify filter).
3. **Promotability.** A marker that is merely descriptive ("this corpus shows X
   prompt trait correlates with better `<type>` sessions") is **reported, not
   promoted** — say it in the report and stop there. Only a marker that has
   generalized across many sessions into a prescriptive principle ("always include
   acceptance criteria") earns a proposed probe under `.oh/evals/probes/` — and a
   probe is **never** auto-written.
4. **Propose, then wait.** Present the block and stop until the user responds:

   ```
   Proposed probe(s) under .oh/evals/probes/:
   - <probe name> asserts <prescriptive principle> [prompt-miner · <stratum>] — basis: <one clause>

   Type APPROVE to record, SKIP to discard any item, or EDIT <n> <new text> to revise.
   ```

5. **Record approved items.** On `APPROVE`, add each approved probe proposal to the
   run report in `$TMPDIR`, beside the weakness records and the ranked marker
   table. `/prompt-miner` writes no tracked file; an approved proposal becomes a
   real probe only through `/spec`, which builds and gates it. `--report-only` and
   `--dry-run` skip this step entirely.

Announce `RESULT: MINING-COMPLETE` once the gate has run.

## Anti-patterns

- **Pooling across session types.** Markers MUST be stratified; a marker that holds
  only after pooling is not reportable.
- **Mining a thin corpus.** Below the `sessions_supporting ≥ 10` floor, emit
  `NO-CORPUS` and stop. Do not manufacture noise-driven markers.
- **Committing transcripts.** Artifacts are gitignored; never stage them, and never
  commit `--include-prompt-text` output.
- **Auto-promoting a marker.** Step 4 is propose-then-confirm. Never record a probe
  proposal without an explicit `APPROVE`, and never from a single run's evidence.
- **Word-splitting `--weights`.** Always invoke the engine via the `args=($ARGUMENTS)`
  array form so the JSON stays one token.
- **Inventing a file to save a marker in.** A descriptive marker that does not
  generalize is reported and dropped. Do not create a ledger or a dated note to
  hold it.

## References

- `references/scoring.md` — the friction + ground-truth scoring model.
- `references/markers.md` — the feature taxonomy + falsifiable marker schema + thresholds.
- `references/report-schema.md` — the emitted JSON/MD dataset shape.
- `references/pi-parity.md` — why `.pi/skills/prompt-miner` needs no byte copy.