Skill Counterfactual Audit

---
name: skill-counterfactual-audit
description: >
  Audit a skill by running a paired probe — the same task once with the
  skill loaded and once without — segment both traces into goal-directed
  phases, align phases, and emit a SIP report (surface anchoring,
  template copy, excess planning, task recovery, off-task artifact).
  Use whenever a skill is created, modified, or proposed for retirement.
  Pass-rate is BLIND to most skill effects: CTA (arxiv 2605.11946v1)
  shows a single skill can produce 522 measurable behavioural changes
  across 49 tasks while pass-rate moves only +0.3%. Triggers: "audit
  this skill", "is this skill helping", "retire skill", "before shipping
  skill", "behavioural impact of skill X", or any skill review event.
user-invocable: true
version: 1.0.0
format: 2025-10-02
triggers:
  - "audit this skill"
  - "is this skill helping"
  - "behavioural impact of skill"
  - "retire this skill"
  - "before shipping a skill"
updated: 2026-05-16
status: ACTIVE
source: arxiv 2605.11946v1 (Counterfactual Trace Audit / CTA)
---

# Skill Counterfactual Audit

## Why

When you ship a skill, pass-rate is the easy metric — and it's *blind* to almost everything that matters. CTA (arxiv 2605.11946v1) shows a single skill can produce **522 measurable behavioural changes** across 49 tasks while pass-rate moves only **+0.3%**. The interesting effects (surface anchoring on the skill's vocabulary, template copy without adaptation, excess planning loops, recovery from off-task drift, generation of off-task artifacts) live entirely below pass-rate.

The remedy is a paired-trace audit: run the same probe task twice, with and without the skill, and *compare the behaviours directly* rather than the headline metric.

## How

For each skill under audit:

1. **Curate a probe-task bank** (3-5 small tasks per skill domain). Each task should have:
   - A clear goal (a single instruction)
   - An expected category of behaviour
   - A "phase decomposition" (rough segmentation rules)
2. **Run paired** — same prompt, same task, run A *with* skill loaded, run B *without*.
3. **Segment each trace** — phases like {plan, retrieve, edit, verify, fix-loop, summary}.
4. **Align phases** — match B phases to A phases; mark new/dropped/displaced phases.
5. **Classify SIP (Skill Influence Pattern)** — for each aligned phase, label the effect:
   - **surface-anchoring** — A copies vocabulary/structure from the skill itself
   - **template-copy** — A reproduces a skill template literally without adaptation
   - **excess-planning** — A adds extra planning steps absent in B
   - **task-recovery** — A recovers from drift that B falls into
   - **off-task-artifact** — A produces artifacts unrelated to the goal
6. **Emit report** to `.planning/patterns/skill-audits/<skill>-<timestamp>.md` with phase comparison table + SIP distribution + retire/refine/keep recommendation.

## Recommendation rule of thumb

| Dominant SIP | Verdict |
|---|---|
| task-recovery + correct artifacts | **keep** (genuine value) |
| surface-anchoring with no task-recovery | **refine** (skill is leaking style, not capability) |
| template-copy on most tasks | **refine** (lower template specificity) |
| off-task-artifact on >25% of probes | **retire** (skill is causing distraction) |
| excess-planning with no completion delta | **refine** (skill adds ceremony) |

## When to skip

- The skill is a workflow guide that doesn't directly shape output (e.g., `loop`, `schedule`) — audit dominant SIPs would always be "task-recovery" trivially.
- No probe-task bank exists for the domain (build one first; defer audit).
- The skill was just created and has no historical baseline. Run the audit when the skill has been active for ≥3 sessions.

## Integration

- `done-retirement` — required gate before retiring a skill.
- `gsd-cleanup` — sweeps audits on a milestone-close cadence.
- Companion to `simplify`, `code-review` — those audit code; this audits skill behaviour.

## Cross-references

- Rosetta concept #9 (Execution-Grounded Selection) — paired-trace audit is the skill-domain instantiation
- College: `agent-systems / integration-evaluation / agent-paired-trace-audit`
- Related skills: `skill-integration` (manages the skill lifecycle that this audits)