Skill Optimize

---
name: skill-optimize
description: Run a structured evaluate-analyze-improve cycle on any GAAI skill to measure quality, detect regressions, and propose targeted improvements. Activate when a skill needs baseline evaluation, after SKILL.md modifications, or when friction-retrospective flags a skill.
license: ELv2
compatibility: Works with any filesystem-based AI coding agent
metadata:
  author: gaai-framework
  version: "1.0"
  category: cross
  track: cross-cutting
  id: SKILL-CRS-026
  updated_at: 2026-03-20
  status: experimental
inputs:
  - target_skill_path: path to the SKILL.md being optimized
  - trigger: baseline | regression | skill-update | friction-flagged
  - evals_file: (optional) path to existing evals.yaml — if absent, Step 1 creates one
  - corpus_dir: (optional) path to existing corpus outputs — if absent, Step 2 creates samples
outputs:
  - "{skill-dir}/eval-corpus/evals.yaml (created if absent)"
  - "{skill-dir}/eval-corpus/score-{iteration}.yaml — score report per iteration"
  - "{skill-dir}/quality/ledger.yaml — persistent quality ledger (accumulated across runs)"
  - "Improvement proposal (inline, for human review — NEVER auto-applied)"
---

# Skill Optimize

## Purpose / When to Activate

Activate when:
- A skill needs a baseline quality measurement (no ledger.yaml exists yet)
- A SKILL.md has been modified and a before/after regression check is needed
- `friction-retrospective` flags a skill as a recurring friction source
- An eval cycle is needed after a manual skill update

This skill formalizes the Skill Optimize protocol referenced by `eval-run` (SKILL-CRS-025). It runs the full evaluate-analyze-improve loop with mandatory human gates at every modification step.

**Scope:** Any GAAI skill with measurable quality criteria — not limited to content-production skills. The inline Skill Optimize protocol in `discovery.agent.md` remains the agent's orchestration logic; this skill provides the structured execution procedure.

---

## Process

### Step 1 — Eval authoring

If no `evals.yaml` exists for the target skill:

1. Read the target `SKILL.md` in full.
2. Identify measurable quality criteria from the `Quality Checks` section.
3. Author `evals.yaml` following the `evals-format.md` spec (see `eval-run/references/evals-format.md`).
4. Include a minimum of 5 assertions with a mix of `code` and `llm-judge` types.
5. Store the file at `{skill-dir}/eval-corpus/evals.yaml`.

**HUMAN CHECKPOINT:** Present the drafted `evals.yaml` for validation. Do not proceed until approved. If rejected, revise based on feedback and re-present.

### Step 2 — Corpus generation

If no corpus outputs exist in `{skill-dir}/eval-corpus/`:

1. Identify the skill's expected inputs from its `inputs:` frontmatter and Process section.
2. Produce 2-3 representative outputs by simulating the skill's expected inputs.
3. Store each output in `{skill-dir}/eval-corpus/` with naming `corpus-{N}.md`.

If corpus outputs already exist (from prior runs or real production), use those. Prefer real outputs over synthetic when available.

### Step 3 — Baseline evaluation

1. Invoke `eval-run` (SKILL-CRS-025) with each corpus output against the `evals.yaml`.
2. Compile per-output scores into `{skill-dir}/eval-corpus/score-baseline.yaml`.
3. Record the aggregate: `passed / total` and `pass_rate`.

### Step 4 — Error analysis

For each failed assertion across all corpus outputs:

1. Identify the root cause in the target SKILL.md: which step, which instruction.
2. Classify the failure:
   - `instruction-gap` — the skill doesn't instruct what is needed
   - `instruction-ambiguity` — the skill instructs ambiguously
   - `eval-design-error` — the assertion is flawed, not the skill
   - `model-limitation` — the model cannot reliably produce what is asked
3. Produce `{skill-dir}/eval-corpus/error-analysis.md` with per-assertion findings.

### Step 5 — Improvement proposal

Based on the error analysis:

1. Propose specific, minimal SKILL.md edits addressing `instruction-gap` and `instruction-ambiguity` failures.
2. For `eval-design-error` failures: propose evals.yaml corrections instead.
3. For `model-limitation` failures: document as known limitations, do not propose changes.
4. Present the proposal to the human.

**HUMAN CHECKPOINT:** The human approves, modifies, or rejects the proposal. NEVER auto-apply SKILL.md changes.

If approved:
1. Apply the edits to SKILL.md.
2. Re-run Steps 3-4 as a new iteration (score file: `score-{iteration}.yaml`).
3. Compare against previous iteration scores.

### Step 6 — Ledger update

After each iteration (including baseline), append an entry to `{skill-dir}/quality/ledger.yaml`:

```yaml
iterations:
  - id: {N}
    date: {ISO 8601}
    trigger: {trigger input value}
    score:
      passed: N
      total: N
      pass_rate: 0.XX
    delta_vs_previous: +/-0.XX  # null for baseline
    failed_assertions: [ANN, ...]
    action_taken: "{description of SKILL.md change, or 'baseline — no action'}"
status:
  current_pass_rate: 0.XX
  trend: improving | stable | degrading
  slo_target: 0.85
  error_budget_remaining: 0.XX
```

The ledger is **append-only** — iteration history is never deleted or overwritten.

For ledger format details, see `references/ledger-format.md`.

### Step 7 — Trend detection

After updating the ledger:

1. If `trend: degrading` over 3+ consecutive iterations: **escalate to human** with full history and recommendation.
2. If `error_budget_remaining < 0` (pass rate below SLO for 3+ iterations): flag the skill as `needs-optimization` in the ledger status. This blocks new deliveries using this skill until the human resolves it.
3. If `trend: improving` or `stable`: report status inline and complete.

---

## Quality Checks

- Every iteration produces a score report — no silent skips
- Ledger is append-only — iteration history never deleted
- SKILL.md modifications require human approval (SkillsBench finding: self-generated skill edits = -1.3pp without human review)
- Per-assertion tracking in every score report, not just aggregate scores (prevents AP-8: aggregation hiding regressions)
- Mixed assertion types mandatory in evals.yaml: both `code` and `llm-judge` (prevents AP-1: self-model bias)
- Error analysis classifies every failure — unclassified failures are not allowed
- Improvement proposals are minimal and targeted — no wholesale rewrites

---

## Outputs

| Output | Path | Persistence |
|---|---|---|
| Eval assertions | `{skill-dir}/eval-corpus/evals.yaml` | Created once, updated on eval-design-error |
| Corpus outputs | `{skill-dir}/eval-corpus/corpus-{N}.md` | Stable across iterations |
| Score reports | `{skill-dir}/eval-corpus/score-{iteration}.yaml` | One per iteration |
| Error analysis | `{skill-dir}/eval-corpus/error-analysis.md` | Overwritten each iteration |
| Quality ledger | `{skill-dir}/quality/ledger.yaml` | Append-only, never overwritten |
| Improvement proposal | Inline in session | Not persisted |

---

## Non-Goals

This skill must NOT:
- Auto-modify SKILL.md without human approval (human gate is non-negotiable)
- Invoke the target skill to produce outputs (skills never chain — it evaluates existing outputs only)
- Compare quality across different skills (only within-skill across iterations)
- Set or modify SLO targets (human decision — skill only reads and reports against them)
- Generate corpus from production data without explicit human authorization
- Skip the error analysis step (every failure must be classified before proposing changes)
- Propose changes for `model-limitation` failures (these are documented, not "fixed")

For documented anti-patterns and mitigations, see `references/anti-patterns.md`.

**No silent assumptions. Every evaluation result, every failure classification, every improvement proposal becomes explicit and governed.**