Agent Quality Rubric

---
name: agent-quality-rubric
description: Score and improve an agent file against a 6-dimension rubric — Mission clarity, Scope sharpness, Capability Grounding alignment, Output-Contract completeness, Escalation paths, Example scenarios. Each dimension scored 1-5 with anchors; includes a remediation template that turns a low score into an actionable PR. Reach for this skill when authoring a new agent, reviewing a PR that adds or modifies an agent, or running a periodic agent-bank audit. Used by `prompt-engineer` (primary) plus `architect`.
---

# Skill: agent-quality-rubric

You are reviewing an agent file (typically `plugins/<plugin>/agents/<role>.md`). The rubric below is a 6-dimension scorecard that surfaces concrete gaps and feeds an actionable remediation PR — not a vibe-check.

The rubric exists because agent quality is the single largest predictor of marketplace output quality, and "looks fine to me" is not a maintainable review standard. Each dimension is anchored 1-5 with text descriptions so two reviewers grade the same agent within roughly one point.

## When to invoke this skill

- Authoring a new agent (before opening the PR).
- Reviewing a PR that adds or modifies an agent (cite dimension + score in your review comments).
- Periodic agent-bank audit (e.g. quarterly): score every agent in `plugins/*/agents/` and queue any dimension ≤2 for remediation.
- The Researcher's Weekly Deep Research can run this against agents in its sweep.

## Gate 0 — the mechanical prerequisites (not scored; a fail here blocks before Dimension 1)

Check the frontmatter before scoring anything. Inside the RavenClaude marketplace this is CI's
frontmatter gate (marketplace-dev tooling, not shipped in the plugin); in a consumer repo apply the
same four rules by hand. The gate fails on a missing/empty `tools:` allowlist, a `description` over
300 chars, a missing scenario schema, and a **missing `model:` or a `model:` that is not a tier
alias** (`haiku` / `sonnet` / `opus` / `fable` / `inherit` — a full model id is rejected because it
goes stale when the SKU rotates). No rubric score compensates for a red gate.

**Then apply the one judgment call the gate cannot make — does the tier fit the role?** Read the
agent's mission (Dimension 1) against the tier table in
[`knowledge/model-tier-delegation.md`](../../knowledge/model-tier-delegation.md): read-a-lot-return-a-little
work (search / grep / classify / extract / inventory) belongs on `haiku`; bounded, well-specified
edits and drafts on `sonnet`; only design, adjudication, and the gates that hold merge
(`security-reviewer`, `code-reviewer`, `architect`) on `opus`. Two mismatches are worth a
review comment: an `opus` agent whose scenarios are all grep-shaped (paying frontier rates for
volume work — in the marketplace the roster-wide frontier share is ratcheted by CI's `check-model-tier-ratchet.py`,
Gate 287, so this also fails the PR if it raises the share), and a `haiku` agent asked to gate a
merge or make a design call (the tier cannot carry the judgment). The marketplace's
`check-model-tier-fit.py` (Gate 288) catches the *mechanical* slice of both — an agent named
`*-implementation-engineer` / `*-developer` or whose description opens with "Use to BUILD" /
"Use for BUILDING" may not be pinned `opus`, and the core merge gates may not be pinned below it — so
the reviewer's judgment is reserved for the roles the name and the opening verb do not settle. Those
are exactly what `check-model-tier-fit.py --report` lists as the **pair-review queue** (a frontier
`*-engineer` beside its plugin's architect/lead, not opening by deciding); tier them by
**sibling-plugin parity** — find the same role shape in the nearest sibling plugin and match its
tier, or write the role-level reason for differing. `inherit` is a deliberate
choice, not a default — it must say in one clause why the worker should run at the session's tier.
One more `tools:` check, mechanical and gated (`check-nested-dispatch.py`, Gate 289): the allow-list
may not grant dispatch — no `Agent`, `Agent(...)`, `Task`, or `"*"` — because a sub-agent that can
call agents pays the handoff tax at its own tier out of the Team Lead's sight, and in a sub-agent
definition the `Agent(type)` list is ignored (unscoped). If a definition genuinely needs it, the
review question is whether it is the one sanctioned shape — a frontier parent fanning out to
fast-tier read-only leaves — and the answer goes by name into
`tests/fixtures/nested-dispatch-exemptions.json`, not into a looser gate.

## The 6 dimensions

### Dimension 1 — Mission clarity

**The one-sentence-mission test:** can you state, in one sentence with no hedges, what this agent does and does not do? If you find yourself writing "...and also..." you've already failed clarity.

| Score | Anchor |
|---|---|
| **5** | Mission stated in the first paragraph in one sentence; reading the rest of the agent only confirms it. |
| **4** | Mission stated, requires one supporting clause but still parseable in a single read. |
| **3** | Mission inferable from the agent body but never stated explicitly; readers will reconstruct different missions. |
| **2** | Two or more competing missions in the file ("this agent does X *and* Y"). |
| **1** | No discernible mission; the file reads like a grab-bag of responsibilities. |

### Dimension 2 — Scope sharpness

**The "NOT for X" test:** does the agent explicitly say what it does NOT handle? Sharpness comes from negation; positive scope alone always overlaps with neighboring agents.

| Score | Anchor |
|---|---|
| **5** | Explicit "Not for X / not for Y" list, with the named alternative agent for each excluded case. |
| **4** | Negation present but no named alternative ("not for security review" with no pointer to security-reviewer). |
| **3** | Implicit scope only; reader has to compare against other agent files to infer the boundary. |
| **2** | Overlap with at least one other agent in the same plugin; no disambiguation. |
| **1** | Drift-prone: the agent claims responsibilities that already live elsewhere in the marketplace. |

### Dimension 3 — Capability Grounding alignment

**The Grounding Protocol inheritance test:** does the agent reference the plugin's CLAUDE.md §5 Capability Grounding Protocol? Does its Output Contract include the line *"Grounding checks performed: ..."*? Does it acknowledge the alternate-methods rule, the pre-action decision-tree traversal, and the environment-context check?

| Score | Anchor |
|---|---|
| **5** | Explicit "inherits the Capability Grounding Protocol" line; output template includes "Grounding checks performed"; alternate-methods rule referenced when relevant. |
| **4** | Protocol referenced but the output template doesn't propagate the checks-performed line. |
| **3** | Protocol named, but no operational tie-in (reader is expected to know what it means). |
| **2** | Protocol not mentioned; agent may silently violate the alternate-methods rule. |
| **1** | Agent contradicts the protocol (e.g. tells the user "this isn't possible" without enumerating alternatives). |

### Dimension 4 — Output-Contract completeness

**The five-line check:** does the Output Contract specify (a) Status, (b) Files-changed list, (c) any mandatory plugin-specific lines (e.g. "Grounding checks performed", "Decision-tree branch chosen"), (d) the `---RESULT_START---` JSON block per [`structured-output.md`](../structured-output/SKILL.md), and (e) a reporting cap (word/line limit)?

| Score | Anchor |
|---|---|
| **5** | All five present; the JSON shape is explicit with all required keys (`status`, `summary`, `deliverables`, `handoff_recommendation`, `confidence`, `risks_or_open_questions`, `next_actions`). |
| **4** | All five present but the JSON shape lists fewer than the canonical 7 keys. |
| **3** | Four of five (typically missing the structured JSON block or the reporting cap). |
| **2** | Two or three of five — agent will produce inconsistent, hard-to-parse handoffs. |
| **1** | Free-form output instructions; downstream automation will break on this agent's reports. |

### Dimension 5 — Escalation paths

**The named-handoff test:** when this agent hits a problem outside its scope, does it know exactly which OTHER agent to escalate to, and does it know why? "Escalate to the Team Lead" alone is necessary but not sufficient — name the specialist + reason.

| Score | Anchor |
|---|---|
| **5** | Table or list of escalation rows: surfacing → likely cause → named target agent. At least 3 rows; covers the most common cross-boundary cases. |
| **4** | List present but missing one obvious case (e.g. agent touches auth but doesn't name `security-reviewer`). |
| **3** | "Escalate to Team Lead" only; no named specialists. |
| **2** | Escalation mentioned but no rules; reader has to guess. |
| **1** | No escalation guidance — agent will either over-claim or silently drop work it can't do. |

### Dimension 6 — Example scenarios

**The agent-scenario-authoring frontmatter test:** does the YAML frontmatter include `audience`, `works_with`, `scenarios`, and `quickstart` per [`docs/best-practices/agent-scenario-authoring.md`](../../../../docs/best-practices/agent-scenario-authoring.md)? The repo-guide generator picks them up automatically for per-agent cards and the Overview use-case lookup table — missing fields = invisible agent.

| Score | Anchor |
|---|---|
| **5** | All four fields populated with at least 3 scenarios; each scenario has trigger phrasing a user would actually type. |
| **4** | All four fields present but scenarios are thin (1-2) or generic. |
| **3** | Three of four fields; one is empty placeholder. |
| **2** | Two or fewer fields; agent is essentially invisible in the repo-guide. |
| **1** | No scenario-authoring frontmatter — repo-guide can't render the agent's card. |

## Scoring guidance

| Total score | Disposition |
|---|---|
| **27-30** (avg 4.5+) | Ships as-is. |
| **22-26** (avg 3.7+) | Ship with minor edits — note the weakest dimensions in the PR description. |
| **16-21** (avg 2.7+) | Revise before merge. Address every dimension scoring ≤3. |
| **10-15** | Rewrite the weakest dimensions individually. The skeleton is salvageable. |
| **6-9** | Block — re-author from a fresh brief via [`draft-agent-brief`](../draft-agent-brief/SKILL.md). |

A single dimension scored 1 is a block regardless of total. Mission-clarity ≤2 is a block regardless of total — the rest of the agent inherits the ambiguity.

## Using the rubric in a PR review

Comment on the PR diff using this shape:

> **Dimension 3 (Capability Grounding) — score 2.** Agent doesn't mention the alternate-methods rule. When the chosen tool fails, this agent will report blocked without enumerating Approach B / C. Add the standard CGP-inheritance paragraph from `agents/architect.md` lines 18-26 as a template.

One comment per dimension scored ≤3 is the right cadence. Don't comment on dimensions scored ≥4 unless you have a concrete suggestion — noise dilutes the load-bearing comments.

## Remediation template (drop into a PR description)

When a PR's purpose is to lift an agent's rubric score, the PR description should follow this shape so the reviewer can verify each gap was closed:

```markdown
## Agent quality rubric — remediation PR

**Agent:** `plugins/<plugin>/agents/<role>.md`
**Pre-remediation score:** <total>/30
**Target score:** <total>/30

### Dimensions addressed

| Dimension | Pre | Post | Change made |
|---|---|---|---|
| Mission clarity | <n> | <n> | <one-line description, or "no change"> |
| Scope sharpness | <n> | <n> | <e.g. added "Not for X — escalate to Y" list> |
| Capability Grounding | <n> | <n> | <e.g. added §5 inheritance paragraph + "Grounding checks performed" line> |
| Output-Contract | <n> | <n> | <e.g. added ---RESULT_START--- JSON block with 7 canonical keys> |
| Escalation paths | <n> | <n> | <e.g. added 4-row escalation table> |
| Example scenarios | <n> | <n> | <e.g. populated audience / works_with / scenarios / quickstart> |

### Verification

- [ ] `python3 -m json.tool` passes on any JSON examples in the agent file
- [ ] Repo-guide preview renders the agent's per-agent card without warnings
- [ ] At least one downstream agent's escalation table now points to this agent (if applicable)
- [ ] Plugin's CLAUDE.md skill/agent index updated if scope changed
```

## Anti-patterns

- **Scoring without quoting.** Every score ≤3 must be paired with a quoted excerpt from the agent file (or a noted absence) so the author knows what to fix.
- **Vibe-grading.** "Feels like a 4" is not a score. Anchor to the dimension's text descriptions.
- **Average-only verdict.** A 30/30 with one dimension at 1 still blocks. Read every row.
- **Re-scoring without reading the remediation.** When a PR lifts a score, re-grade the affected dimensions against the new text, not from memory of the old text.
- **Treating Mission and Scope as one dimension.** They aren't — a clear mission can still overlap with neighbors. Scope sharpness is the negation test, not the mission test.

## See also

- [`structured-output.md`](../structured-output/SKILL.md) — the Output-Contract dimension's reference.
- [`scenario-retrieval.md`](../scenario-retrieval/SKILL.md) — informs the Escalation-paths dimension when agents should consult scenarios before answering.
- [`draft-agent-brief.md`](../draft-agent-brief/SKILL.md) — feeds in when total score ≤9 and the agent needs a re-author.
- [`prompt-pattern-library.md`](../prompt-pattern-library/SKILL.md) — the canonical patterns the rubric checks for compliance with.
- [`../../../docs/best-practices/agent-scenario-authoring.md`](../../../../docs/best-practices/agent-scenario-authoring.md) — the Dimension 6 reference.