Unit Testing Rubric 0.1.0

---
name: unit-testing-rubric
description: Standards for writing, reviewing, and evaluating unit tests. Use whenever writing new tests, reviewing existing tests, deciding what to test, assigning a test to a fast vs. full feedback loop, or diagnosing flaky, slow, or low-value tests. Covers structural fidelity, the FIRST properties, behavioral invariants, and coverage-as-diagnostic.
---

# Unit Testing Rubric

## Purpose

Create a trustworthy, deterministic, affordable test suite that protects
behavioral invariants, supports refactoring, and provides rapid feedback.

Flaky tests are defects in the feedback system and must be fixed, replaced, or
removed.

## Testing Integrity

**1. Fidelity Rule** — Tests must be structurally honest. The production
behavior under test must execute the same way in the test harness as in
production. Introduce seams around dependencies, not around behavior.
*Would production execute the same behavior if this test did not exist?*

**2. Documentation Rule** — A microtest is executable documentation. A developer
should understand the behavior, context, action, expected result, and important
edge cases without reading the implementation.
*Can a developer understand the behavior from the test alone?*

**3. Single Action Rule** — An Arrange–Act–Assert test performs exactly one
action in one context. Multiple assertions are acceptable when they examine
evidence from the same action.
*What is the single action being performed?*

## FIRST Properties

**4. Fast Rule** — Tests should support rapid feedback. The fast suite (however
the project invokes it) should complete in well under a minute; treat ~30
seconds as a reasonable default budget absent a project-specific one.
*Is this test worth spending part of the fast-feedback budget?*

**5. Isolation Rule** — Tests must not depend on execution order, shared mutable
state, hidden fixtures, or environmental assumptions.
*Can this test pass independently on any machine?*

**6. Repeatability Rule** — Accepted tests must be deterministic. Control time,
randomness, IDs, concurrency, and external dependencies.
*Will this test produce the same result every time?*

**7. Self-Verifying Rule** — Tests produce an automatic, unambiguous pass/fail
result.
*Does the test clearly determine success or failure without human
interpretation?*

**8. Timely Rule** — Tests are written alongside behavior, protect regressions,
and evolve with the product.
*Is this test protecting currently desired behavior?*

## Behavioral Quality

**9. Behavioral Invariant Rule** — Every permanent test specifies a behavior
that should remain true until the product no longer wants it. Tests exist to
lock down behavioral invariants.
*What invariant does this test protect?*

**10. Potency Rule** — A test must fail for a plausible violation of the
invariant it specifies. A test that cannot detect a realistic defect provides
little value.
*What bug would make this test fail?*

**11. Failure Clarity Rule** — When a test fails, it should clearly illuminate
the violated invariant and help localize the problem. Failures should be
specific, local, and diagnostic.
*If this test fails, will a developer know what invariant was violated?*

## Portfolio Quality

**12. Affordability Rule** — Confidence should be obtained at the cheapest
reliable test level. Prefer narrow deterministic tests over expensive broad
tests.
*Is there a cheaper deterministic test that provides the same confidence?*

**13. Feedback Loop Rule** — Assign each test to the earliest useful feedback
loop: a fast loop (e.g. `run_tests`) for deterministic feedback on every change,
and a full/slow loop (e.g. `full_test`) for additional deterministic confidence
that doesn't belong in the fast one (end-to-end, slow, or integration-heavy
cases) — name the two loops whatever the project calls them. Most confidence
should live in the fast loop.
*Why doesn't this confidence belong in the earlier feedback loop?*

**14. Coverage Diagnostic Rule** — Coverage is a diagnostic signal, not the
goal. Use it to discover unprotected decisions, transformations, and invariants.
Do not create tests merely to increase coverage percentages.
*What behavioral invariant is missing behind this uncovered code?*

## Lifecycle

**15. Prophet / Guide / Guard Rule** — Every permanent test serves three
purposes: **Prophet** (specifies a future invariant before implementation),
**Guide** (provides the target behavior during implementation), **Guard**
(protects behavior during maintenance and refactoring).
*Does this test successfully act as Prophet, Guide, and Guard?*

## Operational Expectations

- All accepted tests are deterministic.
- No retries, quarantines, reruns-until-pass, or tolerated failures.
- Prefer behavior assertions over implementation assertions.
- JSON object field order is not asserted unless contractually significant.
- Most confidence should come from the fast loop.
- Every test should justify its value against its cost.
- Refactor tests to reduce fragility, redundancy, and cost while preserving
  invariants.
- Delete tests only when the behavior is no longer desired, or when the test is
  redundant, flaky, or provides no unique confidence.

## Common Pitfalls

**Ambiguous fixture selection.** When a test needs to pick one item out of
several candidates that share a common identifying field (e.g. two functions,
signatures, or records with the same name/owner but a different shape),
filtering on that one shared field alone can silently bind to the wrong
candidate — the test then validates the wrong thing while still appearing to
pass. Filter on every field that actually distinguishes the candidates, not
just the most obvious one.

**Widening behavior without breaking coverage.** When an existing function's
behavior is being deliberately widened (recognizing more inputs, covering more
cases) without changing its signature, audit every existing test that asserts
on it *before* changing the implementation, to confirm none assumes the
narrower behavior. A widening change is verifiably non-breaking when no
existing assertion could contradict the new, broader behavior — don't rely on
a regression to reveal a conflict you could have checked for directly.