Mutation Testing

---
name: Mutation Testing
description: Run mutation testing with StrykerJS for TypeScript/JavaScript and mutmut for Python to measure whether your test suite actually catches bugs, enforce mutation score thresholds, and keep runs fast with incremental mode in CI.
version: 1.0.0
author: thetestingacademy
license: MIT
tags: [mutation-testing, stryker, mutmut, test-quality, coverage, code-quality, ci, jest, vitest, pytest]
testingTypes: [unit, code-quality]
frameworks: [jest, vitest, pytest]
languages: [typescript, javascript, python]
domains: [web, api]
agents: [claude-code, cursor, github-copilot, windsurf, codex, aider, continue, cline, zed, bolt, gemini-cli, amp]
---

# Mutation Testing

This skill makes an AI agent set up and interpret mutation testing: StrykerJS for JS/TS projects (Jest or Vitest runners) and mutmut for Python. Mutation testing injects small bugs (mutants) into source code and re-runs the tests — if the tests still pass, the mutant "survived" and your suite has a blind spot that line coverage never showed. Trigger this when a team claims high coverage but still ships regressions, when reviewing test suite quality, or when the user mentions Stryker, mutmut, or mutation score.

## Core Principles

1. **Line coverage measures execution, mutation score measures verification.** A test that calls a function and asserts nothing gives 100% line coverage and a 0% mutation score. Mutation score is the honest metric.
2. **Never mutate everything on every run.** A full mutation run on a real codebase takes hours. Use incremental mode locally and scope CI runs to changed files. Full runs belong in nightly jobs.
3. **A surviving mutant is either a missing assertion or dead code.** Both are findings. If you cannot write a test that kills a mutant, the mutated code path is unreachable — delete it instead of suppressing the mutant.
4. **Set `break` thresholds, not aspirational targets.** Start at your current score minus 2, fail the build below it, and ratchet up monthly. A threshold nobody enforces is documentation fiction.
5. **Equivalent mutants exist; budget for them.** Some mutations (e.g., `i < len` to `i <= len` on an array that is never exactly full) produce identical behavior. Mark them ignored explicitly rather than chasing 100%.
6. **Kill mutants with stronger assertions, not more tests.** The fix for a survived `>` → `>=` mutant is usually one boundary-value assertion in an existing test, not a new test file.

## Setup: StrykerJS (TypeScript + Vitest)

```bash
npm install --save-dev @stryker-mutator/core @stryker-mutator/vitest-runner
npx stryker init
```

```json
// stryker.config.json
{
  "$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
  "testRunner": "vitest",
  "mutate": ["src/**/*.ts", "!src/**/*.test.ts", "!src/**/*.d.ts", "!src/generated/**"],
  "coverageAnalysis": "perTest",
  "reporters": ["html", "clear-text", "progress", "json"],
  "thresholds": { "high": 85, "low": 70, "break": 65 },
  "incremental": true,
  "incrementalFile": ".stryker-tmp/incremental.json",
  "timeoutMS": 10000,
  "concurrency": 4
}
```

For Jest projects swap the runner:

```bash
npm install --save-dev @stryker-mutator/core @stryker-mutator/jest-runner
```

```json
// stryker.config.json (Jest variant)
{
  "testRunner": "jest",
  "jest": { "projectType": "custom", "configFile": "jest.config.js" },
  "coverageAnalysis": "perTest",
  "mutate": ["src/**/*.ts", "!src/**/*.test.ts"],
  "thresholds": { "high": 85, "low": 70, "break": 65 }
}
```

Run it:

```bash
npx stryker run                          # full run, writes reports/mutation/mutation.html
npx stryker run --mutate src/pricing.ts  # scope to one file while fixing survivors
```

## Reading a Survived Mutant and Killing It

Stryker output:

```text
[Survived] ArithmeticOperator
src/pricing.ts:14:31
-   return subtotal * (1 - discountRate);
+   return subtotal / (1 - discountRate);
Ran 3 tests, none failed.
```

The suite never asserts a discounted price with a nonzero rate. Kill it with a boundary assertion:

```typescript
// src/pricing.test.ts
import { describe, expect, it } from 'vitest';
import { applyDiscount } from './pricing';

describe('applyDiscount', () => {
  it('multiplies subtotal by the inverse discount rate', () => {
    // 200 * (1 - 0.25) = 150; the division mutant would return 266.67
    expect(applyDiscount(200, 0.25)).toBe(150);
  });

  it('returns the subtotal unchanged at rate 0', () => {
    expect(applyDiscount(99.5, 0)).toBe(99.5);
  });
});
```

Ignore a genuinely equivalent mutant inline instead of lowering the threshold:

```typescript
// Stryker disable next-line EqualityOperator: loop bound is equivalent for empty input
for (let i = 0; i < items.length; i++) {
```

## Setup: mutmut (Python + pytest)

```bash
pip install mutmut pytest
```

```toml
# pyproject.toml
[tool.mutmut]
paths_to_mutate = ["src/"]
tests_dir = ["tests/"]
also_copy = ["conftest.py"]
```

```bash
mutmut run                 # mutates src/, runs pytest per mutant, caches results
mutmut results             # summary: killed / survived / timeout / suspicious
mutmut show src.pricing.apply_discount__mutmut_3   # diff of one survivor
mutmut run --max-children 4                        # parallel workers
```

Killing a Python survivor follows the same pattern — the mutant tells you the missing assertion:

```python
# tests/test_pricing.py
import pytest
from src.pricing import apply_discount


def test_apply_discount_uses_multiplication():
    # mutant changed * to /; 200 * 0.75 == 150, 200 / 0.75 == 266.67
    assert apply_discount(200, 0.25) == 150


def test_apply_discount_rejects_rate_above_one():
    # kills the mutant that removed the validation guard
    with pytest.raises(ValueError):
        apply_discount(100, 1.5)
```

## CI Integration

Incremental Stryker on pull requests, full run nightly:

```yaml
# .github/workflows/mutation.yml
name: mutation
on:
  pull_request:
    paths: ['src/**', 'stryker.config.json']
  schedule:
    - cron: '0 2 * * *'

jobs:
  stryker:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0 # incremental mode diffs against git history
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: npm
      - run: npm ci
      - name: Restore incremental cache
        uses: actions/cache@v4
        with:
          path: .stryker-tmp/incremental.json
          key: stryker-incremental-${{ github.base_ref || 'main' }}
      - name: Mutation test (incremental on PRs, full nightly)
        run: |
          if [ "${{ github.event_name }}" = "pull_request" ]; then
            npx stryker run --incremental
          else
            npx stryker run --force
          fi
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: mutation-report
          path: reports/mutation/mutation.html
```

The `thresholds.break: 65` in config makes `stryker run` exit non-zero below 65% — no extra scripting needed.

## Best Practices

- Start mutation testing on your most critical module (pricing, auth, parsers), not the whole repo. One high-signal report beats a week-long full run.
- Use `coverageAnalysis: "perTest"` — Stryker then only runs the tests that cover each mutant, often a 10x speedup.
- Commit `.stryker-tmp/incremental.json` to CI cache, never to git.
- Track mutation score per module in the JSON report; aggregate scores hide a 30% module behind a 90% repo average.
- Set `timeoutMS` explicitly. Mutants that create infinite loops are killed by timeout; the default factor-based timeout misbehaves on very fast suites.
- In Python, run `mutmut run` against a single module first: `paths_to_mutate = ["src/billing.py"]`.

## Anti-Patterns

- **Chasing 100% mutation score.** Past ~90% you are writing tests for equivalent mutants. Mark them ignored with a reason and move on.
- **Running full mutation on every PR.** It will take 40+ minutes and the team will delete the workflow within a month. Incremental on PRs, full nightly.
- **Treating "survived" as a Stryker bug.** Over years of use, genuine false positives are rare; assume your suite is the problem first.
- **Mutating test files or generated code.** Always exclude `*.test.ts`, `*.spec.ts`, migrations, and codegen output in `mutate` globs.
- **Lowering `break` to make a red build green.** The threshold ratchets up, never down; fix the survivors or explicitly ignore equivalents inline.
- **Writing assertion-free "kill tests"** that snapshot huge objects just to kill mutants. A surviving mutant deserves a precise boundary assertion, not a snapshot blanket.

## When to Trigger This Skill

- The user mentions Stryker, mutmut, mutation score, or "are my tests actually good".
- A project has high line coverage but recurring production regressions in covered code.
- Reviewing or hardening a test suite for a critical module (payments, auth, data migration).
- Setting up CI quality gates beyond line coverage thresholds.
- A `stryker.config.json`, `stryker.conf.mjs`, or `[tool.mutmut]` section already exists in the repo.