Project Mutation And Coverage

---
name: project-mutation-and-coverage
description: "Use when running mutation testing or code coverage, reading a mutation score or a coverage percentage, fetching a weekly report from GitHub Actions, choosing between just mutation, just mutation-diff, just phpunit-coverage and just e2e-coverage, or writing a test that asserts on elapsed wall-clock time."
---

# Mutation testing and coverage

Two weekly GitHub Actions produce these reports. Neither is a required check and
neither gates a merge. Both write their results to an artifact, so the run page
shows no score.

## The recipes, and where each writes

Four recipes exist and the names do not make the difference obvious.

| Recipe | Measures | Writes |
|---|---|---|
| `just mutation` | mutation, all of `src` | `var/infection/` |
| `just mutation-diff [BASE]` | mutation, the `src/` files you changed | `var/infection/` |
| `just phpunit-coverage` | coverage, the PHPUnit suites | `var/phpunit-coverage/` |
| `just e2e-coverage` | coverage, the Playwright suite | `var/coverage/` |

A hook refuses a bare `just phpunit-coverage` for an agent. Pass a path, such as
`just phpunit-coverage tests/Module/Review`, or fetch the weekly CI report.

`just e2e-coverage` deletes `var/coverage` at the start of every run. Keep any
report you want out of that directory. `just open-coverage` opens the e2e report
and `just open-phpunit-coverage` opens the PHPUnit one.

`just mutation-diff` is the one to run while you work. It mutates the `src/`
files you added or changed against `origin/main`, including uncommitted and
untracked ones, and takes minutes.

## Never run two of these at once in one worktree

They share one test database. `TEST_SCHEMA_READY` stops each mutant process
rebuilding the schema, so a second run that does rebuild it pulls the schema out
from under the first. The symptom is `duplicate key value violates unique
constraint "pg_type_typname_nsp_index"` in `tests/bootstrap.php`.

To stop a run, read `docker exec loupe-php-fpm-1 ps aux | grep -E 'infection|phpunit'`
first, then match your own worktree path. A bare `pkill -f phpunit` kills every
worktree's tests on the machine.

## Fetching a report from CI

The reports exist only as artifacts. Fetch one with `just ci-report`:

```
just ci-report mutation            # summary.log
just ci-report phpunit-coverage    # head of summary.txt
just ci-report e2e-coverage        # head of summary.txt
just ci-report e2e-timing          # per-project time and busy workers
just ci-report e2e-timing <run-id> # any run, a pull request's included
just ci-report phpunit-timing      # slowest PHPUnit classes and tests
```

With no run id, the recipe takes the newest completed run on `main` that still
holds the artifact. It downloads into `var/ci-reports/<artifact>/<run-id>/`,
which git ignores, and it does not download a run twice.

| Report | Workflow | Job | Artifact | Holds | Retention |
|---|---|---|---|---|---|
| mutation | `Mutation testing` | `infection` | `infection-report` | `summary.log`, `infection.log` | 90 days |
| PHPUnit coverage | `Coverage report` | `phpunit-coverage` | `phpunit-coverage` | `summary.txt`, `clover.xml`, `html/` | 90 days |
| e2e coverage | `Coverage report` | `e2e-coverage` | `e2e-coverage` | `summary.txt`, `clover.xml`, `html/` | 90 days |
| e2e timing | `CI` | `e2e-chromium`, `e2e-chromium-2`, `e2e-rest` | `e2e-timing-chromium`, `e2e-timing-chromium-2`, `e2e-timing-rest` | `results.json` | 30 days |
| PHPUnit timing | `CI` | `phpunit` | `phpunit-timing` | `junit.xml` | 30 days |

The e2e timing artifact comes from every CI run, green or red. `bin/e2e-timing.mjs`
reads its Playwright JSON report. Time between two tests on one worker counts as
idle, so worker start-up shows as idle time.

The e2e job runs as three shards on three runners, so a run holds one timing
artifact per shard and `just ci-report e2e-timing` prints one summary for each.
The three ran against separate clocks, so read the wall times side by side and
never add them.

The PHPUnit timing artifact comes from every CI run as well. `bin/phpunit-timing.php`
reads the JUnit XML and prints the mean seconds per test, the slowest classes and
the slowest tests. Its total sums the test times of every ParaTest worker, so it
is not the job's wall clock. PHPUnit opens the log when the run starts and writes
it when the run ends, so a process that dies in between uploads an empty file.
The script names that case rather than reporting zero tests.

A coverage `summary.txt` carries terminal colour codes, so a plain `grep` shows
escape characters around the numbers.

Read `summary.log` or `summary.txt` first. A mutation `summary.log` is a few
hundred bytes. A coverage `summary.txt` is about 80 KB, because every class
follows the summary. The HTML tree downloads as about 7.7 MB and expands to
about 87 MB, so fetch it only when you need a per-file breakdown.

A run with no artifact means the job failed before it wrote one. It does not mean
zero coverage, and it does not mean the upload step is missing. Read the job log.

A red run on its own still produces a report. `just e2e-coverage` runs its
`phpcov merge` before the exit check, deliberately, so a suite that loses a spec
still yields coverage for the specs that passed, and the run still reads as red.
So a missing artifact means the job died earlier than the merge: an app it could
not reach, a stylesheet build that failed, a container that never came up.

`infection.log` is about 380 KB and holds four sections: `Escaped mutants:`,
`Timed Out mutants:`, `Skipped mutants:` and `Not Covered mutants:`. An escaped
mutant names a test that asserts nothing about a line it covers. Judge each one,
because some are harmless.

## A wall-clock assertion is a landmine

xdebug drives both mutation and coverage, and it roughly doubles execution.

Never assert an absolute elapsed time in a test. `MarkdownRendererTest` asserted
that a render finished in under 5 seconds. It took about 2.5s normally and 5.4s
under coverage, so it failed the initial coverage run and aborted Infection
before a single mutant ran. The failure reads as Infection being broken rather
than as one unrelated test. It breaks a coverage run the same way.

Assert a ratio instead. Run the work at two input sizes and compare the times:

```php
[$quarterElapsed] = self::timeRender($renderer, 5_000);
[$fullElapsed, $html] = self::timeRender($renderer, 20_000);

// Four times the headings: linear takes 4x, quadratic 16x.
self::assertLessThan(8.0, $fullElapsed / $quarterElapsed);
```

Coverage slows both renders by the same factor, so the ratio survives it. A
loaded machine can slow one render and not the other, so keep the bound far from
both growth rates. With sizes of 10,000 and 20,000 and a bound of 3.0, one
`just ci` on a loaded machine measured 3.16 and failed.

Playwright carries the same kind of bound in its own config. Its `expect`
timeout is an absolute 5 seconds. Per-request collection takes one page render
from about 0.5s to about 4.7s, so a coverage run sits on that bound and fails on
timeouts rather than on anything a spec asserts. `playwright.config.ts` raises
the timeouts when `COVERAGE` is set, and the per-pull-request gate keeps 5s.

## Never size a run from a local timing

A workload that starts one process per unit of work runs about 8 times slower on
a development Mac than on a hosted runner. Every process autoloads `vendor/`
across the bind mount, and Docker Desktop serves that mount through a virtual
filesystem. The same suite inside one process is only about 2 times slower.

This holds on an idle machine, so waiting for a quiet machine does not fix it. A
local projection of the full mutation run said 1h 22m. The runner does it in 15
to 25 minutes.

Two runs of identical work on identical runners differed by 1.8x, so read one
hosted timing as a sample. Set `timeout-minutes` from a measurement, at a few
multiples of it. A loose timeout turns a degraded run into an hour of runner time
nobody watches.

## These tools fail with a signal that looks valid

Every failure here so far has produced an answer a reader would accept. Distrust
a number from a run you have not confirmed finished.

| What you see | What it means |
|---|---|
| A fast, complete-looking mutation run | It may have crashed in cleanup after printing the metrics. Check the exit status. |
| `ini_get("xdebug.mode")` says `develop` | Nothing. xdebug honours the `XDEBUG_MODE` environment variable, and `ini_get` reports the ini file. Read `getenv`. |
| No artifact on a run | The job failed before writing one. It is not zero coverage. |
| `Command terminated by signal 7` | The container died on host memory, past PHP's own limit. No PHP error and no stack trace. |

## Raising PHPUnit's memory limit

`php -d memory_limit=...` does not reach PHPUnit. `phpunit.dist.xml` sets the
limit with an `<ini>` directive, and PHPUnit applies it with `ini_set()` after
startup, so the command line loses. A run started with `-d memory_limit=4G`
still died at exactly 536870912 bytes. Change the value in `phpunit.dist.xml`,
and note that doing so raises the ceiling for `just phpunit` and the gate too,
which gives a memory regression more room to hide.

This came up while measuring `--path-coverage`, which this project does not use.
Path coverage held 492 MB for 51 tests and killed the container at about 250 of
2,262 tests, so it does not fit this suite.

## Infection needs 2 GB

Both recipes run Infection under `php -d memory_limit=2G`. It deletes about 8,500
mutant directories when it finishes, and `Filesystem::remove()` materialises that
tree before deleting it. At 512M the process dies in the cleanup, after the
reports are written, so the failure looks like something else.