Flake Pattern Reference

---
name: flake-pattern-reference
description: "Reference catalog of the eight flake patterns - async/timing, test ordering, shared parallel state, resource leaks, network, locator drift, environment variance, randomness - with detection heuristics, remediation per pattern, and the concrete code-level fixes: replacing fixed sleeps with framework auto-waits, isolating state in beforeEach fixtures, per-worker DB schemas via workerIndex, try/finally teardown, mocking network + clock at the boundary, stable role-based locators, TZ pinning, and RNG seeding. Use when triaging an unknown flake to identify the category before bisecting, or when a classified flake needs the specific code change to apply."
---

# flake-pattern-reference

> **Terminology note:** "flaky test" is a practitioner-emergent term
> popularized by the Google Testing Blog ([google-causes][gtb-causes],
> [google-flaky][gtb-flaky]); ISTQB does not maintain a canonical
> entry. This catalog reflects industry-engineering consensus, not
> ISTQB authority.

[gtb-flaky]: https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html
[gtb-causes]: https://testing.googleblog.com/2017/04/where-do-our-flaky-tests-come-from.html

A flake is rarely random - it almost always falls into one of eight
recurring patterns. Identifying the pattern early shrinks the bisect
search space dramatically. This catalog is a reference, not a
workflow; the matching workflow is in
`flaky-test-quarantine`.

The Google Testing Blog observed a near-linear correlation between
test size and flakiness rate across ~4.2M tests
([google-causes][gtb-causes]) - larger tests touch more of the eight
patterns at once.

## Pattern 1: async / timing

The most common flake category in UI and integration tests.

| Signal                                                         | What's happening                                              |
|----------------------------------------------------------------|---------------------------------------------------------------|
| Fails ~5 - 20% of runs; passes when the machine is faster        | Test waits for an arbitrary `setTimeout(N)` instead of a deterministic event. |
| Fails on CI but never locally                                  | CI runners have different cold-start timings than dev laptops. |
| Fails after a dependency upgrade with no test code change       | Library's internal timing changed (e.g. Playwright auto-wait window). |

**Remediation:**

- Replace fixed sleeps with **deterministic waits** - `await
  expect(loc).toBeVisible()`, `page.waitForLoadState('networkidle')`,
  `page.waitForFunction(...)`, etc.
- For animations, **disable** them in test setup
  (`animations: 'disabled'` in Playwright; `Cypress.config('animationDistanceThreshold', 0)` in Cypress).
- For absolute clock dependencies, **freeze time** with
  `sinon.useFakeTimers()` / `vi.useFakeTimers()` / Playwright's
  `page.clock.install()`.

## Pattern 2: test ordering

Tests pass alone, fail when run with siblings.

| Signal                                                       | What's happening                                          |
|--------------------------------------------------------------|-----------------------------------------------------------|
| `npm test -- --testNamePattern='^X$'` passes; full run fails  | Test relies on state from a previously-run test.          |
| Adding a new test breaks an unrelated existing one            | Implicit ordering dependency exposed by the new test pushing the old test into a different position. |
| Random-order test runners (Jest `randomize`) flag the suite   | Suite is order-dependent.                                 |

**Remediation:**

- Run the suite with **explicit randomization** in CI to surface
  ordering deps early (`jest --randomize`, `pytest --random-order`,
  `mocha --sort` reverse).
- Move ANY shared setup into `beforeEach` / `afterEach`, never rely on
  `beforeAll` for state that the test mutates.
- Database tests: roll back transactions after each test instead of
  truncating between describe blocks.

## Pattern 3: shared parallel state

Tests pass sequentially, fail when run in parallel workers.

| Signal                                                          | What's happening                                           |
|-----------------------------------------------------------------|------------------------------------------------------------|
| Fails ~50% of runs in CI matrix; passes locally with `-j 1`     | Two workers writing to the same DB row / file / port.     |
| Fails more often as worker count goes up                         | Linear shared-state contention.                            |
| Error message mentions "duplicate key" / "address in use" / "file already exists" | Direct collision evidence.                                  |

**Remediation:**

- Find the shared state two workers collide on - DB rows, files, ports,
  or global module state.
- Per-worker isolation: per-worker DB schemas (`PG_SCHEMA=test_${WORKER}`),
  per-worker temp dirs (`TMPDIR=/tmp/test-${WORKER}`), per-worker port
  ranges.
- For unique IDs: use UUIDs or a per-worker namespace prefix, not
  auto-increment integers shared across workers.

## Pattern 4: resource leaks

Tests pass on a fresh machine, fail after the test process has run for
hours.

| Signal                                                         | What's happening                                            |
|----------------------------------------------------------------|-------------------------------------------------------------|
| Fails increasingly often as suite duration grows                | Memory or file-descriptor leak in the test setup.          |
| `EMFILE` / `EADDRINUSE` errors mid-suite                         | File-descriptor or port exhaustion.                         |
| Long-running processes (Playwright browsers, Cypress runners) crash mid-suite | Process accumulating zombies. |

**Remediation:**

- Always `await browser.close()` / `await server.close()` in `afterAll`,
  with a `try/finally` so failed tests still clean up.
- Set per-test timeouts and ensure the framework kills the process,
  not just the test (`--testTimeout`, `test.setTimeout()`).
- Run `lsof | wc -l` and `ps aux | wc -l` before / after the suite in
  CI to detect leaks; alert when growth exceeds a threshold.

## Pattern 5: network / external service

Tests pass when the upstream is healthy, fail otherwise.

| Signal                                                         | What's happening                                            |
|----------------------------------------------------------------|-------------------------------------------------------------|
| Fails on the same handful of tests that hit the same external URL | Real network call to a flaky third party.                |
| Fails right after a deploy of a non-test service               | Test is hitting prod / staging of a sibling service.        |
| `ETIMEDOUT` / `ECONNRESET` in error logs                         | Network-layer error, not test-logic error.                  |

**Remediation:**

- **Mock at the boundary** - never let test code reach a real network
  endpoint. Use Mock Service Worker (MSW), nock, WireMock, or
  Playwright's `page.route()`.
- For tests that *must* hit a real service (smoke / contract tests),
  isolate them in a separate suite that doesn't gate the main CI.
- DNS-level: pin to specific resolvers in CI to avoid resolver
  variance.

## Pattern 6: locator drift

UI tests pass when the page looks one way, fail when it shifts.

| Signal                                                         | What's happening                                            |
|----------------------------------------------------------------|-------------------------------------------------------------|
| Fails after an unrelated CSS change                             | Selector matched by position rather than identity.          |
| `selector matched 2 elements` errors                            | Ambiguous selector now matches more than one node.          |
| Fails only at certain viewports                                 | Layout shifts cause mobile / desktop selectors to differ.   |

**Remediation:**

- Use **role-based selectors** first (`page.getByRole('button', { name: 'Submit' })`),
  then `data-testid`, only `text=` / CSS as a last resort.
- For Playwright: enable `strict: true` so any ambiguous selector
  fails immediately rather than silently picking the first match.
- For viewport-specific UIs: snapshot at every breakpoint via the
  breakpoint matrix in `playwright-snapshots`; visual signal exposes
  layout-shift flakes faster than text checks.

## Pattern 7: environment variance

Tests pass on Linux CI, fail on macOS dev machines (or vice versa).

| Signal                                                         | What's happening                                            |
|----------------------------------------------------------------|-------------------------------------------------------------|
| Fails only on a specific CI runner / OS                         | OS-specific path separator, line ending, or filesystem case sensitivity. |
| Snapshot tests fail with sub-pixel diffs across OS              | OS font / anti-aliasing differences (see `playwright-snapshots`). |
| Fails in `tz` configurations not set to `UTC`                   | Timezone-sensitive assertion.                               |

**Remediation:**

- Pin CI to one OS / one timezone (`TZ=UTC`) for deterministic runs.
- Run snapshot updates only in CI, never from a developer laptop
  (per `playwright-snapshots`).
- For path-sensitive code, normalize with `path.posix.join()` /
  `node:path`.

## Pattern 8: randomness

Tests use random data without a controlled seed.

| Signal                                                         | What's happening                                            |
|----------------------------------------------------------------|-------------------------------------------------------------|
| Failures don't reproduce on retry                               | Test data was randomized; the failing combination is gone.  |
| Test asserts a property that holds "almost always"              | Property-based test exposing a real edge case (this is good - fix the production bug). |
| Faker-generated data triggers a layout overflow                 | Random string longer than the assertion expected.           |

**Remediation:**

- Seed every random source: `Math.random` via `seedrandom`, faker via
  `faker.seed(N)`, property-based testing via `fc.assert(prop, { seed })`.
- For property-based failures, **don't** mark them as flake - copy the
  failing seed into a regression test.
- Persist the seed used in each CI run as a build artifact so a flake
  can be replayed.

## Triage decision tree

```
Test fails ~50% of runs?
├── Yes → likely "shared parallel state" or "test ordering"
└── No → fails ~5-20% of runs?
    ├── Yes → likely "async/timing" or "network"
    └── No → fails only on specific OS / runner?
        ├── Yes → "environment variance"
        └── No → fails after long suite duration?
            ├── Yes → "resource leaks"
            └── No → fails after unrelated UI change?
                ├── Yes → "locator drift"
                └── No → does the test use random data?
                    ├── Yes → "randomness"
                    └── No → run a structured bisect
```

For systematic bisection, run a structured bisect that varies one axis
at a time per the patterns above.

## Code-level fixes

Once the pattern is named - by inspection above or by experiment via
`flake-axis-bisection` - apply the smallest change the pattern calls for
(a targeted edit, not a rewrite), then re-measure at a real depth: re-run
at an N chosen from the failure rate you are willing to ship, not the
screening N; a clean 0/20 does not prove the flake is gone
(`flake-axis-bisection`). Quarantine via `flaky-test-quarantine` if the
flake blocks the trunk while the fix is in review.

The per-pattern code fixes, grounded in Playwright, Cypress, MSW, and
Faker official docs:

- **Patterns 1-4** (async/timing, test ordering, shared parallel state,
  resource leaks):
  [references/timing-and-isolation-fixes.md](references/timing-and-isolation-fixes.md).
- **Patterns 5-6** (network mock-at-the-boundary with `page.route()` /
  MSW; role-based locators with `data-testid` fallback):
  [references/network-and-locator-fixes.md](references/network-and-locator-fixes.md).
- **Patterns 7-8** (pin `TZ=UTC`, freeze the clock, normalize paths;
  seed every RNG and persist the seed):
  [references/environment-and-randomness-fixes.md](references/environment-and-randomness-fixes.md).

### Worked example

A checkout test, `tests/checkout.spec.ts:42`, fails about 15% of runs in
CI and always passes locally.

1. **Classify.** `flake-axis-bisection` implicates the network-latency
   axis, and reading the source shows the assertion is gated on a fixed
   `page.waitForTimeout(2000)`, not on the response. That is Pattern 1
   (async / timing): the sleep is shorter than the slowest CI response.
2. **Apply the fix.** Replace the fixed sleep with a web-first assertion
   that retries until the condition holds:

```typescript
// Before - the 2s sleep races a variable-latency XHR
await page.getByRole('button', { name: 'Place order' }).click();
await page.waitForTimeout(2000);
expect(await page.getByText('Order confirmed').isVisible()).toBe(true);

// After - retries until the confirmation renders or the timeout expires
await page.getByRole('button', { name: 'Place order' }).click();
await expect(page.getByText('Order confirmed')).toBeVisible();
```

3. **Re-measure and ship.** The team's tolerance is 1%, so re-run at
   N=300 (`flake-axis-bisection` Step 2). A clean 0/300 bounds the rate at
   roughly 1%; a clean 0/20 would have proved nothing. No quarantine was
   needed - the fix landed inside the PR the flake was blocking.

### Quick-reference: pattern to fix

| Pattern | Key fix | Primary API |
|---------|---------|-------------|
| async / timing | Replace sleep with auto-wait assertion | `await expect(loc).toBeVisible()` |
| test ordering | Move setup to `beforeEach`; roll back DB per test | `test.beforeEach` / `test.afterEach` |
| shared parallel state | Per-worker schema / dir / port via `workerIndex` | `testInfo.workerIndex` |
| resource leaks | `browser.close()` in `afterAll` with `try/finally` | `test.afterAll` + `try/finally` |
| network | Mock at boundary; never reach real endpoints | `page.route()` / MSW |
| locator drift | Role-based locators; `data-testid` fallback | `getByRole()` |
| environment variance | Pin `TZ=UTC`; freeze clock; normalize paths | `page.clock.install()` |
| randomness | Seed every RNG; persist seed in CI log | `faker.seed(N)` |

Per-fix citations live in the three reference files above.

## References

- [google-flaky][gtb-flaky] - Google Testing Blog overview.
- [google-causes][gtb-causes] - Google's correlation analysis on
  ~4.2M tests; "test size correlates with flakiness rate."
- `flaky-test-quarantine` - 
  workflow that uses this catalog during triage.