Load Testing

---
name: load-testing
description: Load testing — smoke/load/stress/soak/spike profiles, SLO thresholds, tooling.
---

# Load Testing

Server-side performance validation: how much traffic the system takes before it degrades, and where it breaks first.

## When to Load

Load for backend, API, or infrastructure work where capacity, concurrency or throughput matter — before a launch, before a traffic event, after an architecture change, or when investigating production saturation.

**Boundary with `skills/architecture/performance-budgets/SKILL.md`:**

| Concern | Skill |
|---|---|
| Client-side rendering, Core Web Vitals, bundle size, Lighthouse CI | `performance-budgets` |
| Server latency under concurrency, throughput, saturation, breaking point | this skill |

They are complementary: a page can hit every Core Web Vitals target at one user and still collapse at a thousand.

---

## Load Profiles

Pick the profile from the question being asked. Running the wrong profile answers a question nobody had.

| Profile | Shape | Question answered | Typical duration |
|---|---|---|---|
| **Smoke** | Minimal load, 1–5 virtual users | Does the script work and the system respond correctly at all? | 1–2 min |
| **Load** | Ramp to expected peak, hold | Does the system meet its SLOs at normal peak traffic? | 10–30 min |
| **Stress** | Ramp past peak until degradation | Where is the breaking point, and what breaks first? | 20–60 min |
| **Soak** (endurance) | Moderate load held for hours | Do leaks, unbounded caches, connection or disk growth appear over time? | 2–24 h |
| **Spike** | Instant jump to a multiple of peak, then drop | Does the system survive a sudden surge and **recover** afterwards? | 5–15 min |
| **Breakpoint** | Slow unbounded ramp | What is the maximum sustainable throughput per instance? | Until failure |

Rules:
- **Always run smoke first.** Most "failed" load tests are broken scripts, wrong credentials, or a 404 loop returning fast errors.
- Ramp gradually in load and stress tests — instant ramps only measure cold starts and connection pool warmup.
- Include a **recovery window** after spike and stress runs; a system that never recovers without a restart has a worse defect than the one that slowed down.
- Change one variable per run. A test that changes load *and* configuration explains nothing.

---

## Deriving Thresholds from SLOs

Thresholds are not invented for the test — they come from the service's SLOs (`skills/architecture/observability-slo/SKILL.md`). If no SLO exists, defining one is the first deliverable, not the load test.

1. Take the user-facing SLO (e.g. *99% of checkout requests complete under 800 ms*).
2. Convert to a **percentile threshold with a load level**: `p99 < 800ms at 500 concurrent users`.
3. Set the error-rate threshold **below** the error budget, not at it — the test must fail before users would notice.
4. Add a throughput floor: sustained requests/second the system must handle without breaching the above.
5. Encode all of them as **pass/fail assertions in the tool**, so the run exits non-zero on breach. A load test with no assertions is a graph nobody reads.

| Traffic input | Source |
|---|---|
| Expected peak concurrency | Production analytics — peak hour, not daily average |
| Request mix | Real endpoint distribution from access logs or APM, not an even split |
| Growth headroom | Test at 2× expected peak; capacity planning targets, not wishes |

---

## What to Measure

| Metric | Rule |
|---|---|
| **Latency percentiles** (p50, p90, p95, p99) | Report percentiles, never averages — an average hides the tail that defines user experience |
| **Throughput** (req/s, completed iterations/s) | The real capacity number; must be reported alongside the concurrency that produced it |
| **Error rate** by status and type | Separate 4xx (script/data problem) from 5xx and timeouts (system problem) |
| **Saturation** | CPU, memory, connection pool usage, queue depth, thread pool, disk I/O, file descriptors |
| **Downstream** | Database slow queries, lock waits, cache hit ratio, external API latency |
| **Client-side saturation** | Load generator CPU/network — a saturated generator invents latency that does not exist |

Interpretation rules:
- A rising p99 with flat p50 means **queuing or contention**, not slow code.
- Throughput plateauing while latency climbs marks the **saturation point** — the breaking point is just after it.
- Errors appearing before latency degrades usually means a hard limit (connection pool, rate limiter, file descriptors), not capacity.
- Correlate every run with server-side observability. Numbers from the load generator alone cannot tell you *why*.

---

## Test Data and Environment Pitfalls

| Pitfall | Consequence | Mitigation |
|---|---|---|
| Same user/record for every virtual user | Unrealistic cache hits and row-level lock contention | Parameterize with a realistic data pool |
| Dataset far smaller than production | Every query hits an in-memory index; results are meaningless | Seed representative volume and cardinality |
| Environment scaled down from production | Results cannot be extrapolated linearly | Match topology, or state the scaling factor explicitly and never extrapolate silently |
| Autoscaling enabled without noting it | Measures the scaler's reaction time, not the system's capacity | Test fixed capacity first, then autoscaling behavior separately |
| Caches/CDN in front absorbing the load | Tests the cache, not the origin | Bypass or vary keys deliberately; test both paths |
| Rate limiters or WAF throttling the generator | 429s reported as system failure | Allowlist the generator, or make throttling part of the scenario |
| Third-party APIs called for real | Cost, bans, and someone else's latency in the results | Stub with realistic latency and error injection |
| Test writes into shared/production data | Data corruption and contaminated analytics | Dedicated tenant/dataset, cleanup routine, never load-test production without written approval |
| Metrics polluting production dashboards | False alerts and paged on-call | Tag all synthetic traffic with a header and exclude it from alerts |

Also budget for **think time** between requests — real users pause. A no-pause script produces a load pattern that no real population creates.

---

## Tool Selection

Gate on the signals already present in the project — a tool the team can read and maintain beats a marginally faster one.

| Signal | Tool |
|---|---|
| JS/TS codebase, CI-first, scriptable scenarios, thresholds as code | `k6` |
| Python codebase, custom logic per user, distributed generators | `Locust` |
| Java/JVM shop, GUI-driven test design, protocol variety (JMS, JDBC, LDAP) | `JMeter` |
| Scala/Java, high per-node throughput, HTML reports | `Gatling` |
| Quick sanity check on a single endpoint | `hey`, `wrk`, `oha`, `autocannon` |
| gRPC services | `ghz` |
| Cloud-managed distributed runs, no generator infrastructure to operate | Managed service from the existing cloud provider |
| Browser-level load (real rendering, JS execution) | Browser-driver mode of the chosen tool — expensive, use only when protocol-level load cannot answer the question |

Rules:
- Store scripts in the repo next to the code they exercise, reviewed like tests.
- Run smoke profiles in CI on every PR; run heavier profiles on a schedule or before releases — a 30-minute soak does not belong in the PR pipeline.
- Record results as artifacts and track them over time; a single run has no trend and therefore no verdict.
- Test through the same entry point real traffic uses (gateway, load balancer, TLS) — bypassing it removes the layers that usually fail first.

---

## Reporting a Run

Every result must state, in one place:

- Profile and duration, target environment and its capacity
- Concurrency/arrival rate and the request mix
- Throughput achieved, latency percentiles, error rate by type
- Saturation of the top resource and what limited it
- Pass/fail against each declared threshold
- The single next bottleneck to address

"It handled the load" without the numbers above is not a result.

---

## Decision Checklist

- [ ] Is there a stated question, and does the chosen profile answer it?
- [ ] Do thresholds derive from an SLO rather than from intuition?
- [ ] Are latency results reported as percentiles, with the concurrency that produced them?
- [ ] Did a smoke run validate the script before the real run?
- [ ] Is test data parameterized and the dataset representative in volume?
- [ ] Is the environment's relationship to production documented?
- [ ] Are caches, autoscaling, rate limiters and third parties accounted for deliberately?
- [ ] Is server-side saturation captured alongside client-side latency?
- [ ] Are thresholds encoded as assertions so the run fails automatically?
- [ ] Is synthetic traffic tagged and excluded from production alerting?
- [ ] Is the result stored for trend comparison against previous runs?