Sota Performance

---
name: sota-performance
description: >-
  State-of-the-art performance engineering for building fast systems and
  auditing existing code for bottlenecks. Use when the task involves
  performance, optimization, latency, profiling, slow code, memory usage,
  caching, or throughput — designing latency budgets, fixing N+1 and
  accidental-quadratic patterns, tuning allocation/GC pressure, network and
  I/O efficiency, cache architecture, Core Web Vitals, or setting up
  benchmarks and regression gates. Not for concurrency correctness (races,
  deadlocks, cancellation) — use sota-async-concurrency. Trigger keywords: performance,
  optimization, latency, profiling, slow, memory usage, caching, throughput,
  bottleneck, p99, flamegraph, Core Web Vitals.
---

# SOTA Performance Engineering

## Purpose

Make systems fast by default and find why they are slow by evidence. This skill
encodes two disciplines that share one rule set:

1. **BUILD** — write code whose performance characteristics are known, budgeted,
   and protected by regression tests before it ships.
2. **AUDIT** — read existing code and telemetry to locate bottlenecks, rank them
   by user-facing impact, and prescribe fixes with expected gains.

Core doctrine: **measure first, but fix known pathologies on sight.** Profiling
is mandatory before micro-optimization; it is NOT required to remove an O(n²)
loop, an N+1 query, or an unbounded cache. "Premature optimization" never
excuses shipping a known pathology.

## BUILD mode

When writing new code or features:

1. **Set a budget before writing.** Define the latency budget (p99, not average)
   and decompose it across hops. An endpoint with a 200 ms p99 budget that calls
   auth (10 ms) + 2 DB queries (2×15 ms) + serialization (5 ms) has 155 ms of
   headroom — spend it consciously. See `rules/01-methodology.md`.
2. **Choose data structures by access pattern, not habit.** Know the n. n < 100:
   anything works. n unbounded: complexity class is the design.
   See `rules/02-algorithms-data-structures.md`.
3. **Batch and stream at every boundary.** One round trip per collection, not
   per item. Stream large results; never materialize unbounded data.
4. **Control allocation in hot paths.** Pre-size collections, reuse buffers,
   avoid per-iteration allocation in loops that run > 10⁴ times per second.
   See `rules/03-memory.md`.
5. **Make I/O cheap by construction.** Pooled connections, keep-alive, buffered
   writes, compression chosen per payload type. See `rules/04-io-network.md`.
6. **Cache deliberately or not at all.** Every cache ships with: key schema,
   TTL + jitter, eviction policy, invalidation path, stampede protection, and a
   hit-ratio metric. A cache missing any of these is a future incident.
   See `rules/05-caching.md`.
7. **Protect the win.** Add a benchmark or perf test in CI for any code with a
   budget. A perf improvement without a regression gate is a loan, not an asset.
8. **Frontend ships against Core Web Vitals budgets** (LCP ≤ 2.5 s, INP ≤ 200 ms,
   CLS ≤ 0.1 at p75). See `rules/06-frontend-web.md`.

## AUDIT mode

### How to find performance issues by reading code

Work outside-in, hottest path first:

1. **Identify the hot paths.** Entry points with highest traffic or strictest
   SLO: request handlers, queue consumers, render loops, cron jobs over large
   datasets. Audit those first; ignore cold admin paths until the end.
2. **Grep for pathology signatures** (high hit rate, low effort):
   - Loops containing `await`/network/DB calls → N+1 (`rules/02`)
   - String/array concatenation inside loops → accidental quadratic (`rules/02`)
   - `.includes`/`in list`/linear `find` inside a loop → O(n·m) (`rules/02`)
   - `SELECT *`, queries without LIMIT, missing pagination (`rules/02`, `rules/04`)
   - Caches/maps with insert but no eviction or TTL → leak (`rules/03`, `rules/05`)
   - `addEventListener`/subscribe without matching removal (`rules/03`)
   - Sequential awaits on independent operations → serialized latency (`rules/04`)
   - New client/connection per request instead of pooled (`rules/04`)
   - Sync file/crypto/compression calls on async event loops (`rules/04`)
   - `JSON.parse`/serialize of large payloads in hot loops (`rules/03`)
3. **Check the boundaries.** Most production latency lives at boundaries:
   process↔kernel (syscalls), service↔DB, service↔service, server↔browser.
   Count round trips per user action; > 3 sequential round trips is a finding.
4. **Check resource lifecycle.** Anything created per-request that is expensive
   to create (connections, TLS sessions, regexes, compiled templates, clients)
   should be created once and reused.
5. **Check what's missing**: no timeouts, no pagination, no backpressure, no
   pool bounds, no cache eviction — absent code is the most common perf bug.

### What to measure (when you can run the system)

- **Latency distribution**: p50/p95/p99 per endpoint — never averages
  (`rules/01`). Compare p99 to p50; ratio > 10× means contention, GC, or
  stampedes, not slow code.
- **USE per resource** (Utilization, Saturation, Errors): CPU, memory, disk,
  network, pools, queues. **RED per service** (Rate, Errors, Duration).
- **Where time goes**: CPU flamegraph for compute, off-CPU/wall profile for
  waiting. A request that is slow with idle CPU is blocked on I/O or locks.
- **Allocation rate and GC pause time** for managed runtimes.
- **Cache hit ratios** and **DB round trips per request**.
- **Frontend**: field CWV (CrUX/RUM) at p75, not lab-only Lighthouse.

### Severity conventions (by user-facing impact)

| Severity | Criteria |
|---|---|
| **Critical** | Active or imminent user-facing failure: unbounded growth (memory leak, unpaginated scan) that will OOM/timeout at production scale; O(n²)+ on user-controlled input; stampede-capable cache in front of a fragile origin; p99 SLO breached now. |
| **High** | Measurable user-facing degradation: N+1 on a hot path; missing pool/keep-alive adding RTTs per request; blocking call on event loop; CWV in "poor" band; hot-path complexity that degrades super-linearly with organic growth. |
| **Medium** | Wasteful but currently within budget: avoidable allocations in warm paths; missing compression; suboptimal cache TTLs; sequential awaits worth ~10–50 ms; bundle over budget but CWV still "needs improvement". |
| **Low** | Hygiene: micro-inefficiencies in cold paths, style-level fixes, missing benchmarks for non-critical code. |

Escalate one level if the code path is on the critical user journey (checkout,
login, search) or if growth is super-linear with data/users.

### Finding format

```
[SEVERITY] <one-line title>
Location: <file:line(s)>
Pattern: <pathology name, e.g. "N+1 query", "unbounded cache">
Evidence: <code excerpt or metric>
Impact: <quantified or estimated user-facing effect, with the math>
Fix: <specific change, with expected gain>
Verify: <how to confirm the fix: benchmark, profile, metric to watch>
```

Estimate impact with arithmetic, not adjectives: "200 items × 1 query × ~1 ms
RTT = ~200 ms added per page view" beats "this is slow".

## Rules index

| File | Read this when... |
|---|---|
| `rules/01-methodology.md` | You need to profile, benchmark, set latency budgets, interpret percentiles, apply USE/RED, decide what's worth optimizing (Amdahl), or set up CI perf regression gates. |
| `rules/02-algorithms-data-structures.md` | Auditing loops and data access: N+1, accidental quadratics, repeated scans, hash-vs-tree choices, batching, streaming vs materializing, and high-level DB pointers (indexes, SELECT *, chatty transactions). |
| `rules/03-memory.md` | Dealing with allocation pressure, GC pauses, object pooling, arenas, cache locality, SoA vs AoS, or hunting memory leaks (closures, listeners, unbounded caches) per runtime. |
| `rules/04-io-network.md` | Anything crossing a syscall or the wire: buffering, zero-copy, connection pooling, HTTP/2/3, compression choice (zstd/brotli), TLS resumption, CDN, request coalescing, pagination over the wire. |
| `rules/05-caching.md` | Designing or auditing any cache: hierarchy placement, key design, invalidation, stampede protection (singleflight, jitter, soft TTL), negative caching, when caching is the wrong fix, and the security floor (declared global/tenant/user key scope, authorize before a cache hit, invalidate on role/tenancy change). |
| `rules/06-frontend-web.md` | Web performance: Core Web Vitals thresholds, bundle budgets, code splitting, image formats (AVIF/WebP), font loading, hydration cost, edge rendering. |

## Top-10 non-negotiables

1. **Measure before optimizing; fix pathologies on sight.** Profile before
   micro-tuning. But N+1, O(n²) on unbounded input, unbounded caches, and
   sync-blocking the event loop need no profiler — fix them when you see them.
2. **Percentiles, never averages.** Report and budget p50/p95/p99. An average
   hides the 1% of users who hit every cache miss and GC pause.
3. **No I/O inside a loop over a collection.** Batch it, join it, or
   parallelize it with a bound. One round trip per item is always a finding.
4. **Every cache has bounded size, TTL with jitter, an invalidation path, and
   stampede protection.** Otherwise it's a memory leak with a hit ratio.
5. **Pool expensive resources.** Connections, TLS sessions, threads, compiled
   regexes, HTTP clients: create once, reuse always, bound the pool.
6. **Stream unbounded data; never load "all rows" into memory.** Paginate with
   cursors, process in chunks, set LIMITs.
7. **Never block an async event loop** with sync file I/O, crypto, compression,
   or CPU-heavy work. Offload to workers or use async variants.
8. **Set timeouts and bounds on everything**: requests, queries, pools, queues,
   retries (with backoff + jitter). Missing bounds turn slowness into outage.
9. **Sequential awaits on independent work are stolen latency.** Run
   independent I/O concurrently; the latency of the batch is the max, not sum.
10. **Protect every win with a regression gate.** A benchmark in CI with
    variance-aware thresholds, or the regression returns within a quarter.