Resilience

---
name: resilience
description: Resilience — circuit breaker, retry, bulkhead, health checks.
---

## Patterns Reference

| Pattern | Problem Solved | When to Use |
|---------|---------------|------------|
| Circuit Breaker | Prevents cascading failures to a degraded dependency | Any synchronous external call |
| Retry + Exponential Backoff | Recovers from transient failures automatically | Idempotent operations, network timeouts, 5xx errors |
| Bulkhead | Isolates resource pools so one slow dependency doesn't block others | Services with multiple downstream dependencies |
| Timeout | Prevents indefinite blocking on slow responses | Every network boundary |
| Fallback | Provides a degraded but functional response on failure | Non-critical dependencies (recommendations, ads, enrichment) |
| Rate Limiter | Protects services from traffic spikes and abuse | Inbound API endpoints, outbound third-party calls |
| HTTP Connection Pooling | Reuses TCP/TLS connections instead of paying handshake cost per call | Any outbound HTTP client making repeated calls to the same host |

## Circuit Breaker

States and transitions:

```
CLOSED → (N consecutive failures) → OPEN → (probe timeout) → HALF-OPEN → (probe succeeds) → CLOSED
                                                                         → (probe fails)    → OPEN
```

- **CLOSED:** requests pass through; failure counter increments on error
- **OPEN:** requests fail immediately with a fallback; no calls to the downstream
- **HALF-OPEN:** one probe request is allowed through to test recovery

Rules:
- Open after N failures within a rolling time window (e.g., 5 failures in 30s)
- Never open on client errors (4xx) — those are the caller's fault, not the downstream's
- Set the probe timeout long enough for the downstream to recover (e.g., 30s – 60s)
- Emit metrics on state transitions; alert on circuit opening

## Retry Policy

- Max **3 retries** (4 total attempts) for most cases
- **Exponential backoff with jitter:**

```
attempt 1 delay: 100ms + rand(0–50ms)
attempt 2 delay: 200ms + rand(0–100ms)
attempt 3 delay: 400ms + rand(0–200ms)
```

- Retry only transient failures: 5xx responses, network timeouts, connection resets
- Do NOT retry: 4xx errors (except 429 with `Retry-After`), validation failures, business logic errors
- Ensure the operation is idempotent before retrying — use idempotency keys for non-idempotent operations
- Combine with circuit breaker: stop retrying when the circuit is open

## Timeout

- Set a timeout at **every** network boundary — database, HTTP call, message broker, internal RPC
- Client-side timeout must be **shorter** than server-side timeout:
  - If the client times out first and retries, the server may process the original request twice
  - Rule of thumb: client timeout = 80–90% of the server's configured processing timeout
- Use separate timeouts for connection establishment and total response time
- Log timeout events with the full call target and duration for observability

## Bulkhead

- Assign a **separate thread pool or connection pool** to each critical downstream dependency
- One pool exhaustion must not prevent calls to other dependencies
- Size pools independently based on expected concurrency and dependency SLA:

```
billing-service pool:    max 20 threads, queue 50
inventory-service pool:  max 10 threads, queue 20
notification-service pool: max 5 threads, queue 10
```

- Reject requests with a clear error (503 + explanation) when the pool is saturated — never silently queue indefinitely
- Monitor pool saturation as a leading indicator of downstream degradation

## Fallback

- Always provide an explicit fallback — never return a silent empty response on failure
- Fallback options in priority order:
  1. Return a cached (possibly stale) result
  2. Return a safe default value (empty list, zero, placeholder)
  3. Return a degraded response (feature disabled message)
  4. Return a clear error to the caller with enough context to retry later
- Never swallow exceptions silently — log the original error even when serving a fallback
- Mark fallback responses in the payload or headers so callers can distinguish real from degraded data

## HTTP Client Connection Pooling

Establishing a TCP connection (plus TLS handshake over HTTPS) is expensive relative to the request itself. A pooled HTTP client keeps connections open and reuses them across calls to the same host instead of opening a fresh one every time. This is a client-side concern — separate from the server-side pooling covered under Bulkhead above, and from database connection pooling (see `skills/integrations/database-production/SKILL.md`).

**Use when:**
- The service makes repeated outbound calls to the same host (internal microservice, third-party API, webhook target)
- Call volume is high enough that per-request handshake latency shows up in p95/p99
- The downstream supports HTTP keep-alive (virtually all modern HTTP/1.1+ and HTTP/2 servers do)

**Skip or scope narrowly when:**
- The call is a true one-off (cold start, cron job hitting a host once) — pooling adds idle-connection overhead with no reuse benefit
- The target host rotates behind short-lived DNS (e.g. some load balancers) — see DNS caching pitfall below

**Every HTTP client library exposes some form of this — names vary by stack, but the concepts don't:**

| Concept | What it controls | Typical failure mode if misconfigured |
|---|---|---|
| Max connections per host | Ceiling on concurrent open connections to one downstream | Too low → requests queue/block waiting for a free connection under load; too high → downstream gets overwhelmed or hits its own connection limits |
| Max idle connections | How many idle (kept-alive) connections stay open for reuse | Too low → connections keep getting closed and reopened, losing the reuse benefit; too high → resource waste, may exhaust downstream connection limits |
| Idle timeout | How long an unused connection stays open before being closed | Too short → connections churn like no pooling was configured; too long → stale connections get used and fail (see below), or resources leak |
| Connection timeout | Max time to wait establishing a new connection | Too long → slow failure detection, threads/goroutines pile up waiting; too short → false failures under normal network jitter |

**Common pitfalls:**
- **Stale connection reuse**: a pooled connection can be silently closed by the server or an intermediate proxy/load balancer while idle. Configure the client to validate or retry-once-on-connection-reset rather than surfacing it as a hard failure to the caller.
- **DNS caching mismatch**: a pooled connection binds to a resolved IP. If the client also caches DNS aggressively, it can keep talking to a decommissioned backend after a deploy or failover. Keep idle timeout shorter than the downstream's DNS TTL when the target is behind a rotating load balancer.
- **One pool per downstream, not one global pool**: mirror the Bulkhead pattern — size the pool for each downstream independently based on its expected concurrency and latency profile, so one saturated dependency doesn't starve connections meant for another.
- **Missing pooling entirely**: the default HTTP client in many languages does *not* pool by default, or pools with a very small ceiling (e.g. 1–2 connections per host) — verify the actual default before assuming reuse is happening.

Consult your stack's HTTP client documentation for the exact parameter names (e.g. `MaxIdleConnsPerHost`, `maxConnections`, `pool_maxsize`, `maxSockets`) — the concepts above map onto all of them.

## Rate Limiting

| Algorithm | Behavior | Best For |
|-----------|----------|---------|
| Token Bucket | Smooth average rate; allows short bursts | APIs, outbound calls |
| Sliding Window | Precise request counting per window | Burst detection, abuse prevention |
| Fixed Window | Simple but allows boundary bursts | Internal rate limiting where precision is less critical |
| Leaky Bucket | Constant output rate regardless of input spikes | Traffic shaping |

- Return `429 Too Many Requests` with a `Retry-After` header specifying seconds until the next allowed request
- Apply rate limits per API key, user ID, or IP — never globally only
- Implement rate limits as close to the edge as possible (API gateway, CDN) to reduce backend load
- Log rate-limit hits with the client identifier for abuse analysis

## Health Checks

Three distinct probes — do not conflate them:

| Probe | Question | Failing means |
|-------|---------|--------------|
| Liveness | Is the process alive and not deadlocked? | Kill and restart the container |
| Readiness | Can this instance serve traffic right now? | Remove from load balancer pool |
| Startup | Has initialization completed? | Delay liveness/readiness checks |

- Expose all probes at `/health` with sub-paths: `/health/live`, `/health/ready`, `/health/startup`
- Readiness checks must verify critical dependencies (DB connection, cache availability)
- Liveness checks must be lightweight — never check external dependencies in liveness
- Return `200` on healthy, `503` on unhealthy; include a JSON body with individual component status
- Set appropriate Kubernetes probe parameters: `initialDelaySeconds`, `periodSeconds`, `failureThreshold`