Harness Resilience

# Harness Resilience

> Circuit breakers, rate limiting, bulkheads, retry patterns, and fault tolerance analysis. Detects missing resilience patterns, evaluates failure modes, and recommends concrete configurations for production-grade fault tolerance.

## When to Use

- When adding new external service integrations (APIs, databases, message queues) that need fault tolerance
- On PRs that modify service-to-service communication, HTTP clients, or middleware chains
- To audit existing resilience patterns for correctness, completeness, and observability
- NOT for load testing or capacity planning (use harness-load-testing)
- NOT for incident response after a failure has occurred (use harness-incident-response)
- NOT for security-focused rate limiting like DDoS protection (use harness-security-review)

## Process

### Phase 1: DETECT -- Identify Dependencies and Existing Patterns

1. **Inventory external dependencies.** Scan the codebase for outbound connections:
   - HTTP clients: `axios`, `fetch`, `got`, `HttpClient`, `RestTemplate`, `reqwest`
   - Database connections: connection pool configs, ORM initialization, query builders
   - Message queues: RabbitMQ, Kafka, SQS, Redis pub/sub client initialization
   - gRPC channels: proto client stubs, channel creation, dial options
   - Third-party SDKs: Stripe, Twilio, SendGrid, AWS SDK calls

2. **Map existing resilience patterns.** For each dependency found, check for:
   - Circuit breakers: `opossum`, `cockatiel`, `Polly`, `resilience4j`, `hystrix` usage
   - Retry logic: exponential backoff, jitter, max attempts configuration
   - Timeouts: connection and request timeout settings
   - Rate limiters: token bucket, sliding window, or fixed window implementations
   - Bulkheads: thread pool isolation, semaphore limits, connection pool sizing
   - Fallbacks: cache-aside patterns, default values, degraded responses

3. **Detect anti-patterns.** Flag common resilience mistakes:
   - Unbounded retries without backoff or max attempts
   - Missing timeouts on HTTP clients or database queries
   - Circuit breaker without a fallback handler
   - Retry on non-idempotent operations (POST, DELETE without idempotency keys)
   - Rate limiter with no monitoring or alerting on limit hits

4. **Build the dependency map.** Produce a structured inventory:
   - Dependency name, type (HTTP, gRPC, database, queue), criticality (critical, degraded, optional)
   - Current resilience patterns applied (or "none")
   - Identified gaps and anti-patterns

---

### Phase 2: ANALYZE -- Evaluate Failure Modes

1. **Classify failure modes per dependency.** For each external dependency:
   - **Timeout:** The dependency responds too slowly or not at all
   - **Error burst:** The dependency returns errors at a rate above normal
   - **Partial degradation:** The dependency responds but with reduced functionality
   - **Total outage:** The dependency is completely unreachable
   - **Data inconsistency:** The dependency returns stale or incorrect data

2. **Assess blast radius.** For each failure mode:
   - Which features become unavailable?
   - Which downstream services are affected?
   - What is the user-visible impact?
   - Can the system continue to serve other requests?

3. **Evaluate current coverage.** Score each dependency on resilience coverage:
   - **Full:** Circuit breaker + retry + timeout + fallback + monitoring
   - **Partial:** Some patterns present but gaps exist (e.g., retry without circuit breaker)
   - **None:** No resilience patterns applied

4. **Prioritize gaps by risk.** Combine criticality and coverage:
   - Critical dependency with no resilience = P0 (immediate)
   - Critical dependency with partial resilience = P1 (next sprint)
   - Optional dependency with no resilience = P2 (backlog)
   - Any dependency with anti-patterns = P0 (anti-patterns are active risks)

5. **Check observability.** For existing patterns, verify they emit metrics:
   - Circuit breaker state changes (open/half-open/closed)
   - Retry attempt counts and final outcomes
   - Rate limiter rejection counts
   - Timeout occurrences

---

### Phase 3: DESIGN -- Recommend Resilience Patterns

1. **Select patterns per dependency.** Based on the failure mode analysis:
   - **HTTP APIs:** Circuit breaker (opossum/cockatiel) + exponential backoff with jitter + request timeout + fallback
   - **Databases:** Connection pool sizing + query timeout + read replica fallback + bulkhead isolation
   - **Message queues:** Dead letter queue + retry with backoff + idempotent consumers + circuit breaker on publish
   - **gRPC services:** Deadline propagation + retry policy + load balancing + circuit breaker

2. **Provide concrete configurations.** For each recommended pattern, specify:
   - Library and version to use
   - Configuration values with rationale (e.g., "timeout: 3000ms based on p99 latency of 1200ms with 2.5x headroom")
   - Threshold values for circuit breakers (failure rate, sample window, reset timeout)
   - Retry parameters (max attempts, base delay, max delay, jitter factor)
   - Rate limits (requests per window, window size, burst allowance)

3. **Design fallback strategies.** For each critical dependency:
   - **Cache fallback:** Serve stale data from Redis/memory cache with a staleness indicator
   - **Default fallback:** Return a safe default value with a degraded flag
   - **Queue fallback:** Accept the request and process it asynchronously when the dependency recovers
   - **Feature flag fallback:** Disable the feature entirely via feature flag

4. **Generate implementation templates.** Produce code snippets for:
   - Circuit breaker wrapping an existing HTTP client
   - Retry middleware with exponential backoff and jitter
   - Rate limiter middleware for Express/Fastify/NestJS
   - Bulkhead pattern using semaphore or connection pool limits

5. **Define health check contracts.** Specify how each dependency should be health-checked:
   - Endpoint or query to use for liveness check
   - Timeout for the health check itself
   - Frequency and failure threshold before marking unhealthy

---

### Phase 4: VALIDATE -- Verify Implementation and Observability

1. **Check pattern correctness.** For each implemented pattern:
   - Circuit breaker: Verify threshold configuration, half-open behavior, and reset timeout
   - Retry: Verify idempotency of retried operations, backoff curve, and max attempts
   - Timeout: Verify timeout values are set on both client and server sides
   - Rate limiter: Verify limit values, window type, and rejection response format

2. **Verify test coverage.** Check that resilience patterns are tested:
   - Circuit breaker tests: closed-to-open transition, open rejection, half-open recovery
   - Retry tests: successful retry, max attempts exhaustion, non-retryable error bypass
   - Timeout tests: timeout triggers fallback, timeout does not leak connections
   - Rate limiter tests: under-limit passes, at-limit rejects, window reset behavior

3. **Verify observability.** Confirm that metrics are emitted:
   - Check for Prometheus counters/histograms or StatsD calls on pattern events
   - Verify structured logging includes circuit breaker state, retry attempt number, and rate limit headers
   - Confirm dashboard or alert configurations reference the new metrics

4. **Produce the resilience report.** Output a summary:
   - Number of dependencies analyzed
   - Coverage before and after (percentage with full/partial/none resilience)
   - Anti-patterns found and resolved
   - Remaining gaps with priority and recommended timeline

5. **Run integration verification.** If integration tests exist:
   - Execute tests that exercise the resilience patterns (chaos test stubs, fault injection)
   - Verify graceful degradation under simulated failure conditions
   - Confirm that fallbacks produce acceptable user-facing responses

---

## Harness Integration

- **`harness skill run harness-resilience`** -- Primary CLI entry point. Runs all four phases.
- **`harness validate`** -- Run after implementing recommended patterns to verify project integrity.
- **`harness check-deps`** -- Verify that new resilience libraries are properly declared and within boundary rules.
- **`emit_interaction`** -- Used at pattern selection (checkpoint:decision) when multiple valid patterns exist and trade-offs require human judgment.
- **`Glob`** -- Discover HTTP clients, middleware chains, and existing resilience pattern files.
- **`Grep`** -- Search for timeout configurations, retry logic, circuit breaker initialization, and anti-patterns.
- **`Write`** -- Generate implementation templates and resilience configuration files.
- **`Edit`** -- Add resilience wrappers to existing service clients.

## Success Criteria

- All external dependencies are inventoried with their resilience coverage level
- Anti-patterns are identified with specific file locations and line numbers
- Recommendations include concrete library versions and configuration values, not just pattern names
- Fallback strategies are defined for every critical dependency
- Implementation templates compile and follow the project's existing code style
- Observability is addressed: every pattern emits metrics or structured logs

## Examples

### Example: Express.js API with Stripe and PostgreSQL

```
Phase 1: DETECT
  Dependencies found:
    - Stripe API (HTTP, critical): axios client in src/payments/stripe-client.ts
      Resilience: timeout=5000ms, no retry, no circuit breaker, no fallback
    - PostgreSQL (database, critical): pg pool in src/db/pool.ts
      Resilience: pool max=20, no query timeout, no read replica fallback
    - SendGrid (HTTP, optional): @sendgrid/mail in src/notifications/email.ts
      Resilience: none

  Anti-patterns:
    - src/payments/stripe-client.ts:45 — retry on POST /charges without idempotency key
    - src/db/pool.ts — no statement_timeout configured

Phase 2: ANALYZE
  Stripe failure modes:
    - Timeout: Payment page hangs, user retries, duplicate charges possible
    - Outage: All payments fail, revenue impact immediate
    - Blast radius: checkout flow, subscription renewal, refund processing
  Risk: P0 (critical + partial coverage + anti-pattern)

Phase 3: DESIGN
  Stripe recommendations:
    - Add opossum circuit breaker: failureThreshold=50%, resetTimeout=30s
    - Add idempotency key to all Stripe charge requests
    - Set timeout to 8000ms (Stripe p99 is ~3s, 2.5x headroom)
    - Fallback: queue payment for async retry via Bull queue
  PostgreSQL recommendations:
    - Set statement_timeout=5000 in pool config
    - Add pg-pool error handler with connection retry
    - Configure read replica for GET endpoints via pgBouncer

Phase 4: VALIDATE
  Resilience coverage: 33% -> 100% (3/3 dependencies covered)
  Anti-patterns resolved: 2/2
  Tests needed: circuit breaker state transitions, idempotency key generation
```

### Example: NestJS Microservices with gRPC and Redis

```
Phase 1: DETECT
  Dependencies found:
    - user-service (gRPC, critical): @grpc/grpc-js in src/clients/user.client.ts
      Resilience: deadline=5s, no retry, no circuit breaker
    - inventory-service (gRPC, critical): no resilience configured
    - Redis (cache, degraded): ioredis in src/cache/redis.ts
      Resilience: reconnectOnError, no bulkhead, no fallback

Phase 2: ANALYZE
  inventory-service outage:
    - Product pages return 503, search results empty
    - Blast radius: catalog, search, cart validation
    - Risk: P0 (critical + no coverage)

Phase 3: DESIGN
  inventory-service recommendations:
    - Add cockatiel circuit breaker with ConsecutiveBreaker(5)
    - Add retry with exponentialBackoff(1000, 2) maxAttempts=3
    - Add deadline propagation from gateway timeout
    - Fallback: serve cached inventory from Redis with staleness header
  Redis recommendations:
    - Add bulkhead: maxPoolSize=50, separate pools for cache vs sessions
    - Add fallback: in-memory LRU cache (lru-cache, max 1000 items)
    - Monitor: emit redis.command.duration histogram

Phase 4: VALIDATE
  Coverage: 33% -> 100%
  Tests verified: gRPC circuit breaker opens after 5 failures,
    Redis fallback serves from LRU when Redis is down
```

## Rationalizations to Reject

| Rationalization                                                                     | Reality                                                                                                                                                                                                                                                                                                                        |
| ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| "That third-party API has 99.99% uptime — we don't need a circuit breaker"          | 99.99% uptime means 52 minutes of downtime per year. That downtime will not occur as one predictable window — it will happen as degraded responses and timeouts during a traffic spike. Without a circuit breaker, every caller blocks for the full timeout duration, exhausting thread pools and cascading across the system. |
| "We have retry logic, so failures are handled"                                      | Retry logic without a circuit breaker amplifies failures. When the downstream service is degraded, retries multiply the load on an already struggling system. Circuit breakers and retries are complementary controls, not alternatives.                                                                                       |
| "The fallback adds complexity — we'll add it if the circuit breaker actually opens" | A circuit breaker without a fallback is a different kind of failure mode, not resilience. When the circuit opens, users see an error instead of a degraded-but-functional experience. Fallbacks must be designed and tested before the circuit ever opens in production.                                                       |
| "Our database connection pool is 100 connections — that's plenty"                   | Connection pool size without query timeouts means slow queries hold connections indefinitely. A single slow query spike can exhaust the pool, causing every subsequent request to wait. Pool sizing and query timeouts are both required.                                                                                      |
| "The service is internal — it doesn't need rate limiting"                           | Internal services are often called by automated processes, CI pipelines, and batch jobs that can spike traffic in ways user-facing services do not. Missing rate limiting on internal services is a common cause of self-inflicted outages during deployments and data migrations.                                             |

## Gates

- **No retry on non-idempotent operations without idempotency keys.** Retrying a POST or DELETE that lacks an idempotency mechanism can cause data duplication or data loss. This is a blocking finding. The operation must be made idempotent before retry logic is added.
- **No circuit breaker without a fallback.** A circuit breaker that opens and returns a raw error to the user is not resilience -- it is a different kind of failure. Every circuit breaker must have a defined fallback behavior (cache, default, queue, or feature flag).
- **No unbounded retries.** Retry logic must have a max attempt limit and use exponential backoff with jitter. Unbounded retries with fixed delays cause thundering herd problems and amplify failures.
- **No resilience pattern without observability.** A circuit breaker that opens silently is invisible to operations. Every pattern must emit metrics or structured logs that can trigger alerts.

## Escalation

- **When a dependency has no documentation on failure behavior:** Report: "The [dependency] has no documented error codes or failure modes. Recommend contacting the provider for SLA details, or instrumenting the client to collect failure statistics over a 2-week baseline period."
- **When resilience patterns conflict with latency requirements:** Adding retries and circuit breakers increases tail latency. Report: "The recommended retry configuration adds up to [N]ms to worst-case latency. If the latency budget is [M]ms, consider reducing max attempts or using a hedged request pattern instead."
- **When the team has no experience with the recommended library:** Report: "The team has not used [library] before. Recommend starting with a single non-critical dependency as a pilot, with a production bake time of 2 weeks before rolling out to critical paths."
- **When existing resilience patterns use a different library than recommended:** Do not recommend switching libraries mid-project. Report: "The project already uses [existing library] for resilience. Recommend continuing with [existing library] for consistency, adapting the configuration recommendations to its API."