Qa Resilience

---
name: qa-resilience
description: "Designs and tests distributed-system resilience. Use when adding retries, deadlines, hedging, circuit breakers, overload protection, chaos experiments, or SLO reliability gates."
compatibility: Portable core. Works on Claude Code and Codex.
version: "1.2"
last_validated: 2026-07-11
---

# QA Resilience

## Quick Reference

| Symptom | Start With |
|--------|------------|
| slow or hanging dependency | deadline and timeout budget |
| transient dependency failure | bounded retry with jitter and retry budget |
| sustained dependency failure | circuit breaker and fallback |
| rate-limited dependency | honor `Retry-After`, expose degraded behavior, and test quota paths intentionally |
| one bad host in a healthy pool | outlier detection or endpoint ejection |
| queue or pool saturation | bulkheads, concurrency limits, load shedding |
| static concurrency limit keeps going stale under changing capacity | adaptive concurrency limit with a static backstop; verify the goodput plateau — [load-shedding-backpressure.md](references/load-shedding-backpressure.md#adaptive-concurrency-limits) |
| non-critical feature outage | graceful degradation or feature flag fallback |
| resilience validation | deterministic fault injection before chaos |

## When to Use This Skill

- retries, deadlines, hedging, breakers, bulkheads, and overload protection
- degraded-mode UX or API behavior
- service-mesh or gateway resilience policy
- chaos engineering, game days, DR drills, and fault injection
- release gates based on failure behavior, not only happy-path load tests

## Route Elsewhere

- simple CRUD or single-process utilities -> ordinary error handling may be enough
- frontend-only failure behavior -> [software-frontend](../software-frontend/SKILL.md)
- service implementation details -> [software-backend](../software-backend/SKILL.md)
- telemetry instrumentation -> [qa-observability](../qa-observability/SKILL.md)
- broader platform and incident operating model -> [ops-devops-platform](../ops-devops-platform/SKILL.md)

---

## Workflow

1. Identify the critical user journeys and the dependencies that can break them.
2. Define the contract per dependency:
   - timeout or deadline budget
   - retry ownership
   - breaker or outlier policy
   - concurrency and queue limits
   - degraded behavior if the dependency is unavailable
3. Decide where the policy lives:
   - app code
   - client library
   - mesh or gateway
4. Test in stages:
   - design review or checker output establishes policy coherence only
   - deterministic fault injection proves the response to a named controlled fault
   - staged chaos in non-production proves bounded behavior and recovery in that environment
   - narrow prod canary or game day proves only the observed target path and window, with guardrails
5. Define pass or fail signals:
   - burn rate
   - p95 or p99
   - fallback rate
   - breaker transitions
   - shed volume
   - recovery time
6. Record the injected fault, steady-state hypothesis, blast-radius limit, abort condition, observation window, recovery result, and dependencies simulated rather than observed.

---

## Selective verification and hazard analysis

- Use [formal methods](../foundations-formal-methods/SKILL.md) when the question is
  whether a recovery/failover protocol preserves an invariant under concurrent
  faults, or when a counterexample is needed. Return the property, initial
  states, transitions/environment assumptions, checked bound, result/trace, and
  model-to-code limits. A finite invariant check does not establish liveness or
  recovery time. Skip for routine timeout/retry configuration and named fault
  tests whose behavior is already directly observable.
- Use [safety engineering](../foundations-safety-engineering/SKILL.md) when
  retries, rollback, failover, operators, or other functioning controls can
  interact to cause a specified unacceptable loss, such as duplicate financial
  effects or destructive recovery. Return loss → hazard → unsafe control action
  and context → constraint → owner → verification evidence, including residual
  unknowns. Skip for ordinary availability or latency tuning without a harmful
  interaction question. Uptime and a hazard diagram do not prove safety.

Keep resilience policy, fault-injection execution, recovery evidence, and release
recommendations with this skill; use the foundation only for the named gap.

## Pattern Rules

- retries happen at one layer only
- deadlines come before retries
- hedging is only for idempotent or cancellation-safe reads
- overload handling must shed early instead of collapsing late
- overload tests pass on goodput plateauing past capacity, not on "503s appeared"; shed responses are retried only at the retry-owning layer, within budget, after `Retry-After`
- in regulated firms (EU DORA, UK FCA/PRA operational resilience), test each important business service against its impact tolerance, third parties included; counsel owns scoping — see [disaster-recovery-testing.md](references/disaster-recovery-testing.md#regulated-financial-firms-eu-dora-and-uk-operational-resilience)
- readiness and liveness must stay bounded and shallow
- resilience behaviors belong in targeted checks; do not let rate-limit or degraded-mode coverage leak into unrelated happy-path suites
- prod experiments require blast-radius limits, abort criteria, dashboards, and owners

## Failure Modes to Validate

- timeouts and deadline propagation
- retry storms and duplicate side effects
- partial dependency outages
- slow downstreams and long-tail latency
- queue buildup and connection-pool exhaustion
- one-bad-host behavior inside a pool
- degraded-mode responses and stale-data fallbacks
- rate-limit handling, `Retry-After`, and client backoff expectations
- visible state convergence after recovery or backend resets
- failover and failback behavior
- metastable failure: a self-sustaining feedback loop (retry amplification, cache-miss stampede, queue backlog, connection-pool churn) that does not self-resolve after the original trigger clears — see [references/cascading-failure-prevention.md](references/cascading-failure-prevention.md#metastable-failures--the-class-cascading-failure-fixes-do-not-cure)

---

## Testing Ladder

### Deterministic first

- inject latency, errors, timeouts, malformed payloads, and unavailable endpoints in controlled tests
- verify the intended control activates and the wrong controls do not

### Chaos second

- start in non-production
- use a small blast radius and a fixed time window
- stop immediately on error-budget or customer-impact breach

### DR and game days last

- validate RTO and RPO claims explicitly
- rehearse recovery ownership, not only technical failover

---

## Operational Guardrails

- every experiment needs a stated hypothesis and steady-state metric
- every run should capture timestamps, targets, blast radius, and dashboard links
- telemetry fields for retries, breaker transitions, hedging, shedding, and fallback are part of the resilience contract
- releases should be gated on reliability behavior, not only resource usage

## Anti-Patterns

- no timeouts
- retries at every hop
- fixed-interval or unbounded retries
- fixed per-call retry count with no system-wide retry budget (does not tighten as error rate rises)
- retries without idempotency
- hedging unsafe writes
- no bulkheads or queue bounds
- deep readiness or liveness checks
- silent degraded mode
- untested failover plans
- happy-path-only load testing

---

## Scripts

| Script | Purpose |
|--------|---------|
| [scripts/resilience_checker.py](scripts/resilience_checker.py) | Scores declared pattern coverage and reports gaps; rejects malformed claims |

Typical usage:

```bash
python scripts/resilience_checker.py assess --input data/sample-service-profile.json
python scripts/resilience_checker.py gaps --input data/sample-service-profile.json
python scripts/resilience_checker.py report --input data/sample-service-profile.json --output resilience-report.md
```

Supply actual JSON booleans and explicit hedging N/A rationale. The heuristic score describes declared coverage; it does not establish fault behavior or recovery. See [scripts/README.md](scripts/README.md) for the input format and scoring logic.

## ASCII Flow

```text
Resilience request
  -> Identify failure mode: slow, transient, sustained, overloaded, or degraded
  -> Set deadlines, retry budgets, isolation, and fallback ownership
  -> Add telemetry for saturation, errors, latency, and degraded behavior
  -> Validate with deterministic fault injection before chaos experiments
  -> Gate release on recovery evidence, SLO impact, and rollback path
  -> Document runbook actions and residual risk
```

## Navigation

### Foundation applied recipes

- [references/reliability-theory-applied.md](references/reliability-theory-applied.md)
- [references/distributed-systems-applied.md](references/distributed-systems-applied.md)

### Core references

- [references/circuit-breaker-patterns.md](references/circuit-breaker-patterns.md)
- [references/retry-patterns.md](references/retry-patterns.md)
- [references/timeout-policies.md](references/timeout-policies.md)
- [references/deadlines-hedging.md](references/deadlines-hedging.md)
- [references/bulkhead-isolation.md](references/bulkhead-isolation.md)
- [references/load-shedding-backpressure.md](references/load-shedding-backpressure.md)
- [references/gateway-mesh-resilience.md](references/gateway-mesh-resilience.md)
- [references/graceful-degradation.md](references/graceful-degradation.md)
- [references/health-check-patterns.md](references/health-check-patterns.md)
- [references/cascading-failure-prevention.md](references/cascading-failure-prevention.md)
- [references/disaster-recovery-testing.md](references/disaster-recovery-testing.md)

### Operational resources

- [references/resilience-checklists.md](references/resilience-checklists.md)
- [references/chaos-engineering-guide.md](references/chaos-engineering-guide.md)
- [references/chaos-tooling-recipes.md](references/chaos-tooling-recipes.md)
- [references/idempotency-key-design.md](references/idempotency-key-design.md)
- [references/resilience-telemetry.md](references/resilience-telemetry.md)
- [references/slo-as-code.md](references/slo-as-code.md)
- [references/ai-llm-resilience-failure-modes.md](references/ai-llm-resilience-failure-modes.md)
- [assets/runbooks/resilience-runbook-template.md](assets/runbooks/resilience-runbook-template.md)
- [assets/testing/fault-injection-playbook.md](assets/testing/fault-injection-playbook.md)
- [assets/testing/template-resilience-test-plan.md](assets/testing/template-resilience-test-plan.md)
- [data/sample-service-profile.json](data/sample-service-profile.json)

## Related Skills

- [ops-devops-platform](../ops-devops-platform/SKILL.md)
- [software-backend](../software-backend/SKILL.md)
- [software-architecture-design](../software-architecture-design/SKILL.md)
- [qa-observability](../qa-observability/SKILL.md)
- [qa-debugging](../qa-debugging/SKILL.md)
- [software-security-appsec](../software-security-appsec/SKILL.md)
- [data-sql-optimization](../data-sql-optimization/SKILL.md)
- [dev-api-design](../dev-api-design/SKILL.md)

## Learnings Loop

When prior decisions or pitfalls are relevant, consult `learnings.consolidated.md` if present; use `learnings.md` only for needed history or as the available fallback. Otherwise skip both.

After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to `learnings.md` via `agents-skills-feedback-loop/scripts/append_learning.py`. Do not modify `SKILL.md` itself.