System Design Resilience Ops

---
name: system-design-resilience-ops
description: "Make a design survivable and operable: eliminate single points of failure, pick failover topology, set RPO/RTO, define readiness signals and observability, choose a deployment and rollback strategy. Use when reviewing availability, planning DR, or deciding rollout mechanics."
metadata:
  triggers:
    keywords:
      - single point of failure
      - failover
      - disaster recovery
      - rpo
      - rto
      - multi-region
      - autoscaling
      - deployment strategy
      - blue-green
      - canary
---

# Resilience and Operations

## **Priority: P1 (HIGH)**

A design is not done until its failure and its rollout are designed.

## SPOF Elimination

- Walk every component and ask what happens when exactly one instance dies, then when the whole zone dies.
- Any component with one instance, one writer, or one shared config plane is a single point of failure. Name it or remove it.
- Redundancy only helps when failure modes are independent: shared credentials, shared config, and a shared control plane cancel the benefit.
- Blast radius: state which users or flows are affected per component failure, and cap it with cells, bulkheads, or per-tenant quotas.

## Failover and Recovery

| Topology | Recovery time | Cost | Fits |
| --- | --- | --- | --- |
| Single region, multi-AZ | Minutes, automatic | Low | Most products |
| Active-passive across regions | Minutes to hours, drill-dependent | Medium | Regulated or high-value flows |
| Active-active across regions | Seconds | High | Global low-latency, conflict-tolerant data |

- Set **RPO** (tolerable data loss) and **RTO** (tolerable downtime) as numbers before choosing a topology; the numbers pick the topology, not the reverse.
- Untested failover is a hypothesis. Schedule a drill and record the measured RTO against the target.
- Backups need a restore test. A backup that has never been restored is not a backup.

## Observability

- Instrument the four signals per service: traffic, error rate, latency percentiles, saturation.
- Alert on user-visible symptoms and on error-budget burn rate, not on raw CPU.
- Propagate a trace and correlation id across every hop, including queue messages.
- Every alert needs an owner, a runbook link, and a defined next action; an alert nobody acts on is noise.

## Rollout

| Strategy | Blast radius | Rollback | Cost |
| --- | --- | --- | --- |
| Rolling | Grows during the roll | Roll forward or back, slow | Low |
| Blue-green | Full switch at cutover | Instant switch back | Double capacity |
| Canary | Small cohort first | Stop and drain the cohort | Needs routing plus metrics |
| Feature flag | Per user or tenant | Instant, no redeploy | Flag lifecycle debt |

- Schema and code deploy separately: expand, migrate, contract. Never ship a migration that only the new code can read.
- Define the rollback trigger as a metric threshold and a time box before the deploy starts.

## Anti-Patterns

- **No untested failover**: no DR claim without a drill date and a measured RTO.
- **No unbounded retry**: retries need budget, backoff with jitter, and a stop condition, or they amplify an outage.
- **No liveness probe on dependencies**: a downstream outage must not restart the fleet.
- **No deploy without rollback**: irreversible releases are outages waiting for a bad build.
- **No autoscaling without a floor and ceiling**: unbounded scaling turns a bug into a bill.

## References

- [Reliability Operations](references/reliability-operations.md) - failure drills, health check design, DR runbook shape, scaling policy notes