Run Chaos Experiment

---
name: run-chaos-experiment
locale: caveman-ultra
source_locale: en
source_commit: 82c77053
fence_basis_commit: 82c77053
translator: "Julius Brussee homage — caveman"
translation_date: "2026-05-03"
description: >
  Chaos eng experiments via Litmus|Chaos Mesh. Test resilience → controlled fault injection, hypothesis-driven, improve failure recovery. Use → pre-launch, post-arch change, GameDays|DR drills, validate failure mode assumptions, SRE maturity.
license: MIT
allowed-tools: Read Write Edit Bash Grep Glob
metadata:
  author: Philipp Thoss
  version: "1.0"
  domain: observability
  complexity: advanced
  language: multi
  tags: chaos-engineering, litmus, chaos-mesh, resilience, fault-injection
---

# Run Chaos Experiment

Inject controlled failures → test+improve resilience.

## Use When

- Pre-major launch (load test)
- Post-arch change (validate resilience)
- GameDays|DR drills
- Validate failure mode assumptions
- SRE maturity program

## In

- **Required**: K8s cluster (Litmus|Chaos Mesh)
- **Required**: Steady-state def ("normal")
- **Required**: Hypothesis ("API up if 1 pod crashes")
- **Optional**: Observability (Prometheus, Grafana) → measure impact
- **Optional**: Rollback plan

## Do

### Step 1: Define Steady State + Hypothesis

```markdown
## Steady State Definition

### Service: API Gateway
- **Availability**: 99.9% (< 0.1% error rate)
- **Latency**: p95 < 200ms
- **Throughput**: 1000 req/s
- **Dependencies**: Database (Postgres), Cache (Redis), Auth Service

### Metrics
- `rate(http_requests_total{job="api"}[5m])`
- `histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))`
- `rate(http_requests_total{status=~"5.."}[5m])`

## Hypothesis
**"If one API pod is killed, the remaining pods will handle the load with <5s
disruption and no increase in error rate."**

### Validation Criteria
- Error rate remains <1%
- p95 latency stays <300ms (50ms grace)
- Service recovers within 5 seconds
- No cascading failures to downstream services
```

→ Clear, measurable normal + success criteria.

If err: can't define steady state → observability insufficient. Add metrics first.

### Step 2: Blast Radius Limits

Scope → minimize risk:

```yaml
# chaos-config.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: chaos-testing

---
# Label pods participating in chaos experiments
apiVersion: v1
kind: Pod
metadata:
  labels:
    chaos-enabled: "true"
    environment: staging  # NEVER production for first run
```

Safeguards:

```markdown
## Blast Radius Controls

### Environment
- **Scope**: Staging only (first 5 runs)
- **Production**: Only after 5 successful staging runs
- **Timing**: Business hours (09:00-17:00 local), never weekends/holidays

### Target Selection
- **Limit**: Max 1 pod per service
- **Percentage**: Max 25% of replicas
- **Exclusions**: Database, payment service, auth service (critical path)

### Auto-Abort Conditions
- Error rate >10% for >30 seconds
- Customer-facing alerts fire
- Manual abort signal from on-call engineer

### Rollback Plan
- Kubernetes will auto-restart killed pods
- Manual rollback: `kubectl rollout undo deployment/api`
- Incident declared if recovery takes >5 minutes
```

→ Clear bounds, won't take down whole sys.

If err: blast too large → narrow. Start non-critical service.

### Step 3: Install Chaos Mesh

```bash
# Add Chaos Mesh Helm repo
helm repo add chaos-mesh https://charts.chaos-mesh.org
helm repo update

# Install Chaos Mesh in isolated namespace
helm install chaos-mesh chaos-mesh/chaos-mesh \
  --namespace chaos-mesh \
  --create-namespace \
  --set dashboard.create=true \
  --set controllerManager.replicaCount=1

# Verify installation
kubectl get pods -n chaos-mesh

# Access dashboard
kubectl port-forward -n chaos-mesh svc/chaos-dashboard 2333:2333
# Open http://localhost:2333
```

Alt: Litmus (vendor-neutral):

```bash
# Install Litmus
kubectl apply -f https://litmuschaos.github.io/litmus/litmus-operator-v2.14.0.yaml

# Wait for Litmus pods
kubectl get pods -n litmus

# Install Litmus CRDs
kubectl apply -f https://hub.litmuschaos.io/api/chaos/master?file=charts/generic/experiments.yaml
```

→ Chaos Mesh|Litmus running, dashboard accessible.

If err: check RBAC. Tools need cluster-wide access.

### Step 4: Create+Exec Experiment

Pod Kill (Chaos Mesh):

```yaml
# pod-kill-experiment.yaml
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: api-pod-kill-test
  namespace: chaos-testing
spec:
  action: pod-kill
  mode: one  # Kill one pod only
  selector:
    namespaces:
      - production
    labelSelectors:
      app: api-gateway
      chaos-enabled: "true"
  duration: "30s"
  scheduler:
    cron: "@every 5m"  # Repeat every 5 minutes (for sustained testing)
```

Apply:

```bash
# Apply experiment
kubectl apply -f pod-kill-experiment.yaml

# Watch experiment status
kubectl get podchaos -n chaos-testing -w

# View detailed status
kubectl describe podchaos api-pod-kill-test -n chaos-testing

# Check which pods were affected
kubectl get events -n production --sort-by=.metadata.creationTimestamp | grep api-gateway
```

Monitor Grafana:

```promql
# Error rate during experiment
rate(http_requests_total{status=~"5..", job="api"}[1m])

# Latency spike
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket{job="api"}[1m]))

# Pod restarts
rate(kube_pod_container_status_restarts_total{pod=~"api-.*"}[5m])
```

→ Pod killed, K8s restarts, service continues w/ minor blip.

If err: err spike|service degrades → abort + investigate.

### Step 5: Analyze + Iterate

```markdown
# Chaos Experiment Report: API Pod Kill

**Date**: 2025-02-09
**Hypothesis**: API stays available if one pod crashes
**Tool**: Chaos Mesh
**Environment**: Staging
**Duration**: 30 seconds (pod kill + recovery)

## Results

### Metrics During Experiment
- **Error Rate**: Increased from 0.1% to 2.3% (spike lasted 8 seconds)
- **p95 Latency**: Increased from 180ms to 450ms (spike lasted 12 seconds)
- **Recovery Time**: 8 seconds (pod restart + load balancer update)

### Hypothesis Outcome
**FAILED**: Error rate exceeded 1% threshold, latency spike >300ms

## Root Cause Analysis
- Load balancer continued routing to killed pod for 8 seconds (stale endpoint)
- Readiness probe set to 10s interval (too slow)
- No pre-stop hook to drain connections gracefully

## Improvements Made
1. **Reduced readiness probe interval**: 10s → 2s
2. **Added pre-stop hook**: 5-second sleep for connection draining
3. **Tuned load balancer**: Enabled faster endpoint updates

## Follow-Up Experiment
- Re-run with same parameters in 1 week
- Expected: Error rate <1%, recovery <5s
```

Track in log:

```bash
# chaos-experiment-log.csv
date,experiment,environment,status,error_rate_peak,recovery_time_s,outcome
2025-02-09,pod-kill-api,staging,complete,2.3%,8,failed
2025-02-16,pod-kill-api,staging,complete,0.8%,4,passed
2025-02-23,network-delay-db,staging,aborted,15%,N/A,failed
```

→ Learnings captured, fixes implemented, follow-up scheduled.

If err: no action post-exp = chaos theater. Prioritize fixes.

### Step 6: Graduate to Prod (Carefully)

After consistent staging passes:

```yaml
# Production pod-kill experiment (more conservative)
apiVersion: chaos-mesh.org/v1alpha1
kind: PodChaos
metadata:
  name: api-pod-kill-prod
  namespace: chaos-testing
spec:
  action: pod-kill
  mode: one
  selector:
    namespaces:
      - production
    labelSelectors:
      app: api-gateway
      chaos-enabled: "true"
  duration: "10s"  # Shorter than staging
  scheduler:
    cron: "0 10 * * 2"  # Tuesdays at 10 AM only (predictable, low-risk time)
```

Prod safeguards:

```bash
# Create a kill switch for production chaos
kubectl create configmap chaos-killswitch \
  -n chaos-testing \
  --from-literal=enabled=true

# Update experiments to check kill switch
# (implementation depends on chaos tool)
```

→ Prod runs in low-risk windows w/ kill switch ready.

If err: prod exp causes incident → disable immediately + post-mortem.

## Check

- [ ] Steady state + hypothesis defined
- [ ] Blast radius limited (env, scope, timing)
- [ ] Tool installed + tested
- [ ] Exp runs in staging
- [ ] Results documented w/ metrics + analysis
- [ ] Improvements implemented
- [ ] Follow-up validates fixes
- [ ] Prod only after 5+ staging successes

## Traps

- **No hypothesis**: "See what happens" wastes time. Always have one.
- **Too broad scope**: Kill all pods = DR test, not resilience. Start small.
- **Prod-first**: Never first run in prod. Staging first, always.
- **Ignore results**: Chaos w/o action = theater. Fix what you learn.
- **Alert fatigue**: Exps trigger alerts. Annotate Grafana|silence expected.
- **No abort plan**: Need kill switch ready.

## →

- `setup-prometheus-monitoring` — metrics to measure exp impact
- `configure-alerting-rules` — alerts during chaos (expected)
- `define-slo-sli-sla` — steady state tied to SLOs