Network Partition Recovery

---
name: network-partition-recovery
enabled: true
description: |
  Use when performing network partition recovery — runbook for diagnosing and
  recovering from network partition events across distributed systems. Covers
  partition detection, impact assessment, split-brain resolution, data
  reconciliation, connectivity restoration, and post-recovery validation to
  restore full cluster consistency.
required_connections:
  - prefix: aws
    label: "AWS (or cloud provider)"
  - prefix: datadog
    label: "Datadog (or monitoring tool)"
config_fields:
  - key: affected_systems
    label: "Affected Systems"
    required: true
    placeholder: "e.g., us-east-1 cluster, payment-service"
  - key: partition_type
    label: "Partition Type"
    required: true
    placeholder: "e.g., AZ isolation, region split, service mesh failure"
features:
  - DEVOPS
  - INCIDENT_RESPONSE
---

# Network Partition Recovery Skill

Recover from **{{ partition_type }}** affecting **{{ affected_systems }}**.

## Workflow

### Phase 1 — Partition Detection and Scoping

```
PARTITION ASSESSMENT
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Partition detected — timestamp: ___
[ ] Affected systems: {{ affected_systems }}
[ ] Partition type: {{ partition_type }}
[ ] Scope:
    - Nodes/services on side A: ___
    - Nodes/services on side B: ___
    - Fully isolated nodes: ___
[ ] Impact assessment:
    - Services degraded: ___
    - Services fully unavailable: ___
    - Users affected (estimated): ___
```

### Phase 2 — Split-Brain Assessment

```
SPLIT-BRAIN CHECK
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Determine if split-brain has occurred:
    - Multiple leaders elected: [ ] YES  [ ] NO
    - Divergent writes detected: [ ] YES  [ ] NO
    - Quorum status:
      Side A: ___ nodes (quorum: [ ] YES  [ ] NO)
      Side B: ___ nodes (quorum: [ ] YES  [ ] NO)

DECISION MATRIX
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Scenario              | Action
No split-brain        | Restore connectivity, verify
Split-brain, one side | Fence minority side, restore
  has quorum          |
Split-brain, no       | Manual intervention, pick
  quorum either side  | canonical side
Divergent writes      | Data reconciliation required
```

### Phase 3 — Connectivity Restoration

```
NETWORK RECOVERY
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Root cause identified:
    [ ] Security group / firewall change
    [ ] Route table misconfiguration
    [ ] VPN/peering connection failure
    [ ] Physical network issue
    [ ] Service mesh / overlay network failure
[ ] Fix applied — timestamp: ___
[ ] Connectivity verified (ping, traceroute, TCP checks):
    - Side A -> Side B: [ ] OK
    - Side B -> Side A: [ ] OK
    - Latency restored to baseline: [ ] YES
```

### Phase 4 — Data Reconciliation

```
DATA RECONCILIATION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] Identify divergent data (if split-brain occurred):
    - Conflicting records: ___
    - Conflict resolution strategy:
      [ ] Timestamp-based (last write wins)
      [ ] Application-specific merge
      [ ] Manual review required
[ ] Reconciliation executed — timestamp: ___
[ ] Data consistency verified across all nodes
[ ] Replication caught up and healthy
```

### Phase 5 — Post-Recovery Validation

```
VALIDATION
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
[ ] All services healthy and serving traffic
[ ] Cluster membership correct (all nodes visible)
[ ] No remaining network errors in logs
[ ] Metrics returned to baseline:
    - Error rate: ___% (baseline: ___%)
    - Latency: ___ms (baseline: ___ms)
[ ] Monitoring alerts cleared
[ ] Incident timeline documented
[ ] Preventive measures identified:
    - ___
    - ___
```

## Counter-Rationalizations

| Shortcut | Counter | Why |
|----------|---------|-----|
| "We can skip some steps for this case" | Adapt the workflow steps, don't skip them | Skipped steps are where incidents and oversights originate |
| "The user seems to already know what to do" | Complete all workflow phases with the user | The workflow catches blind spots that experience alone misses |
| "This is a minor case, full process is overkill" | Scale the process down, don't turn it off | Minor cases become major when unstructured; the process scales, not disappears |
| "I'll fill in the details later" | Complete each section before moving on | Deferred details are forgotten; real-time capture is more accurate |
| "The template output isn't necessary" | Always produce the structured output format | Structured output enables comparison, audit trails, and handoff to other teams |

## Output Format

Produce a partition recovery report with:
1. **Incident summary** (partition type, scope, duration)
2. **Split-brain analysis** (whether divergence occurred, resolution)
3. **Root cause** (what caused the partition)
4. **Data reconciliation** (conflicts found and how resolved)
5. **Preventive measures** (changes to prevent recurrence)