Failover Clustering

---
name: failover-clustering
description: "Windows Server Failover Clustering (WSFC) across Windows Server 2016-2025: cluster architecture, quorum models, Cluster Shared Volumes, Cluster-Aware Updating, cluster networking, resource groups, and high availability patterns. Use when: \"failover cluster\", \"WSFC\", \"quorum\", \"CSV\", \"cluster shared volume\", \"CAU\", \"cluster-aware updating\", \"cluster validation\", \"Windows cluster\", \"cluster node\", \"cluster resource\", \"stretch cluster\", \"cluster sets\", \"split-brain\", \"node majority\", \"cloud witness\", \"disk witness\", \"file share witness\". Do NOT use for Linux HA clustering (Pacemaker/Corosync) — use the `sles-ha-extension` skill. Do NOT use for the Hyper-V hypervisor layer itself — use the `hyper-v` skill. Do NOT use for Storage Spaces Direct disk pool/volume design — that's the `storage-spaces-direct` skill (in the `storage` plugin)."
license: MIT
---

# Windows Server Failover Clustering (WSFC)

This skill covers Windows Server Failover Clustering (WSFC) across Windows Server 2016, 2019, 2022, and 2025. It provides deep knowledge of:

- Cluster service internals (clussvc, RHS, CLUSDB, GUM, NetFT)
- Quorum models (node majority, disk witness, file share witness, cloud witness, dynamic quorum)
- Resource groups, dependency chains, failover policies, and restart behaviors
- Cluster Shared Volumes (CSV) architecture, I/O modes, and coordinator management
- Cluster-Aware Updating (CAU) for zero-downtime patching
- Cluster networking (heartbeat, cross-subnet, multi-site, live migration networks)
- High availability patterns (active/passive, active/active, N+1, stretch clusters, cluster sets)
- Storage Spaces Direct (S2D) hyper-converged deployments
- Integration with SQL Server Always On Availability Groups and Hyper-V

When a question is version-specific, note the relevant version differences. When the version is unknown, provide general guidance and flag where behavior varies. For general Windows Server administration, see the `windows-server` skill.

## How to Approach Tasks

Route by request type: **troubleshooting** → `references/diagnostics.md`; **architecture** → `references/architecture.md`; **best practices/optimization** → `references/best-practices.md`; **validation/health check** → the diagnostic scripts; **administration** → apply cluster management expertise directly.

**Identify the Windows Server version first** — feature availability shifts by version (cloud witness requires 2016+, cluster sets require 2019+). Reason from quorum math, resource dependencies, network topology, and storage architecture — not generic HA advice. Validate with `Test-Cluster`, `Get-ClusterLog`, event log queries, and quorum math checks.

## Core Expertise

### Cluster Service Architecture

The Cluster Service (clussvc.exe) coordinates all cluster operations: node membership, quorum management, CLUSDB replication, resource group failover, and health monitoring. It communicates with the Resource Host Subsystem (rhs.exe) which hosts resource DLLs in isolated processes. The cluster network driver (netft.sys) handles heartbeat communication over UDP port 3343.

CLUSDB is the cluster configuration database stored as a registry hive at `C:\Windows\Cluster\CLUSDB`. The Global Update Manager (GUM) protocol ensures CLUSDB consistency through two-phase commit across all nodes. All configuration writes route through the GUM coordinator.

Key diagnostic entry points:
- Cluster log: `Get-ClusterLog -Destination "C:\Logs" -TimeSpan 30`
- Event channel: `Microsoft-Windows-FailoverClustering/Operational`
- Registry: `HKLM:\Cluster\` when hive is mounted

### Quorum Models

Quorum prevents split-brain by requiring a majority of votes before the cluster can operate. Each node contributes 1 vote; a witness adds 1 additional vote.

| Model | Witness Type | Best For |
|---|---|---|
| Node Majority | None | Odd-node clusters (3, 5, 7 nodes) |
| Node and Disk Majority | Shared disk (1 GB min) | Even-node clusters with shared storage |
| Node and File Share Majority | UNC file share | 2-node or multi-site without shared storage |
| Cloud Witness (2016+) | Azure Blob Storage | Multi-site clusters, any topology |
| USB Witness (2019+) | USB drive | Physical edge deployments |

**Dynamic quorum** (enabled by default) adjusts node vote counts as nodes leave cleanly, allowing smaller surviving sets to maintain quorum. **Dynamic witness** adds or removes the witness vote based on whether the cluster has an even or odd number of voting nodes.

Quorum math: For N total votes, quorum requires floor(N/2) + 1 votes present.

### Resource Groups and Failover

Resource groups (called "Roles" in the GUI) are the unit of failover. A group contains related resources with dependency chains that move together between nodes.

Key failover properties per group:
- **FailoverThreshold** -- Maximum failures within the period before the group stays failed (default: 3)
- **FailoverPeriod** -- Time window in hours for counting failures (default: 6)
- **AutoFailbackType** -- Whether to return to the preferred owner (Prevent / Allow / Allow with Window)
- **Preferred Owners** -- Ordered list of nodes for group placement
- **Possible Owners** -- Nodes permitted to host each resource

Resource health checks run at two levels:
- **LooksAlive** -- Lightweight poll every 5 seconds (is the process running?)
- **IsAlive** -- Deep health check every 60 seconds (can we connect to the service?)

### Cluster Shared Volumes (CSV)

CSVs allow multiple nodes to simultaneously access the same NTFS or ReFS volume. Critical for Hyper-V live migration and Scale-Out File Server deployments.

CSV I/O modes:
- **Direct I/O** -- Normal mode. Each node reads/writes directly to storage; metadata routes to the coordinator node. Best performance.
- **Redirected I/O** -- All I/O routes through the coordinator over the cluster network (SMB3). Triggered by storage errors, BitLocker operations, VSS backups, or maintenance mode. Significantly slower.

**CSV namespace**: All nodes access volumes at `C:\ClusterStorage\Volume1`, `C:\ClusterStorage\Volume2`, etc. The coordinator node manages NTFS metadata, locks, and VSS coordination.

**CSV cache**: A block-level read cache on each node in memory (separate from Windows file cache). Configured per-cluster: `(Get-Cluster).BlockCacheSize = 1024` (value in MB). Reduces IOPS to backend storage for frequently-read data.

**CSV vs non-CSV disks**: Non-CSV disk resources are owned by one node at a time and mounted only on that node (used for SQL Server FCI data drives). CSV volumes are accessible by all nodes simultaneously (used for Hyper-V VMs, SOFS shares).

Monitor CSV state: `Get-ClusterSharedVolume | Select-Object Name, State, OwnerNode`

### Cluster-Aware Updating (CAU)

CAU automates Windows Update application across cluster nodes while maintaining availability. It orchestrates: drain node, apply updates, restart, resume node, proceed to next node.

**Self-updating mode**: CAU runs as a cluster role on a schedule (no external coordinator).
**Remote-updating mode**: CAU invoked from a remote machine for one-off runs.

Plug-ins: `Microsoft.WindowsUpdatePlugin` (default, uses WSUS/WU) and `Microsoft.HotfixPlugin` (applies CAB/MSP from a file share).

Key commands:
```powershell
Add-CauClusterRole -ClusterName "Cluster1" -DaysOfWeek Saturday -StartTime "02:00" -Force
Invoke-CauScan -ClusterName "Cluster1" -CauPluginName Microsoft.WindowsUpdatePlugin
Get-CauReport -ClusterName "Cluster1" -Last | Format-List
```

### Cluster Networking

Dedicate separate NICs per traffic type:
- **Cluster heartbeat** (Role=1): UDP 3343, dedicated NIC, no default gateway, no DNS registration
- **Client access** (Role=2): Client-facing traffic only
- **Both** (Role=3): Acceptable for small clusters but not recommended for production

For Hyper-V clusters, live migration traffic should use a separate high-bandwidth NIC (10 GbE+) with SMB Direct (RDMA) for best performance. Configure in Hyper-V settings, not cluster settings.

Switch Embedded Teaming (SET) is required for S2D environments and integrates NIC teaming directly into the Hyper-V virtual switch. SET supports RDMA and SR-IOV, unlike legacy LBFO teaming.

Heartbeat tuning parameters:
- `SameSubnetDelay` / `SameSubnetThreshold` -- Intra-subnet (default: 1000ms / 5 missed = ~5s detection)
- `CrossSubnetDelay` / `CrossSubnetThreshold` -- Cross-subnet (tune up for WAN latency in stretch clusters)

Network quarantine: When a node loses connectivity briefly, it enters quarantine for `QuarantineDuration` seconds (default: 7200) after `QuarantineThreshold` failures (default: 3).

### SQL Server Always On AG Integration

SQL Server Always On Availability Groups rely on WSFC for health monitoring and automatic failover. Key cluster-level considerations:

- Each AG is a cluster resource of type "SQL Server Availability Group"
- The AG listener is a cluster Network Name + IP Address resource with OR dependencies for multi-subnet
- Automatic failover requires: synchronous-commit replicas, healthy quorum, and the AG resource online
- Set `RegisterAllProvidersIP = 1` on the listener to register all subnet IPs in DNS
- Set `HostRecordTTL = 300` (or lower) for faster multi-subnet client failover
- Configure the AG group's `FailoverThreshold` conservatively to avoid failover storms during transient issues

```powershell
# View AG cluster resources
Get-ClusterResource | Where-Object ResourceType -eq 'SQL Server Availability Group'

# Check AG resource health
Get-ClusterResource "AG-Name" | Select-Object Name, State, OwnerNode, OwnerGroup
```

### High Availability Patterns

| Pattern | Description | Capacity Planning |
|---|---|---|
| Active/Passive | One node active, others standby | Simple; standby node idle |
| Active/Active | Multiple roles on different nodes | N+1: surviving nodes must absorb failed node's load |
| Stretch Cluster | Nodes across sites with Storage Replica | Quorum witness in third site or cloud; tune cross-subnet heartbeat |
| Cluster Sets (2019+) | Multiple clusters under one namespace | Cross-cluster live migration; SOFS referrals |

For stretch clusters, configure site-aware fault domains:
```powershell
New-ClusterFaultDomain -Name "SiteA" -Type Site
Get-ClusterNode -Name "Node1","Node2" | Set-ClusterFaultDomain -Parent "SiteA"
```

### Failover Policies

Per-group failover configuration controls how aggressively the cluster responds to failures:

- **FailoverThreshold**: Maximum failures within `FailoverPeriod` before the group stays failed (default: 3 in 6 hours)
- **AutoFailbackType**: Prevent (stay on failover node), Allow (return immediately), Allow with Window (return during scheduled hours)
- **Preferred Owners**: Ordered list -- first owner is most preferred for placement and failback

```powershell
# View failover policy for a specific group
Get-ClusterGroup "SQL Server Role" | Select-Object FailoverThreshold, FailoverPeriod, AutoFailbackType

# Set conservative failover policy
(Get-ClusterGroup "SQL Server Role").FailoverThreshold = 3
(Get-ClusterGroup "SQL Server Role").FailoverPeriod = 6
```

### Anti-Affinity and Workload Separation

Prevent critical workloads from co-locating on the same node:
```powershell
(Get-ClusterGroup "VM1").AntiAffinityClassNames = "CriticalVMs"
(Get-ClusterGroup "VM2").AntiAffinityClassNames = "CriticalVMs"
```

Control placement with preferred and possible owners:
```powershell
Set-ClusterOwnerNode -Group "SQL Server Role" -Owners "Node1","Node2","Node3"
Set-ClusterOwnerNode -Resource "SQL Server" -Owners "Node1","Node2"
```

## Version-Specific Changes

| Feature | 2016 | 2019 | 2022 | 2025 |
|---|---|---|---|---|
| Rolling Cluster OS Upgrade | Introduced | -- | -- | -- |
| Cloud Witness (Azure Blob) | Introduced | -- | -- | -- |
| Site-Aware Fault Domains | Introduced | -- | -- | -- |
| VM Resiliency (storage loss) | Introduced | -- | -- | -- |
| Storage Spaces Direct | Introduced (Datacenter) | Improved resync | Improved stretch | NVMe-oF, better tiering |
| Workgroup / Multi-Domain Clusters | Introduced | Improved cert auth | -- | -- |
| Cluster Sets | -- | Introduced | -- | -- |
| USB Witness | -- | Introduced | -- | -- |
| Kubernetes Integration | -- | Foundation | -- | -- |
| CAU Improvements | -- | -- | Simplified profiles, WUfB integration | -- |
| Azure Arc Integration | -- | -- | Introduced | Improved health service |
| SMB over QUIC | -- | -- | Introduced | -- |
| Network ATC | -- | -- | Introduced | -- |
| Adjustable Quorum Thresholds | -- | -- | Improved | -- |
| NVMe over Fabrics (NVMe-oF) | -- | -- | -- | Introduced |
| Delegated Managed Service Account | -- | -- | -- | Introduced |
| Cluster REST API | -- | -- | -- | Introduced |
| Max Nodes per Cluster | 64 | 64 | 64 | 64 |

### Windows Server 2016 Highlights

- **Rolling Cluster OS Upgrade**: Upgrade nodes one at a time from 2012 R2 to 2016 without downtime. Cluster operates in mixed-OS mode. Finalize with `Update-ClusterFunctionalLevel`.
- **Cloud Witness**: Azure Blob Storage as quorum witness. No Azure VM required -- just blob storage API access. Ideal for multi-site clusters.
- **Site-Aware Clusters**: Fault domain awareness (site, rack, node). Cluster prefers partitions spanning more fault domains.
- **VM Resiliency**: When storage is temporarily unavailable, VMs continue running in "Running -- Critical" state instead of immediately failing over.
- **Storage Spaces Direct**: Hyper-converged storage on local disks. Datacenter edition only.

### Windows Server 2019 Highlights

- **Cluster Sets**: Group multiple clusters under a single management namespace. Cross-cluster live migration for VMs.
- **USB Witness**: USB drive as quorum witness for physical edge deployments.
- **Cross-Domain Improvements**: Better certificate-based authentication for workgroup and multi-domain clusters.
- **Cluster Hardening**: Improved Kerberos integration for cluster network names.
- **Faster VM Failover**: Improved checkpoint and state restoration speeds.

### Windows Server 2022 Highlights

- **CAU Improvements**: Simplified run profiles, better Windows Update for Business integration, improved reporting.
- **Azure Arc Integration**: Register clusters with Azure Arc for cloud management, Azure Monitor, and Azure Policy.
- **Improved Stretched Clusters**: Active-Active stretch support, Storage Replica compression, reduced failover times.
- **SMB over QUIC**: File share access over HTTPS/QUIC without VPN.
- **Network ATC**: Intent-based NIC configuration for cluster, storage, and management traffic.

### Windows Server 2025 Highlights

- **NVMe over Fabrics**: S2D with NVMe-oF targets for lower-latency storage fabric.
- **Delegated Managed Service Account (dMSA)**: Improved cluster service account management, reduced Kerberos overhead.
- **Improved Health Service**: More granular fault reporting, better Windows Admin Center integration.
- **Enhanced S2D Performance**: Mirror-accelerated parity improvements, better NVMe + SSD + HDD tiering.
- **Cluster REST API**: Programmatic cluster management complementing PowerShell and WMI.

## Storage Spaces Direct (S2D)

S2D is a hyper-converged storage solution built into Windows Server (Datacenter edition only). It uses local NVMe, SSD, and HDD disks on each cluster node to create shared storage without external SAN infrastructure.

Key characteristics:
- Minimum 2 nodes, maximum 16 nodes per S2D cluster
- Cache tier: NVMe or SSD (absorbs writes, caches reads)
- Capacity tier: SSD or HDD
- Resilience: 2-way mirror (2+ nodes), 3-way mirror (3+ nodes), dual parity / erasure coding (4+ nodes)
- Network: RDMA required for production (25 GbE or 100 GbE)
- Fault domains: Site, Rack, Chassis, Node granularity for intelligent data placement

```powershell
Enable-ClusterStorageSpacesDirect
Get-StoragePool | Where-Object IsPrimordial -eq $false | Select-Object FriendlyName, HealthStatus, Size
Get-VirtualDisk | Select-Object FriendlyName, HealthStatus, ResiliencySettingName, Size
```

**S2D Health Service** (2016+) provides proactive health monitoring with faults, actions, and performance history. Query with `Get-HealthFault` and `Get-ClusterPerformanceHistory`.

## Node Drain and Maintenance

Always drain a node before maintenance to gracefully relocate all roles:

```powershell
# Drain (moves roles according to preferred owners)
Suspend-ClusterNode -Name "Node2" -Drain

# Check drain progress
Get-ClusterNode | Select-Object Name, State, DrainStatus

# Resume after maintenance
Resume-ClusterNode -Name "Node2" -Failback Immediate
```

For quick reboots where the node will return before failover thresholds trigger, use `Suspend-ClusterNode` without the `-Drain` flag.

## Common Pitfalls

**1. No witness configured on an even-node cluster**
A 2-node or 4-node cluster without a witness has zero tolerance for symmetric failures. Always configure a witness -- cloud witness is the recommended default.

**2. Heartbeat thresholds too tight for WAN links**
Default `CrossSubnetThreshold` of 5 missed heartbeats at 1000ms intervals means ~5 seconds to declare a cross-subnet node dead. High-latency WAN links may trigger false positives. Increase to 10-20 for stretch clusters.

**3. CSV in redirected I/O for extended periods**
Redirected I/O routes all storage traffic through the coordinator over SMB, dramatically reducing performance. Investigate and resolve the underlying cause (storage connectivity, BitLocker, pending VSS operation) promptly.

**4. FailoverThreshold exhaustion**
If a role fails and recovers repeatedly within the FailoverPeriod, it can exhaust the FailoverThreshold and stay in a Failed state even though the underlying issue is intermittent. Investigate root cause rather than simply increasing the threshold.

**5. MPIO not configured or misconfigured**
Shared storage without MPIO is a single point of failure. Install `Multipath-IO` on all nodes and configure a load-balance policy (Round Robin or Least Queue Depth) before adding disks to the cluster.

## Reference Files

Load these when you need deep knowledge for a specific area:

- `references/architecture.md` -- Cluster service internals, CLUSDB, heartbeat mechanism, CSV architecture, quorum models, resource model, cluster networks. Read for "how does X work" questions.
- `references/diagnostics.md` -- Cluster validation, logging, critical event IDs, troubleshooting workflows for resource failures, unexpected failovers, split-brain, CSV errors, and node isolation. Read when troubleshooting.
- `references/best-practices.md` -- Cluster design, hardware requirements, network design, storage design, CAU configuration, HA patterns, anti-affinity, node drain procedures, stretch clusters. Read for design and operations questions.

## Diagnostic Scripts

Run these for rapid cluster assessment:

| Script | Purpose |
|---|---|
| `scripts/01-cluster-health.ps1` | Overall cluster health: nodes, groups, resources, recent events |
| `scripts/02-cluster-network.ps1` | Network configuration, heartbeat settings, interface status |
| `scripts/03-csv-health.ps1` | CSV volume status, redirected I/O, free space analysis |
| `scripts/04-resource-groups.ps1` | Resource group config, dependency chains, failover policies |
| `scripts/05-quorum-witness.ps1` | Quorum model, witness health, node votes, quorum math |
| `scripts/06-cau-status.ps1` | CAU role config, run history, per-node update compliance |
| `scripts/07-cluster-validation.ps1` | Test-Cluster wrapper with result parsing |
| `scripts/08-ag-cluster-health.ps1` | SQL Server AG cluster resources, replica sync, failover readiness |

## Key Paths and Ports

| Path | Description |
|---|---|
| `C:\Windows\Cluster\CLUSDB` | Cluster database binary hive |
| `C:\Windows\Cluster\Reports\` | Cluster log output directory |
| `C:\ClusterStorage\` | CSV namespace root |
| `HKLM:\Cluster\` | Cluster registry hive (when mounted) |

| Port | Protocol | Purpose |
|---|---|---|
| 3343 | UDP/TCP | Cluster heartbeat and communication |
| 445 | TCP | SMB (CSV redirected I/O, file share witness) |
| 135 | TCP | RPC endpoint mapper |
| 49152-65535 | TCP | RPC dynamic range |