Sles Ha Extension

---
name: sles-ha-extension
description: "SUSE Linux Enterprise High Availability Extension: Corosync cluster communication (Totem protocol, transport modes, knet), Pacemaker resource management (CIB, Policy Engine, CRMd), resource types, fencing/STONITH/SBD, quorum configuration, SAP HANA HA, HAWK console, crm shell, and cluster diagnostics. Use when: \"Pacemaker\", \"Corosync\", \"HAWK\", \"SBD\", \"STONITH\", \"HA cluster\", \"high availability\", \"crm_mon\", \"crm configure\", \"resource agent\", \"fencing\", \"cluster resource\", \"failover Linux\". Do NOT use for general SLES administration — use the `sles` skill. Do NOT use for Windows Server Failover Clustering — use the `failover-clustering` skill."
license: MIT
---

# SUSE HA Extension

This skill covers the SUSE Linux Enterprise High Availability Extension on SLES 15 SP5+. It provides deep knowledge of:

- Corosync cluster communication (Totem protocol, transport modes, knet, redundant links)
- Pacemaker cluster resource management (CIB, Policy Engine, CRMd, LRMd, DC election)
- Resource types: primitive, group, clone, promotable (multi-state/master-slave)
- Resource agents: OCF, systemd, LSB, STONITH classes
- Fencing: STONITH architecture, SBD (STONITH Block Device), IPMI, VMware, cloud agents
- Quorum: votequorum, two-node mode, wait_for_all, quorum device (qdevice)
- SAP HANA HA: SAPHana, SAPHanaTopology resource agents, performance-optimized vs cost-optimized
- HAWK web console: dashboard, resource management, constraint editor, history explorer
- crm shell: configuration, constraint management, resource operations
- Maintenance operations: cluster-wide maintenance mode, node standby, resource unmanage
- Cluster diagnostics: crm_mon, crm_report, log analysis, troubleshooting workflows

When a question involves general SLES administration, defer to the `sles` skill.

## How to Approach Tasks

Route by request type: **troubleshooting** → `references/diagnostics.md`; **architecture** → `references/architecture.md`; **best practices/config** → `references/best-practices.md`; **health check** → the diagnostic scripts.

**Identify the cluster context first** — node count, fencing method (SBD, IPMI, cloud), workload type (SAP, NFS, generic), and transport mode (knet, udpu). Reason from split-brain prevention, fencing requirements, and quorum implications. Always include fencing verification in `crm` recommendations. Validate with `crm_mon`, `corosync-quorumtool`, `sbd dump`, and `crm_verify`.

## Core Expertise

### Corosync Cluster Communication

Corosync is the messaging and membership layer beneath Pacemaker. It provides ordered message delivery, membership tracking, and quorum information via the Totem Single Ring Order protocol.

```bash
# Ring/link status
corosync-cfgtool -s

# Quorum status
corosync-quorumtool -s

# Service status
systemctl status corosync
```

Transport modes:
- `knet` (default SLES 15 SP2+): supports multiple redundant links, compression, encryption
- `udpu`: UDP unicast, requires explicit node list
- `udp`: UDP multicast, requires multicast routing

Key configuration in `/etc/corosync/corosync.conf`:
```
totem {
    version: 2
    cluster_name: mycluster
    transport: knet
    token: 5000
}
quorum {
    provider: corosync_votequorum
    two_node: 1
    wait_for_all: 1
}
```

### Pacemaker Resource Management

Pacemaker decides where resources run, responds to failures, and enforces constraints. All configuration lives in the CIB (Cluster Information Base).

```bash
# Cluster status (comprehensive)
crm_mon -1 -r -f -A

# Show configuration
crm configure show

# Verify configuration
crm_verify -L -V

# Designated Controller
crm_mon -1 | grep "Current DC"
```

### Resource Types

**Primitive**: Single instance on one node at a time.
```
primitive rsc_ip IPaddr2 \
    params ip=192.168.1.100 cidr_netmask=24 nic=eth0 \
    op monitor interval=10s timeout=20s
```

**Group**: Ordered set, always on the same node, start/stop together.
```
group grp_sap rsc_ip rsc_fs rsc_sapinstance \
    meta target-role=Started
```

**Clone**: Runs simultaneously on multiple (or all) nodes.
```
clone cl_sbd stonith-sbd \
    meta globally-unique=false target-role=Started
```

**Promotable (Multi-state)**: Clone with Master/Slave states (SAP HANA SR).
```
ms msl_SAPHana_HDB rsc_SAPHana_HDB \
    meta notify=true clone-max=2 clone-node-max=1
```

### Fencing (STONITH)

Fencing is mandatory in production. Without it, dual ownership of shared resources causes data corruption.

**SBD (recommended for on-premises)**:
```bash
# Initialize SBD device
sbd -d /dev/sdb create

# Verify SBD
sbd -d /dev/sdb dump
sbd -d /dev/sdb list

# SBD configuration: /etc/sysconfig/sbd
# SBD_DEVICE="/dev/sdb;/dev/sdc;/dev/sdd"
# SBD_PACEMAKER=yes
# SBD_WATCHDOG_DEV=/dev/watchdog
```

SBD device count: 1 (simple, SPOF), 3 (majority vote, recommended), diskless (watchdog-only, 3+ nodes required).

**IPMI**: `stonith:fence_ipmilan` for physical servers with BMC.

**VMware**: `stonith:fence_vmware_rest` for vSphere VMs.

**Cloud**: `fence_azure_arm` (Azure), `fence_aws` (AWS).

### Quorum

```
quorum {
    provider: corosync_votequorum
    two_node: 1         # Required for 2-node clusters
    wait_for_all: 1     # Wait for all nodes at startup
}
```

`no-quorum-policy` options: `stop` (default, safest), `freeze`, `ignore` (dangerous), `suicide`.

For 2-node clusters without SBD, use quorum device (`corosync-qdevice`) on a third machine.

### SAP HANA HA

**SAPHanaTopology**: Clone on all nodes, discovers SR topology (does not manage HANA).

**SAPHana**: Promotable clone managing HANA primary/secondary with automated takeover.

Key parameters:
- `PREFER_SITE_TAKEOVER=true`: prefer failover to secondary
- `AUTOMATED_REGISTER=true`: auto-register old primary as new secondary
- `DUPLICATE_PRIMARY_TIMEOUT`: seconds before treating dual-primary as error

### HAWK Web Console

Browser-based cluster management on port 7630 (HTTPS).

```bash
systemctl enable --now hawk
# Access: https://<node>:7630
# Auth: PAM (root or haclient group)
```

Features: real-time dashboard, resource management, constraint editor, history explorer, configuration wizard.

### crm Shell Configuration

```bash
# Resource management
crm configure primitive <id> <class>:<provider>:<type> params <params>
crm configure group <id> <primitives>
crm configure colocation <id> <score>: <rsc1> <rsc2>
crm configure order <id> <kind>: <rsc1> <rsc2>
crm configure location <id> <resource> <score>: <node>
crm configure property stonith-enabled=true
crm configure verify
crm configure commit

# Node operations
crm node standby node1
crm node online node1

# Resource operations
crm resource cleanup rsc_ip
crm resource start rsc_ip
crm resource stop rsc_ip
crm resource migrate rsc_ip node2
```

### Maintenance Operations

```bash
# Cluster-wide maintenance (resources unmanaged)
crm configure property maintenance-mode=true
crm configure property maintenance-mode=false

# Single node standby (resources migrate)
crm node standby node1
crm node online node1

# Single resource unmanage
crm resource unmanage rsc_ip
crm resource manage rsc_ip

# Cleanup failure counts
crm resource cleanup rsc_ip
crm resource cleanup rsc_ip node1
```

## Troubleshooting Decision Tree

```
1. Resource won't start?
   → crm_mon -1 -r -f (check Failed Actions)
   → crm resource cleanup <id>
   → Run resource agent manually with OCF_ROOT=/usr/lib/ocf

2. Unexpected failover?
   → grep "pengine\|Initiating" /var/log/pacemaker/pacemaker.log
   → Check fencing: grep stonith pacemaker.log
   → Check corosync: corosync-cfgtool -s (ring faults?)

3. Split-brain suspected?
   → corosync-quorumtool -s (does this node have quorum?)
   → Only ONE partition should show "partition with quorum"
   → Verify SBD: sbd -d /dev/sdb list

4. Fencing failure?
   → stonith_admin -I (list known devices)
   → Check SBD devices: sbd -d /dev/sdb dump
   → Check watchdog: ls -l /dev/watchdog

5. Node won't join cluster?
   → systemctl status corosync (running?)
   → corosync-cfgtool -s (ring faults?)
   → firewall-cmd --list-all | grep 5405 (firewall?)
   → md5sum /etc/corosync/authkey (matches other nodes?)
```

## Common Pitfalls

**1. Disabling STONITH for production clusters**
Setting `stonith-enabled=false` is unsupported for production. Without fencing, failed nodes may still write to shared storage, causing data corruption.

**2. Using a single SBD device without fallback**
One SBD device is a single point of failure. Use 3 SBD devices (majority vote) or configure fencing topology with IPMI fallback.

**3. Not using wait_for_all in two-node clusters**
Without `wait_for_all: 1`, a single node can claim quorum immediately after restart, potentially starting resources before the other node has stopped them.

**4. Forgetting to configure resource stickiness**
Default stickiness of 0 causes resources to move back to the original node after recovery, creating unnecessary failover events. Set `resource-stickiness=1000`.

**5. Running cluster operations without understanding the transition graph**
Use `crm_simulate` to preview what the cluster will do before making changes. Unexpected constraint interactions can cause cascading resource movements.

## Reference Files

- `references/architecture.md` -- Corosync, Pacemaker, resource types, fencing, quorum. Read for "how does X work" questions.
- `references/best-practices.md` -- Cluster design, resource configuration, maintenance, SAP HA, HAWK. Read for design and operations.
- `references/diagnostics.md` -- crm_mon, logs, crm_report, troubleshooting workflows. Read when troubleshooting.

## Diagnostic Scripts

| Script | Purpose |
|---|---|
| `scripts/01-cluster-status.sh` | Service status, quorum, DC, maintenance mode, node list |
| `scripts/02-resource-health.sh` | Resource status, failed actions, constraints, placement |
| `scripts/03-fencing-audit.sh` | STONITH config, SBD devices, watchdog, fence history |
| `scripts/04-corosync-health.sh` | Ring status, quorum votes, token config, membership |
| `scripts/05-hawk-status.sh` | HAWK service, port 7630, TLS certificate, firewall |