Vmware

---
name: vmware
description: "VMware vSphere expertise across supported versions (8.x, 9.x): ESXi hypervisor, vCenter Server, vMotion, vSphere HA, Fault Tolerance, DRS, vSAN (OSA and ESA), distributed networking (vDS, NSX), storage (VMFS, NFS, vVols), PowerCLI automation, esxcli administration, govc scripting, VADP-based backup, esxtop performance analysis, and VM lifecycle management. Use for \"vSphere\", \"ESXi\", \"vCenter\", \"VMware\", \"PowerCLI\", \"vMotion\", \"vSAN\", \"DRS\", \"VCSA\", \"VMFS\", \"esxcli\", \"esxtop\", \"govc\", \"NSX\", \"vDS\", \"vim-cmd\", \"VADP\", \"PVSCSI\", \"VMXNET3\", \"Content Library\", \"Host Profiles\", \"vLCM\". Do NOT use for NSX overlay/SDN deep-dive networking design — that's the `networking` plugin's `vmware-nsx` skill; this skill covers NSX only as it touches vSphere host/cluster config."
license: MIT
---

# VMware vSphere

This skill covers VMware vSphere across all supported versions (8.x and 9.x). It covers:

- ESXi hypervisor architecture (VMkernel, userworlds, CIM providers, DCUI)
- vCenter Server (VCSA deployment, PSC, linked mode, statistics, alarms)
- Virtual machine lifecycle (VMX, VMDK, snapshots, hardware versions, VMware Tools)
- vMotion and Storage vMotion (EVC, cross-vCenter migration)
- vSphere HA (heartbeating, admission control, FDM, VM restart priority)
- Fault Tolerance (vLockstep, Fast Checkpointing, SMP-FT)
- DRS (fully automated, affinity/anti-affinity rules, resource pools)
- vSAN (OSA disk groups, ESA flat pool, storage policies, stretched clusters)
- Networking (vSS, vDS, VMkernel ports, NetIOC, LACP, NSX integration)
- Storage (VMFS 6, NFS, iSCSI, FC, vVols, SPBM, Storage DRS)
- Backup (VADP, CBT, hot-add/NBD/SAN transport, snapshot-based backup)
- Security (lockdown mode, certificates, syslog, NTP, hardening)
- Performance (esxtop, performance charts, NUMA alignment, balloon driver)
- Automation (PowerCLI, esxcli, govc, vim-cmd, REST API, Terraform)
- Lifecycle management (vLCM, image-based management, HSM firmware)

## How to Approach Tasks

When you receive a request:

1. **Classify** the request type:
   - **Troubleshooting** -- Load `references/diagnostics.md`
   - **Optimization** -- Load `references/best-practices.md`
   - **Architecture** -- Load `references/architecture.md`
   - **Administration** -- Follow the admin guidance below
   - **Scripting** -- Apply PowerCLI, esxcli, or govc expertise directly

2. **Identify version** -- Determine which vSphere version the user is running. If unclear, ask. Feature availability varies significantly between 8.x and 9.x.

3. **Load context** -- Read the relevant reference file for deep knowledge.

4. **Analyze** -- Apply vSphere-specific reasoning, not generic virtualization advice.

5. **Recommend** -- Provide actionable guidance with specific commands (PowerCLI, esxcli, or govc).

6. **Verify** -- Suggest validation steps using esxtop, vCenter alarms, or CLI checks.

## Core Expertise

### ESXi Architecture

ESXi is a Type-1 bare-metal hypervisor. The VMkernel runs directly on hardware and provides CPU scheduling (NUMA-aware), memory management (ballooning, TPS, swap, large pages), the Pluggable Storage Architecture (PSA), and Network I/O Control (NetIOC).

Key processes (userworlds):
- `hostd` -- local VM management and VI API handler
- `vpxa` -- vCenter agent relaying commands from vCenter to hostd
- `fdm` -- Fault Domain Manager for HA heartbeating and failover
- `sfcbd` -- CIM broker for hardware health via WBEM

Management interfaces: DCUI (local console), ESXi Shell (busybox CLI), SSH, vSphere Client (HTML5). Lockdown mode forces all management through vCenter.

### vCenter Server

vCenter Server Appliance (VCSA) is the sole deployment option (Windows vCenter removed in vSphere 7+). Embedded PostgreSQL database. Deployment sizes from Tiny (10 hosts / 100 VMs) to X-Large (2,500 hosts / 35,000 VMs).

```powershell
# Connect to vCenter
Connect-VIServer -Server vcenter.corp.local -Credential (Get-Credential)

# Get vCenter version
$global:DefaultVIServer | Select-Object Name, Version, Build

# List all managed hosts
Get-VMHost | Select-Object Name, Version, Build, ConnectionState, PowerState
```

### Virtual Machines

Each VM consists of a `.vmx` configuration file, one or more `.vmdk` virtual disks, and optional snapshot delta files. Disk provisioning types: thin (grows on demand), thick lazy zeroed (allocated but not pre-zeroed), thick eager zeroed (allocated and pre-zeroed, required for FT).

```powershell
# Get VM inventory with key details
Get-VM | Select-Object Name, NumCpu, MemoryGB, PowerState, VMHost,
    @{N="HWVersion";E={$_.ExtensionData.Config.Version}},
    @{N="ToolsStatus";E={$_.ExtensionData.Guest.ToolsVersionStatus}} |
    Format-Table -AutoSize

# Find VMs with snapshots
Get-VM | Get-Snapshot | Select-Object VM, Name, Created,
    @{N="AgeDays";E={[math]::Round(((Get-Date)-$_.Created).TotalDays,1)}}, SizeGB |
    Sort-Object AgeDays -Descending
```

VMware Tools (or open-vm-tools for Linux) provides paravirtualized drivers (PVSCSI, VMXNET3), guest heartbeat, quiescing for snapshots, and time synchronization.

### vMotion and Storage vMotion

vMotion migrates running VMs between hosts with near-zero downtime. Memory is iteratively copied; final stun phase is typically <1 second. Requirements: shared storage (or combined with Storage vMotion), vMotion VMkernel ports, compatible CPUs (or EVC enabled), matching network labels.

```powershell
# vMotion a VM to another host
Move-VM -VM "web01" -Destination (Get-VMHost "esxi02.corp.local")

# Storage vMotion to different datastore
Move-VM -VM "web01" -Datastore (Get-Datastore "DS-SSD-02")

# Combined host + storage migration
Move-VM -VM "web01" -Destination (Get-VMHost "esxi02") -Datastore (Get-Datastore "DS-SSD-02")
```

EVC (Enhanced vMotion Compatibility) masks CPU features at the cluster level to allow migration between different CPU generations within the same vendor family.

### vSphere HA

HA restarts VMs on surviving hosts after host failure. Heartbeats exchanged every second; host declared failed after 12 seconds of missed heartbeats. Datastore heartbeating provides secondary verification.

Key configuration:
- **Admission Control** -- Reserve capacity for failover. Percentage-based policy recommended (e.g., 25% for a 4-host cluster = N+1).
- **VM Restart Priority** -- Highest/High/Medium/Low/Disabled. Critical infrastructure VMs get highest priority.
- **VM Monitoring** -- Optionally restarts VMs that stop sending VMware Tools heartbeats.

```powershell
# Check HA status
Get-Cluster | Select-Object Name, HAEnabled, HAAdmissionControlEnabled,
    @{N="HAFailoverLevel";E={$_.ExtensionData.Configuration.DasConfig.FailoverLevel}}
```

### DRS (Distributed Resource Scheduler)

DRS balances CPU and memory load across cluster hosts using vMotion. Modes: Fully Automated (recommended), Partially Automated, Manual.

```powershell
# Get DRS configuration
Get-Cluster | Select-Object Name, DrsEnabled, DrsAutomationLevel

# Get and apply DRS recommendations
Get-DrsRecommendation -Cluster "Production" | Apply-DrsRecommendation

# Create anti-affinity rule (separate HA pairs)
New-DrsRule -Cluster "Production" -Name "Separate-DB-Nodes" -KeepTogether $false `
    -VM (Get-VM "db-primary","db-secondary")
```

### vSAN

vSAN pools local storage across ESXi hosts into a shared datastore. Two architectures:
- **OSA (Original Storage Architecture)** -- Disk groups with dedicated cache + capacity devices
- **ESA (Express Storage Architecture, vSAN 8+)** -- Single flat pool, all-NVMe, always-on compression/dedup

Storage policies enforce data protection: FTT (Failures to Tolerate), RAID method (mirror or erasure coding), stripe width.

```powershell
# Get vSAN cluster configuration
Get-VsanClusterConfiguration -Cluster "Production"

# Check vSAN health
Get-VsanHealthSummary -Cluster "Production"
```

### Networking

**vSphere Standard Switch (vSS)** -- Per-host Layer 2 switch. Basic NIC teaming and VLAN tagging.

**vSphere Distributed Switch (vDS)** -- Cluster-level switch managed through vCenter. Adds NetIOC, LACP, port mirroring, health check, and per-port policies.

VMkernel ports handle host traffic: Management, vMotion, vSAN, FT Logging, NFS/iSCSI, Replication.

VLAN modes: VST (switch tags at port group), EST (physical switch tags), VGT (trunk to guest, VLAN 4095).

### Storage

| Type | Protocol | Key Features |
|---|---|---|
| VMFS 6 | Block (FC/iSCSI/local) | Cluster filesystem, UNMAP, ATS locking |
| NFS | NFS v3/v4.1 | Simple setup, v4.1 adds Kerberos and trunking |
| iSCSI | Block over IP | Software initiator, port binding for multipath |
| Fibre Channel | FC/FCoE | Lowest latency, enterprise SAN |
| vVols | VASA provider | Per-VM array objects, policy-driven |
| vSAN | Distributed | Hyper-converged, SPBM policies |

```bash
# List datastores from ESXi
esxcli storage filesystem list

# Check multipath policy
esxcli storage nmp device list

# Set Round Robin policy
esxcli storage nmp device set --device=naa.xxxx --psp=VMW_PSP_RR
```

### PowerCLI Essentials

```powershell
# Install PowerCLI
Install-Module -Name VMware.PowerCLI -Scope CurrentUser -Force

# Host inventory report
Get-VMHost | Select-Object Name, ConnectionState, NumCpu, MemoryTotalGB,
    @{N="VMs";E={($_ | Get-VM).Count}}, Version, Build | Format-Table

# Datastore space report
Get-Datastore | Select-Object Name, Type, CapacityGB, FreeSpaceGB,
    @{N="UsedPct";E={[math]::Round((1-($_.FreeSpaceGB/$_.CapacityGB))*100,1)}} |
    Sort-Object UsedPct -Descending

# Export VM inventory to CSV
Get-VM | Select-Object Name, NumCpu, MemoryGB, PowerState, VMHost, Folder, GuestId |
    Export-Csv -Path "vm-inventory.csv" -NoTypeInformation
```

### esxcli Essentials

```bash
# System info
esxcli system version get
esxcli system hostname get

# Network overview
esxcli network nic list
esxcli network ip interface list
esxcli network vswitch standard list

# Storage overview
esxcli storage core adapter list
esxcli storage core device list

# Kill a stuck VM
esxcli vm process list
esxcli vm process kill --type=soft --world-id=<worldID>

# Maintenance mode
esxcli system maintenanceMode set --enable=true
```

### govc Essentials

```bash
# Environment setup
export GOVC_URL=https://vcenter.corp.local
export GOVC_USERNAME=administrator@vsphere.local
export GOVC_PASSWORD='password'
export GOVC_INSECURE=true

# Inventory
govc ls /DC01/vm
govc find / -type m -name "web*"
govc vm.info /DC01/vm/web01

# Power operations
govc vm.power -on /DC01/vm/web01
govc vm.power -off /DC01/vm/web01

# Snapshots
govc snapshot.create -vm /DC01/vm/web01 "pre-patch"
govc snapshot.tree -vm /DC01/vm/web01
```

## Common Pitfalls

**1. Letting snapshots accumulate**
Snapshots are not backups. Each snapshot adds a delta VMDK; deep chains degrade I/O and complicate consolidation. Monitor with vCenter alarms and delete within 24-72 hours.

**2. Over-allocating vCPUs**
More vCPUs than needed increases CPU scheduler overhead (co-stop, ready time). Start with 2-4 vCPUs and scale based on `esxtop` %RDY data. Keep VMs within a single NUMA node.

**3. Running without EVC when CPU generations differ**
Without EVC, vMotion fails between hosts with different CPU features. Enable EVC at the cluster level before deploying VMs.

**4. Disabling SELinux/firewall in ESXi**
ESXi has its own firewall. Do not disable it. Use `esxcli network firewall ruleset` to manage rules. Keep lockdown mode enabled in production.

**5. Using thick eager zeroed disks unnecessarily**
Thick eager zeroed is required only for FT. Use thin provisioning for general workloads to conserve datastore space.

**6. Not configuring syslog forwarding**
ESXi logs are volatile across reboots on diskless or USB-boot hosts. Forward to a central syslog server immediately after deployment.

**7. Ignoring vSAN health warnings**
vSAN Health checks detect degraded objects, network issues, and disk failures. Review weekly; resolve all red/yellow warnings before they escalate.

**8. Skipping VMware Tools upgrades before hardware version upgrades**
Always upgrade VMware Tools first, then upgrade VM hardware version. Reversing the order can cause driver compatibility issues.

**9. Using vSS instead of vDS in Enterprise Plus environments**
vDS provides NetIOC, LACP, port mirroring, and centralized management. There is no reason to use vSS when vDS licensing is available.

**10. Not testing backup restores**
VADP backups with CBT can have silent integrity issues. Regularly test full VM restores in an isolated environment.

## Version-specific guidance

| Version | Reference | What's version-specific |
|---|---|---|
| vSphere 8.x | `references/versions/8.md` | vSAN ESA, DPU/SmartNIC support, vLCM image-based management, vSphere+, Configuration Profiles, VM hardware versions 20-21 |
| vSphere 9.x | `references/versions/9.md` | Broadcom licensing changes (subscription-only, per-core), AI/ML workload support (GPU partitioning, vGPU live migration), Confidential Computing (SEV-SNP/TDX), REST API v9, GraphQL API |

## Reference Files

Load these when you need deep knowledge for a specific area:

- `references/architecture.md` -- ESXi internals, VM architecture, vMotion, HA/FT/DRS, networking, storage, vSAN. Read for "how does X work" questions.
- `references/best-practices.md` -- VM sizing, NUMA alignment, DRS tuning, HA configuration, snapshot management, hardening, backup strategy. Read for design and operations questions.
- `references/diagnostics.md` -- esxtop panels and thresholds, vm-support bundles, PSOD analysis, vMotion failure troubleshooting, storage latency diagnosis, HA failover debugging. Read when troubleshooting performance or errors.

## Script Library

- `scripts/01-esxcli-health.sh` -- ESXi host health check via esxcli (system, network, storage)
- `scripts/02-powercli-inventory.ps1` -- PowerCLI VM/host/datastore inventory report
- `scripts/03-powercli-health.ps1` -- PowerCLI cluster health (HA, DRS, alarms, snapshots)