Virtualization

---
name: virtualization
description: >
  Manage VMs: Proxmox, QEMU/KVM, libvirt, XCP-ng, VMware/ESXi; debug hypervisors, storage, and GPU passthrough.
license: MIT
compatibility: "Varies by hypervisor. Proxmox: pvesh, qm, pct. Libvirt: virsh, virt-install. Optional: packer, terraform, fio, cloud-localds"
metadata:
  source: iuliandita/skills
  date_added: "2026-04-02"
  effort: high
  argument_hint: "[hypervisor-or-task]"
---

# Virtualization: Hypervisors, VMs, and Infrastructure

Create, configure, and manage virtual machines across hypervisors - from single-node Proxmox
setups to multi-node clusters with HA, live migration, and GPU passthrough. The goal is
production-ready VM infrastructure with correct storage, memory, and CPU config that won't
bite you at 3 AM.

**Target versions** (October 2026):

| Tool | Version | Release date | Notes |
|------|---------|-------------|-------|
| Proxmox VE | 9.2 | May 2026 | Current production lane; review upgrade notes from 9.1 |
| Proxmox Backup Server | 4.2 | Apr 2026 | Dedup, incremental, prune policies |
| bpg/proxmox (Terraform) | 0.115.0 | Oct 2026 | Primary Proxmox IaC provider |
| QEMU | 11.1.2 | Sep 2026 | Stable 11.1 maintenance release |
| libvirt | 12.8.0 | Oct 2026 | Hypervisor abstraction layer |
| XCP-ng | 8.3 LTS | Oct 2024 | Xen-based, LTS since Jun 2025, EOL Nov 2028 |
| VMware ESXi | 8.0 U3k (25595708) | Jul 2026 | Security floor for the 8.0 U3 lane (build number unverified October 2026); check the appliance lane |
| VirtualBox | 7.2.20 | Sep 2026 | Current 7.2 maintenance release |
| Packer | 1.16.1 | Sep 2026 | Image builder, multi-platform |
| cloud-init | 26.2 | Jul 2026 | Instance initialization standard |

Security recheck (2026-09-10): [VMSA-2026-0006](https://brcm.tech/vmsa-2026-0006)
addresses critical CVE-2026-47876 (ESX VMXNET3 VM escape, CVSS 9.3). The 8.0 fixed builds are
ESXi80U3k-25595708 or ESXi80U2f-25626445 (build numbers unverified October 2026: the Broadcom advisory is not fetchable); select the matching update lane.
The advisory also covers critical vCenter CVE-2026-59309/CVE-2026-59310;
patch vCenter separately using its response matrix. ESX/vCenter 7.0 require
Broadcom extended-support guidance rather than assuming an 8.0 patch applies.

## When to use

- Creating or configuring VMs on Proxmox, libvirt/KVM, XCP-ng, or VMware
- Provisioning Proxmox VMs with Terraform (bpg/proxmox provider)
- Building VM templates with Packer and cloud-init
- Configuring PCI/GPU passthrough for compute or display GPUs
- Managing storage backends (LVM-thin, ZFS, Ceph, NFS)
- Setting up Proxmox clustering, HA, and live migration
- Troubleshooting VM performance (disk I/O, memory, CPU)
- Planning backup strategies (Proxmox Backup Server, snapshots)
- Tuning disk performance (virtio-scsi, iothread, discard/fstrim)
- Memory management (ballooning, NUMA topology, hugepages)

## When NOT to use

- Kubernetes manifests, Helm charts, container orchestration (use **kubernetes**)
- General Terraform/OpenTofu HCL patterns, state, modules (use **terraform**)
- Network config not hypervisor-specific: DNS, VPNs, reverse proxies (use **networking**)
- Ansible playbooks and configuration management (use **ansible**)
- Docker/container image optimization (use **docker**)
- OPNsense/pfSense firewall management (use **opnsense-pfsense**)
- Synology DSM administration (use **synology-dsm**)
- NixOS configuration and image generation (use **nixos**)

---

## AI Self-Check

AI tools consistently produce the same VM configuration mistakes. **Before returning any
generated VM config, Terraform HCL, or Packer template, verify against this list:**

- [ ] No hardcoded IPs, passwords, or SSH keys - use variables or cloud-init injection
- [ ] Disk interface is virtio (scsi0 with virtio-scsi controller), not IDE, unless legacy OS
- [ ] `iothread = true` on SCSI disks only with the `virtio-scsi-single` controller
- [ ] `ssd = true` emulation enabled when backing store is SSD (enables guest TRIM)
- [ ] `discard = on` on QEMU disk config for thin-provisioned storage (fstrim passthrough)
- [ ] Memory ballooning disabled unless the guest loads the `virtio_balloon` driver (minimal
  Alpine builds, some BSDs); raising `memory:` past the running maximum is DIMM hotplug, not
  ballooning, and needs `hotplug: memory` plus NUMA or a full stop/start
- [ ] In bpg/proxmox Terraform, disable ballooning in the `memory` block with
  `floating = 0`; keep native Proxmox `balloon: 0` syntax separate
- [ ] CPU type is `host` for production (full feature passthrough), not `kvm64`/`qemu64`
- [ ] NUMA enabled for multi-socket or large-memory VMs
- [ ] QEMU guest agent enabled (cloud-init installs it, but verify)
- [ ] Cloud-init interface specified (bpg/proxmox defaults to ide2 when null)
- [ ] Terraform lifecycle: `prevent_destroy` on VMs, `ignore_changes` on `node_name`; ignore `disk` only when disks are grown on the host
- [ ] One owner of disk size: grow via `disk.size` in bpg/proxmox (live when `disk` is in `hotplug`), or via `qm resize` with `disk` ignored, never both
- [ ] PCI passthrough: `pcie = false` for standard passthrough, `xvga = false` unless display GPU
- [ ] PCI passthrough: machine type is `q35` when `pcie = true` is needed
- [ ] GPU passthrough: AMD GPUs are prone to reset bugs (vendor-reset kernel module or `pcie_port_pm=off` may be required); NVIDIA generally resets cleanly but verify with your card model before production use
- [ ] BIOS type matches use case: `seabios` default, `ovmf` for UEFI/Secure Boot/Windows 11
- [ ] Backup retention configured (not unlimited snapshots eating storage)
- [ ] Network device uses `virtio` model, not `e1000` or `rtl8139`
- [ ] `fstrim.timer` enabled in guest for thin-provisioned storage (completes the discard chain)
- [ ] SCSI controller explicitly set (`virtio-scsi-single` when using per-disk I/O threads;
  `virtio-scsi-pci` only when they are disabled)
- [ ] Machine type matches BIOS: `i440fx` with `seabios`, `q35` with `ovmf` (UEFI). Mixing
  `i440fx` + `ovmf` causes boot failures. `q35` + `seabios` works but wastes q35 features.
- [ ] VGA type matches use case: `serial0` for headless cloud images, `virtio` for GUI VMs,
  omit for PCI passthrough display GPUs (`x-vga=1` replaces the virtual display)
- [ ] **Hypervisor/version checked**: Proxmox, QEMU/KVM, libvirt, XCP-ng, vSphere, and cloud-init advice matches the target platform
- [ ] **Storage risk gated**: disk format, snapshot, passthrough, and migration commands preserve data and rollback
- [ ] Cross-cutting agent hygiene applied - see `references/agent-hygiene.md`

---

## Best Practices

- Snapshot before guest-agent, disk, boot, passthrough, or hypervisor upgrades, but do not treat snapshots as backups.
- Keep host, guest, and storage backups independently restorable.
- Document PCI/GPU passthrough bindings so kernel updates do not strand the host.


## Restore drills

Restore a representative guest to an isolated network without duplicate production IPs or identities. Verify boot, application data, dependent services, and encryption-key availability; measure recovery time and data loss against RTO/RPO. Record the backup selected, checks, and cleanup. A successful backup job or snapshot is not evidence of recoverability.

## Workflow

For changes to an existing VM or host, copy this checklist and track progress:

```markdown
- [ ] Step 1: Task identified and matching reference read (plus `references/gotchas.md`)
- [ ] Step 2: Hypervisor, guest OS, storage, and backup state recorded
- [ ] Step 3: Change applied on a non-critical VM first
- [ ] Step 4: Validation passed (fix and return to Step 3 on any failure)
```

### Step 1: Identify the task

| Task | Start with | Reference |
|------|------------|-----------|
| **Proxmox VM creation** | CLI (qm) or API (pvesh), cloud-init | `references/proxmox.md` |
| **Terraform provisioning** | bpg/proxmox provider, lifecycle rules | `references/proxmox.md` (Terraform section) |
| **Image building** | Packer + cloud-init templates | `references/image-building.md` |
| **libvirt/KVM management** | virsh, XML domain definitions | `references/libvirt-qemu-kvm.md` (section 8: quick start) |
| **GPU/PCI passthrough** | IOMMU groups, vfio-pci | `references/proxmox.md` (section 7, incl. Windows 11 + NVIDIA quick path) |
| **Performance tuning** | Disk, memory, CPU config | This file + references |
| **Migration to Proxmox** | From VMware, XCP-ng, or bare metal | `references/proxmox.md` |

### Step 2: Gather requirements

Before creating or modifying VMs:

- **Hypervisor and version** - Proxmox VE 9.x? libvirt? VMware migration?
- **Guest OS** - Linux distro, Windows, BSD? (affects virtio drivers, ballooning, agent)
- **CPU** - core count, type (host vs emulated), pinning needs, NUMA topology
- **Memory** - dedicated amount, ballooning (usually: don't), hugepages for databases
- **Storage** - backend (LVM-thin, ZFS, Ceph, NFS), disk size, format (raw vs qcow2)
- **Network** - bridge, VLAN tag, virtio, firewall
- **Passthrough** - GPU/PCI devices, USB, serial ports
- **Provisioning method** - manual, Terraform, Packer template + cloud-init
- **HA requirements** - clustered? live migration? fencing?
- **Backup strategy** - PBS, snapshots, vzdump, frequency

### Step 3: Build

Follow the domain-specific reference file. Key principles:

- **Use cloud-init for provisioning.** Don't manually configure VMs after creation. Inject SSH
  keys, network config, packages, and user accounts via cloud-init.
- **Use virtio everywhere.** Disk (virtio-scsi), network (virtio-net), display (virtio-gpu for
  headless). IDE and e1000 exist for legacy OS compatibility only.
- **Pin CPU type to `host`.** Emulated CPU types (kvm64, qemu64) hide features the guest needs
  (AES-NI, AVX, SSE4). Use `host` unless you need live migration across heterogeneous hardware.
- **Test disk config changes with shutdown/start, not reboot.** Shut down gracefully with
  `qm shutdown`, verify `qm status` reports stopped, then start a fresh QEMU process.

### Step 4: Validate

| What to validate | How |
|-----------------|-----|
| VM boots and agent responds | `qm agent <vmid> ping` (Proxmox) |
| Cloud-init completed | `cloud-init status --wait` in guest |
| Disk performance | `fio --name=test --rw=randread --bs=4k --direct=1 --numjobs=4 --runtime=30` |
| TRIM/discard working | `fstrim -v /` in guest, check thin pool data_percent on host |
| PCI passthrough | `lspci` in guest shows passed device, driver loaded |
| Network connectivity | `ping gateway`, check `ip addr` matches cloud-init config |
| Memory | `free -h` in guest matches expected (not balloon-reduced) |
| Live migration | Test with `qm migrate <vmid> <target> --online` on non-critical VM first |

`fio` is often missing in guests: `command -v fio >/dev/null || echo "fio missing: sudo apt install fio (Debian/Ubuntu)"`.
If any check fails, fix the config and return to Step 3; a hardware-level fix needs a fresh
QEMU start, not a guest reboot.

---

## Quick Task: VM from Cloud Image (Proxmox)

The fastest path to a production-ready VM. Skip Packer and ISO installs for standard setups.

1. Download a stable cloud image to the Proxmox node (not `daily/`, which can have cloud-init gaps):
   `wget -P /var/lib/vz/template/iso/ https://cloud.debian.org/images/cloud/trixie/latest/debian-13-generic-amd64.qcow2`
2. Create the VM shell:
   `qm create 100 --name myvm --memory 2048 --cores 2 --cpu host --net0 virtio,bridge=vmbr0 --agent enabled=1 --scsihw virtio-scsi-single`
3. Import and attach the disk with SSD optimizations:
   `qm importdisk 100 /var/lib/vz/template/iso/debian-13-generic-amd64.qcow2 local-lvm`
   `qm set 100 --scsi0 local-lvm:vm-100-disk-0,discard=on,iothread=1,ssd=1 --boot order=scsi0`
4. Add cloud-init drive and configure:
   `qm set 100 --ide2 local-lvm:cloudinit`
   `qm set 100 --ciuser admin --sshkeys ~/.ssh/id_ed25519.pub --ipconfig0 ip=10.0.0.100/24,gw=10.0.0.1`
5. Start: `qm start 100`
6. Verify: `qm agent 100 ping` (may take 1-2 min on first boot while cloud-init runs)

To make a reusable template, stop the VM after verification and run `qm template 100`.
Clone with `qm clone 100 101 --name new-vm --full`.

---

## Proxmox VE Quick Reference

Read `references/proxmox.md` for full coverage of API, CLI, storage backends, clustering,
HA, migration, PCI passthrough, backup, and Terraform patterns.

### Essential CLI

```bash
# VM management
qm list                              # List all VMs
qm create 100 --name test --memory 2048 --cores 2 --net0 virtio,bridge=vmbr0
qm start/stop/shutdown/reset 100     # Power operations
qm config 100                        # Show VM config
qm set 100 --memory 4096             # Modify config (some need stop/start)
qm resize 100 scsi0 +10G             # Extend disk (can't shrink)
qm agent 100 ping                    # Test QEMU agent
qm migrate 100 pve2 --online --with-local-disks  # Live migrate

# Container management
pct list                              # List LXC containers
pct create 200 local:vztmpl/debian-12-standard_12.7-1_amd64.tar.zst

# Storage
pvesm status                          # Storage pool overview
lvs -a -o+devices                     # LVM thin pool status (on LVM backend)

# Cluster
pvecm status                          # Cluster status
pvecm nodes                           # Node list
ha-manager status                     # HA status
```

### Critical gotchas (battle-tested)

These come from production Proxmox environments and will save hours of debugging:

**Stop/start vs reboot:** Guest `reboot` does NOT restart the QEMU process. Disk config
changes (discard, cache mode, iothread, bus type) only apply when QEMU starts fresh.
Use `qm shutdown`, verify stopped state, then `qm start` for hardware config changes.
This also applies to memory balloon device changes. Reserve `qm stop` for an explicitly
justified forced-stop recovery after graceful shutdown fails; it cuts guest power.

**LVM thin pool at 100%:** When data_percent hits 100%, ALL VM I/O on that pool fails
instantly - guests hang, no graceful degradation. Recovery requires `lvextend` on the
thin pool or migrating VMs off. Monitor thin pool usage and alert well before 100% (80%
warning, 90% critical). `data_percent` is the share of the pool's data space currently
allocated to its thin volumes (lvmthin(7)), not in-guest filesystem usage - a VM that wrote
then deleted 50GB keeps that 50GB allocated until a discard reaches the pool.

**Live migration via SSH:** `qm migrate` runs in the foreground. If the SSH session drops,
the migration aborts. For large VMs (32GB+ disk), use:
```bash
nohup qm migrate <vmid> <target> --online --with-local-disks \
  > /tmp/migrate-<vmid>.log 2>&1 &
```
Migration is abort-safe: source VM stays running on failure, target LVs are cleaned up.

**KVM ballooning:** The balloon device lets the host reclaim unused guest memory between the
configured minimum and `memory:`. It needs the guest's `virtio_balloon` driver: minimal Alpine
images and some BSDs do not load it, so the host's request is simply ignored. Raising `memory:`
above the running maximum is a different mechanism - DIMM hotplug, which needs `hotplug: memory`
plus NUMA and a guest that onlines new blocks, or a full stop/start. Even on Debian, balloon
behavior is unpredictable under memory pressure. Recommendation: use `floating = 0` in the
bpg/proxmox Terraform `memory` block, or set `balloon: 0` in native Proxmox config, and
provision VMs with the memory they actually need.

**fail2ban on Proxmox (Debian 13):** `/var/log/daemon.log` doesn't exist under journald.
Use `backend = systemd` with `journalmatch = _COMM=pvedaemon` in the jail config.

**openipmi on non-IPMI hardware:** Fails on boot, generates spurious alerts. Safe to
`systemctl mask openipmi` on nodes without BMC/IPMI hardware. Masking survives package
updates; disabling doesn't.

---

## Storage Performance

| Backend | Best for | Thin provision | Snapshot | Live migration |
|---------|----------|---------------|----------|----------------|
| LVM-thin | Local SSDs, production | Yes | Yes (copy-on-write) | With --with-local-disks |
| ZFS | Data integrity, compression | Yes | Yes (native) | With replication |
| Ceph/RBD | Multi-node shared storage | Yes | Yes | Native (shared) |
| NFS | ISOs, templates, backups | Depends on NAS | Depends | Yes (shared) |
| local (dir) | Small/test | No (file-based) | qcow2 only | No |

**Disk interface hierarchy** (fastest to slowest):
1. **virtio-scsi-single** + iothread - one controller per disk, best IOPS
2. **virtio-scsi-pci** without per-disk iothreads - shared controller, good for moderate I/O
3. **virtio-blk** - legacy virtio, good performance but fewer features
4. **IDE** - legacy only, needed for some old OSes

**SSD optimization checklist:**
- [ ] `ssd = 1` on disk config (tells guest it's on SSD, enables TRIM in guest)
- [ ] `discard = on` on QEMU disk (passes TRIM/UNMAP to storage backend)
- [ ] `fstrim.timer` enabled in guest (weekly by default on systemd distros)
- [ ] Verify with: `fstrim -v /` in guest, then check `lvs -o data_percent` on host

**Disk resize (the Terraform trap):** growth works from either side, but only one side may
own it. The bpg/proxmox provider grows a disk in place when `disk.size` increases - live when
`disk` is in the VM's `hotplug` list, otherwise with a provider-initiated reboot. Shrinking is
unsupported everywhere. Pick one owner:
- **Terraform-owned:** raise `size` in the `disk` block and apply. Do not put `disk` in
  `ignore_changes`, or the growth is never applied.
- **Host-owned:** `qm resize <vmid> scsi0 +10G` on the Proxmox host, keep `disk` in
  `ignore_changes`, and update the Terraform size afterwards so plans stay clean.

Either way, finish inside the guest: `growpart /dev/sda 1` (expand partition), then
`resize2fs /dev/sda1` or `xfs_growfs /` (expand filesystem). Mixing both owners produces
plans that shrink or recreate the disk.

---

## Memory Management

**Ballooning - the short version:** Don't use it unless you've tested it on your exact
guest OS and workload. Disable with `balloon: 0` in VM config.

**NUMA:** Enable for VMs with 4+ cores or 8GB+ RAM. Proxmox: `numa: 1` in VM config.
QEMU auto-creates NUMA nodes matching the host topology.

**Hugepages:** 2MB or 1GB pages reduce TLB misses. Significant for databases and
memory-intensive workloads. Configure on the host:
```bash
# Reserve 1024 x 2MB hugepages (2GB total)
echo 1024 > /proc/sys/vm/nr_hugepages
# Persistent: add to /etc/sysctl.d/
vm.nr_hugepages = 1024
```
Then enable in VM config. Note: hugepages memory can't be shared or ballooned.

**CPU hotplug vs memory hotplug:** CPU hotplug works live on most modern Linux guests.
Memory hotplug (adding DIMMs at runtime) needs `hotplug: memory` plus NUMA on the VM and a
guest that onlines the new blocks, usually through a udev rule; without all three the change
fails silently. This is a different mechanism from ballooning, which only moves memory between
the configured minimum and `memory:`. Size memory correctly at creation time.

---

## Hypervisor Selection

| Hypervisor | Type | Best for | Avoid when |
|------------|------|----------|------------|
| **Proxmox VE** | Type 1 (KVM+LXC) | Homelab, SMB, API-driven automation | Need VMware ecosystem tooling |
| **libvirt/KVM** | Type 1 (bare) | Custom setups, OpenStack, direct control | Want a GUI or clustering OOB |
| **XCP-ng** | Type 1 (Xen) | Xen-based infra, XenOrchestra UI | KVM-specific features (virtio-fs) |
| **VMware ESXi** | Type 1 | Enterprise with existing VMware investment | Post-Broadcom: licensing costs exploded |
| **VirtualBox** | Type 2 | Dev workstations, testing | Production. Ever. |

**VMware post-Broadcom (2026):** Broadcom acquired VMware (closed Nov 2023). Perpetual
licenses eliminated, subscription-only model, free ESXi discontinued then partially
reinstated (ESXi 8.0 U3e "free hypervisor", April 2025). Many organizations are
migrating to Proxmox or XCP-ng. The migration path from VMware is well-documented
but non-trivial for large estates.

---

## Reference Files

- `references/proxmox.md` - Proxmox VE deep-dive: API, CLI, storage, clustering, HA,
  live migration, PCI passthrough (including the Windows 11 + NVIDIA quick path), Proxmox
  Backup Server, and Terraform (bpg/proxmox provider patterns, lifecycle gotchas, cloud-init)
- `references/libvirt-qemu-kvm.md` - libvirt/QEMU/KVM: virsh commands, XML domain
  definitions, QEMU command-line, KVM modules, disk formats, networking, and a cloud-image
  quick start for libvirt without Proxmox
- `references/image-building.md` - Packer templates, cloud-init configuration,
  cloud image workflows, template management
- `references/gotchas.md` - Battle-tested pitfalls and failure modes from production
  Proxmox/KVM environments. Read this before any non-trivial change.

---

## Output Contract

See `references/output-contract.md` for the full contract.

- **Skill name:** VIRTUALIZATION
- **Deliverable bucket:** `audits`
- **Mode:** conditional. When invoked to **analyze, review, audit, or improve** existing repo content, apply the reporting size and evidence rules in `references/output-contract.md` and write the deliverable to `docs/local/audits/virtualization/<YYYY-MM-DD>-<slug>.md`. When invoked to **answer a question, teach a concept, build a new artifact, or generate content**, respond freely without the contract.
- **Severity scale:** `P0 | P1 | P2 | P3 | info` (see shared contract; only used in audit/review mode).

## Related Skills

- **terraform** - owns HCL patterns, module design, state management. This skill owns
  Proxmox-specific provider patterns (bpg/proxmox lifecycle rules, cloud-init interface,
  disk resize workarounds). Use terraform for general IaC; this skill for Proxmox-specific
  Terraform.
- **kubernetes** - for container orchestration running on top of VMs. This skill provisions
  the VM infrastructure; kubernetes manages what runs inside the cluster.
- **networking** - for network config not specific to hypervisors (DNS, VPNs, reverse
  proxies, nftables). This skill covers VM networking (bridges, VLANs, virtio-net).
- **ansible** - for day-2 configuration of VMs after provisioning. This skill creates the
  VM; ansible configures what runs on it.
- **docker** - for container image optimization. This skill manages VMs that may host
  Docker/container workloads.

---

## Rules

1. **Shutdown/start for hardware changes, not reboot.** Guest reboot doesn't restart QEMU.
   Shut down gracefully, verify stopped state, then start; forced stop needs separate justification.
2. **One owner of disk size.** Either grow via `disk.size` in bpg/proxmox with `disk` not
   ignored, or grow with `qm resize` on the host with `disk` ignored - never both. Finish with
   growpart plus resize2fs or xfs_growfs in the guest.
3. **Disable ballooning by default.** Enable only after testing on the specific guest OS.
4. **Monitor thin pool data_percent.** Alert at 80%, critical at 90%. At 100%, all I/O fails.
5. **nohup for long migrations.** SSH disconnect kills foreground `qm migrate`.
6. **`prevent_destroy` + `ignore_changes` on Terraform VMs.** Protect against accidental
   destruction, and ignore `node_name` and the generated MAC so live migration does not read as
   drift. Add `disk` to `ignore_changes` only on the host-owned resize path (rule 2).
7. **Test before production.** New VM configs, passthrough setups, storage backends - test
   on a non-critical VM first.