Hetzner

---
name: hetzner-cloud
description: "Hetzner Cloud servers, volumes, private networks, firewalls, backups and snapshots, through the hcloud CLI or Pulumi/Terraform. Use when provisioning, resizing or scripting Hetzner resources, or choosing a server type."
---

# Hetzner Cloud

Facts that drift (prices, SKUs, locations, limits) go stale — **prefer retrieval over pre-training**: `hcloud server-type list -o json`, `hcloud location list`, docs.hetzner.com. Everything below is behavior, not price sheets. If the project documents a topology (e.g. `docs/architecture/`), read it before provisioning anything.

## Provisioning Workflow

1. **Pick type + location together** — CAX (Ampere arm64, best RAM/€) exists **only in EU DCs** (fsn1/nbg1/hel1); verify with `hcloud server-type list`. arm64 requires multi-arch container images end-to-end.
   - Checkpoint: `hcloud server-type describe <type>` shows the target location.
2. **Create network + firewall before servers** — attach at create time; firewall via label selector (`role=web`) scales better than per-server IDs.
   - Checkpoint: firewall rules list SSH restricted to admin IPs, never `0.0.0.0/0`.
3. **Create server with cloud-init user-data** — runs once at first boot only.
   - Recovery: broken cloud-init → don't patch by hand; fix the template, `server rebuild` (keeps IPs) or replace via IaC.
4. **Attach volumes, then mount explicitly** — see Volumes below.
   - Checkpoint: mount survives reboot (`fstab` entry, not just `automount`).
5. **Enable protection on stateful resources** — `hcloud server enable-protection <s> delete rebuild` + same for volumes.
   - Gate: protection verified via `hcloud server describe` before real data lands.

## Non-Obvious Behaviors

### Networking
- **Cloud Firewalls filter ONLY the public interface.** Private-network traffic is never touched by them — segment the private net with host firewalls (`ufw`) or not at all. This is the most common Hetzner security misconception.
- Private network **MTU is 1450**, not 1500 — override in Docker (`daemon.json` `"mtu": 1450`) and any overlay/VPN on top, or suffer silent large-packet stalls.
- Private networks are **not encrypted** — treat as shared fabric, keep TLS for sensitive cross-node traffic.
- Network zones (e.g. `eu-central` = fsn1/nbg1/hel1) span DCs — cross-DC private traffic works and is free.
- **Floating IPs need manual OS config** (netplan alias); primary IPs configure themselves. Floating IP ≠ instant failover without host-side automation.
- IPv4 primary IPs bill separately and **survive server deletion** unless `auto_delete` is set — orphaned IPs keep billing.

### Servers & billing
- Hourly billing capped at the monthly price. Deleting a server stops its billing but **volumes, snapshots, and primary IPs keep billing** — sweep for orphans after teardowns.
- `change-type` (resize) requires **ACPI shutdown first** (`hcloud server shutdown`, not `poweroff` — that's a hard cut risking fs damage). CPU/RAM can downgrade later only with `--keep-disk`; disk never shrinks.
- `server rebuild` wipes the disk but **keeps all IPs** — the fast DR path when the root cause is software.
- 20 TB egress included per cloud server; ingress + private-net traffic free.
- Metadata endpoint: `http://169.254.169.254/hetzner/v1/metadata` (useful in cloud-init to self-discover private IP).

### Volumes
- **Location-bound** — attach only within the same DC; a "move to another region" is snapshot-less (rsync) by design.
- **Grow-only, online** — after `hcloud volume resize`, run `resize2fs`/`xfs_growfs` yourself; nothing automatic.
- `automount` mounts under `/mnt/<volume-name>` via udev — fine for experiments, wrong for services. Use an explicit `fstab` entry (`discard,nofail`) at a stable path; `nofail` so a detached volume doesn't hang boot.

### Backups vs snapshots
| | Backups | Snapshots |
|---|---|---|
| Trigger | Automatic daily, 7 rolling | Manual |
| Cost | +20% of server price | Per GB stored |
| Survives server deletion | **No** | Yes |
| Can seed a new server | No (convert to snapshot first) | Yes (`--image <snapshot-id>`) |

Backup→snapshot conversion exists — do it before deleting a server whose history matters. Neither replaces app-level data exports — Hetzner backups are crash-consistent block copies, not application backups.

### IaC (Pulumi / Terraform `hcloud` provider)
- **Changing `userData` replaces the server.** Deliberate for cattle; catastrophic for stateful nodes. Keep volatile config out of cloud-init (runtime config management owns it) and treat any cloud-init diff as a planned node replacement — or `ignoreChanges: ["userData"]` on pet servers.
- `serverType` change is a **replacement** in IaC even though the console/CLI resizes in place. To resize without replacement: `hcloud server change-type` manually, then refresh state + update code.
- Pulumi: `firewallIds` wants numbers — `firewall.id.apply(Number)`. Inline `networks` on the Server conflicts with a separate `ServerNetwork` resource — pick one.
- `backups: true` toggles in place (safe anytime). IaC-level `protect` only blocks the IaC tool — set **hcloud-side** delete/rebuild protection too; they guard different doors.
- Auth: `HCLOUD_TOKEN` env (CI/agents) — per-project tokens, read-write only where needed.

## CLI Patterns

```bash
export HCLOUD_TOKEN=...                       # stateless auth for CI/agents; `hcloud context` for humans
hcloud server list -o columns=name,status,ipv4,type -o noheader
hcloud server ssh my-server                   # no IP lookup, uses your agent keys
hcloud server create-image my-server --type snapshot --description "pre-change $(date +%F)"
hcloud volume list -o json | jq -r '.[] | select(.server==null) | .name'   # orphaned volumes
hcloud primary-ip list -o json | jq -r '.[] | select(.assignee_id==null) | .ip'  # orphaned IPs
hcloud server shutdown my-server \
  && hcloud server change-type my-server --server-type cax31 --keep-disk \
  && hcloud server poweron my-server          # in-place upgrade, downgrade-capable
```

## Anti-Patterns

| Anti-pattern | Fix |
|-------------|-----|
| Trusting Cloud Firewall for private-net isolation | Host-level `ufw`; Cloud Firewall covers public interface only |
| SSH open to `0.0.0.0/0` | Admin-IP allowlist in firewall rules (or SSH over tunnel only) |
| `poweroff` before resize/maintenance | `shutdown` (ACPI graceful), then operate |
| Editing `userData` on a stateful server in IaC | Post-boot config mgmt; `ignoreChanges` or planned replacement |
| Relying on `automount` for service data | Explicit `fstab` (`discard,nofail`) at a stable mount point |
| Assuming CAX exists in every location | EU-only; check `server-type list` before designing |
| Deleting servers and assuming cleanup | Sweep volumes/snapshots/primary IPs — they persist and bill |
| Hetzner Backups as the only backup | App-level exports to object storage; backups die with the server |

## References

| Resource | Purpose |
|----------|---------|
| <https://docs.hetzner.com/cloud/> | Authoritative, current limits/prices |
| <https://www.pulumi.com/registry/packages/hcloud/> | Pulumi provider resource reference |
| <https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs> | Terraform provider reference |
| **deployment** capability skill | Platform-level deploy/env-var/cron patterns |