Slv Validator

# SLV Validator Skill

Ansible playbooks and Jinja2 templates for deploying and managing Solana validators (mainnet and testnet).

## Supported Validator Types

| Type | Description |
|---|---|
| `jito` | Jito MEV client (default for mainnet) |
| `allnodes-jito` | Allnodes-Jito (jito fork with snapshot/voting/POH/SHA-NI patches) |
| `agave` | Standard Agave validator |
| `firedancer-agave` | Firedancer with Agave consensus |
| `firedancer-jito` | Firedancer with Jito consensus |

## Directory Structure

```
ansible/
  mainnet-validator/   — Mainnet validator playbooks
  testnet-validator/   — Testnet validator playbooks
  cmn/                 — Shared common playbooks
jinja/
  mainnet-validator/   — Mainnet Jinja2 templates
  testnet-validator/   — Testnet Jinja2 templates
  cmn/                 — Shared templates
```

## CLI Command ↔ Playbook Mapping

The `slv v` CLI commands map directly to these playbooks. `{net}` = `mainnet-validator` or `testnet-validator`.

| CLI Command | Playbook | Description |
|---|---|---|
| `slv v init` | *(no playbook — interactive prompts)* | Generate `~/.slv/inventory.<network>.validators.yml` |
| `slv v deploy` | `{net}/init.yml` | Full node initialization and deployment |
| `slv v start` | `{net}/start_node.yml` | Start validator |
| `slv v stop` | `{net}/stop_node.yml` | Stop validator |
| `slv v restart` | `{net}/restart_node.yml` | Graceful restart (`agave-validator exit`, then systemd) |
| `slv v build:solana` | `{net}/install_solana.yml` | Build Solana from source (dispatches by `validator_type`) |
| `slv v install:solana` | `cmn/install_solana.yml` | Install Solana binary (deprecated, prefer build) |
| `slv v setup:firedancer` | `{net}/setup_firedancer.yml` | Setup/update Firedancer |
| `slv v update:firedancer` | `{net}/update_firedancer.yml` | Update Firedancer binary |
| `slv v update:script` | `{net}/update_startup_config.yml` | Re-render start-validator.sh from template |
| `slv v set:identity` | `{net}/set_identity_key.yml` | Set validator identity key |
| `slv v set:unstaked` | `{net}/set_unstaked_key.yml` | Switch to unstaked identity |
| `slv v register:bls` | *(no playbook — solana CLI)* | Register the BLS pubkey on each vote account (SIMD-0387). Also auto-runs at the end of `slv v deploy` |
| `slv v get:snapshot` | `{net}/wget_snapshot.yml` | Download snapshot via aria2c |
| `slv v cleanup` | `cmn/rm_ledger.yml` | Remove ledger/snapshot files |
| `slv disable pwd-login` | `cmn/disable_pwd_login.yml` | Disable SSH password authentication |
| `slv v switch` | `{net}/nodowntime_migrate.yml` | Zero-downtime identity migration (auto-swaps inventory) |
| `slv v list` | *(no playbook)* | List validators (CLI only) |
| `slv v gen:vote-account` | *(no playbook)* | Create vote account (solana CLI) |

## All Playbooks

### Mainnet Validator (`mainnet-validator/`)

| Playbook | Description |
|---|---|
| `init.yml` | Full node initialization (Agave/Jito) |
| `init-jito.yml` | Jito-specific initialization |
| `init-firedancer.yml` | Firedancer initialization |
| `restart_node.yml` | Restart validator |
| `start_node.yml` | Start validator |
| `stop_node.yml` | Stop validator |
| `install_solana.yml` | Build Solana from source |
| `install_agave.yml` | Build Agave from source |
| `install_jito.yml` | Build Jito from source |
| `install_rust.yml` | Install Rust toolchain |
| `setup_firedancer.yml` | Setup Firedancer binary and config |
| `update_firedancer.yml` | Update Firedancer binary |
| `update_startup_config.yml` | Update start-validator.sh from Jinja template |
| `deploy-start-validator-sh.yml` | Deploy start script to remote |
| `create-start-validator-sh.yml` | Generate start script from template |
| `nodowntime_migrate.yml` | Zero-downtime identity migration between nodes |
| `set_identity_key.yml` | Set validator identity key |
| `set_identity_to_active.yml` | Activate identity key |
| `set_unstaked_key.yml` | Switch to unstaked identity |
| `switch_on_identity.yml` | Enable identity (tower copy + key deploy) |
| `switch_off_identity.yml` | Disable identity (tower backup) |
| `switch_on_firedancer_identity.yml` | Enable Firedancer identity |
| `switch_off_firedancer_identity.yml` | Disable Firedancer identity |
| `copy_keys.yml` | Copy validator keys to node |
| `copy_restart_sh.yml` | Copy restarter script |
| `create_overrides.yml` | Generate overrides.yml from template |
| `setup_solv_service.yml` | Setup systemd service |
| `start-solv-service.yml` | Start systemd service |
| `setup_ufw.yml` | Configure UFW firewall |
| `setup_fb_ufw.yml` | Configure Firedancer-specific UFW rules |
| `setup_logrotate.yml` | Setup log rotation |
| `configure_hugetlbfs.yml` | Configure hugepages for Firedancer |
| `fail2ban_solana_rate_limit.yml` | Setup fail2ban rate limiting |
| `run_snapshot_finder.yml` | Find and download best snapshot |

### Testnet Validator (`testnet-validator/`)

| Playbook | Description |
|---|---|
| `init.yml` | Full initialization (Jito) |
| `init-agave.yml` | Agave-specific initialization |
| `init-firedancer.yml` | Firedancer initialization |
| `restart_node.yml` | Restart validator |
| `start_node.yml` | Start validator |
| `stop_node.yml` | Stop validator |
| `install_solana.yml` | Build Solana from source |
| `install_agave.yml` | Build Agave from source |
| `install_jito.yml` | Build Jito from source |
| `install_firedancer.yml` | Build Firedancer from source |
| `setup_firedancer.yml` | Setup Firedancer |
| `setup_firedancer_agave.yml` | Setup Firedancer with Agave |
| `setup_firedancer_jito.yml` | Setup Firedancer with Jito |
| `update_firedancer.yml` | Update Firedancer binary |
| `update_startup_config.yml` | Update start script from template |
| `deploy-start-validator-sh.yml` | Deploy start script |
| `create-start-validator-sh-agave.yml` | Generate Agave start script |
| `create-start-validator-sh-jito.yml` | Generate Jito start script |
| `nodowntime_migrate.yml` | Zero-downtime identity migration |
| `set_identity_key.yml` | Set identity key |
| `set_identity_to_active.yml` | Activate identity |
| `set_unstaked_key.yml` | Switch to unstaked identity |
| `switch_on_identity.yml` / `switch_off_identity.yml` | Toggle identity |
| `switch_on_firedancer_identity.yml` / `switch_off_firedancer_identity.yml` | Toggle Firedancer identity |
| `change_identity_and_restart.yml` | Change identity and restart in one step |
| `copy_keys.yml` | Copy keys to node |
| `rm_ledger.yml` | Remove ledger data |
| `restart_agave_with_rm_ledger.yml` | Restart with ledger removal |
| `restart_firedancer.yml` | Restart Firedancer |
| `restart_firedancer_with_rm_ledger.yml` | Restart Firedancer with ledger removal |
| `restart_solv.yml` | Restart solv service |
| `setup_agave.yml` | Setup Agave |
| `setup_agave_ufw.yml` | Agave UFW rules |
| `setup_solv_service.yml` | Setup systemd service |
| `setup_solv_service_init.yml` | Initialize systemd service |
| `setup_snapshot_finder.yml` | Setup snapshot finder |
| `add_solv.yml` | Add solv user |

### Shared Common (`cmn/`)

| Playbook | Description |
|---|---|
| `build_solana.yml` | Build Solana from source (dispatches to build_agave/build_jito) |
| `build_agave.yml` | Build Agave from GitHub source |
| `build_jito.yml` | Build Jito from GitHub source |
| `install_solana.yml` | Install Solana binary (deprecated) |
| `install_package.yml` | Install system packages |
| `install_rust.yml` | Install Rust toolchain |
| `mount_disks.yml` | Mount and format disks |
| `optimize_system.yml` | Optimize system settings (sysctl, limits) |
| `disable_swap.yml` | Disable swap |
| `disable_pwd_login.yml` | Disable SSH password authentication and restart sshd |
| `setup_logrotate.yml` | Configure log rotation |
| `setup_node_exporter.yml` | Setup Prometheus node exporter |
| `setup_norestart.yml` | Disable auto-restart |
| `setup_ufw.yml` | Configure UFW firewall |
| `setup_unstaked_identity.yml` | Setup unstaked identity keypair |
| `restart_solv.yml` | Restart solv service |
| `copy_restart_sh.yml` | Copy restarter script |
| `update_ubuntu.yml` | Update Ubuntu packages |
| `wget_snapshot.yml` | Download snapshot |
| `add_solv.yml` | Add solv user |
| `rm_ledger.yml` | Remove ledger data |
| `fix_permissions.yml` | Fix file permissions |

## Key Variables (extra_vars)

| Variable | Description | Default |
|---|---|---|
| `validator_type` | Validator type (`jito`, `allnodes-jito`, `agave`, `firedancer-agave`, `firedancer-jito`) | `jito` |
| `solana_version` | Solana/Agave version to build (used when `validator_type == 'agave'`) | — |
| `jito_version` | Jito version to build (used when `validator_type == 'jito'`) | — |
| `allnodes_jito_version` | Allnodes-Jito version (used when `validator_type == 'allnodes-jito'`); template appends `-allnodes` to form the git tag | — |
| `firedancer_version` | Firedancer version | — |
| `snapshot_url` | Snapshot download URL | — |
| `identity_account` | Validator identity pubkey | — |
| `vote_account` | Vote account pubkey | — |
| `block_engine_url` | Jito block engine URL | `https://frankfurt.mainnet.block-engine.jito.wtf` |
| `shred_receiver_address` | Jito shred receiver. **Accepts a single string or a YAML list** — a list emits one `--shred-receiver-address` flag per entry | `64.130.50.14:1002` |
| `bam_url` | BAM scheduler URL — the validator starts with `--bam-url <value>` and joins the BAM pipeline. Applies to `jito` / `allnodes-jito` and must be paired with `block_engine_url` + `shred_receiver_address` (and the tip/merkle MEV flags). Can also be toggled at runtime without a restart: `agave-validator -l <ledger> set-bam-config --bam-url <url>` (empty to disable). Replaces the removed `jito-bam` validator_type | — |
| `commission_bps` | Commission in basis points | `0` |
| `dynamic_port_range` | Validator port range | `8000-8025` |
| `limit_ledger_size` | Ledger size limit | `200000000` |
| `expected_shred_version` | Expected shred version (testnet, epoch-dependent) | — |
| `expected_bank_hash` | **Cluster-restart only.** Emitted only when explicitly set (gated `is defined`). Do NOT set for normal operation — a stale value triggers a bank-hash-mismatch panic on restart | — |
| `wait_for_supermajority` | **Cluster-restart only.** Emitted only when explicitly set (gated `is defined`). Do NOT set for normal operation — a stale slot makes the validator hang/fail at startup | — |
| `xdp_enabled` | Enable XDP (eXpress Data Path) Turbine-retransmit acceleration. `agave` / `jito` only (Firedancer uses XDP natively). Gates the `--experimental-retransmit-xdp-*` flags and the systemd `CAP_NET_RAW/NET_ADMIN/BPF/PERFMON` capabilities. Per-host (NIC-/CPU-dependent) | `false` |
| `xdp_interface` | XDP NIC (bond member, e.g. the active data interface). Required when `xdp_enabled`; the flag block is skipped if blank | — |
| `xdp_cpu_cores` | XDP retransmit CPU core count | `1` |
| `xdp_zero_copy` | Enable `--experimental-retransmit-xdp-zero-copy`. Do NOT enable on `bnxt_en` / `ice` drivers | `false` |
| `xdp_poh_pinned_cpu_core` | Optional PoH pinned CPU core (`--experimental-poh-pinned-cpu-core`) | — |
| `source_host` | Source host for nodowntime migration | — |
| `target_host` | Target host for nodowntime migration | — |

## Three Core Workflows

Almost every validator task is one of these three. Use the `slv v ...` shortcuts
where possible — they wrap the right ansible command and inventory lookup.

### A. Initial setup
```bash
slv v init                           # interactive — writes inventory
slv v deploy -n <network> -p <host>  # ansible: {net}-validator/init.yml
```
The `init.yml` playbook handles user creation, package install, disk mount,
performance tuning, source build, snapshot download, systemd setup, and start.

### B. Version / config update
```bash
# 1. Edit ~/.slv/versions.yml (e.g. version_jito: 3.1.13 -> 3.1.14)
# 2. Build new binary from source
slv v build:solana -n <network> -p <host>
# 3. Re-render start-validator.sh (picks up inventory changes such as bam_url
#    being added or shred_receiver_address turning into a list)
slv v update:script -n <network> -p <host>
# 4. Restart
slv v restart -n <network> -p <host>
```

### C. Zero-downtime identity migration
```bash
slv v switch -n <network> -f <from_host> -t <to_host>
```
Runs `{net}-validator/nodowntime_migrate.yml`:
1. `set-identity` to unstaked on `from_host`, copy tower file to local.
2. Upload tower to `to_host`, `set-identity` to the staked key, register as
   `authorized-voter`.
3. Swap the host entries in the inventory file (so `slv v ...` keeps targeting
   the right physical box afterward).

`from_host` / `to_host` are inventory keys (e.g. `validator-primary`,
`validator-spare`), not IPs. Both must have `validator_type` set
(`agave`, `jito`, or `allnodes-jito`), the local key file at
`~/.slv/keys/<identity_account>.json`, and `unstaked-identity.json` on each
remote box.

## Validator Requirements: BLS (SIMD-0387), XDP, BAM

### BLS public key (SIMD-0387)

Voting validators must register the BLS public key derived from their authorized
voter keypair. Without it the vote account behaves as **unstaked** once SIMD-0357
(Alpenglow voting) activates. The authorized voter defaults to the validator
identity, so the identity keypair signs and fills the derived BLS key:

```bash
slv v register:bls -n <network> -p <host>   # all hosts if -p omitted
```

- Runs `solana vote-authorize-voter-checked <vote> <identity> <identity> --use-v2-instruction`.
  Forcing the v2 instruction means that while the feature gate is still inactive
  the tx fails cleanly at simulation **without mutating state** (no wasted
  voter re-authorization). The authorized voter is unchanged.
- Idempotent: skips hosts whose on-chain `blsPubkeyCompressed` is already set and
  re-reads it to confirm success.
- Auto-runs (non-fatal) at the end of `slv v deploy`. Re-run manually after the
  feature activates on the target cluster.
- Requires a solana/agave CLI new enough to expose `vote-authorize-voter-checked
  --use-v2-instruction` and `solana-keygen bls_pubkey` (v4.x).

### XDP retransmit acceleration

`agave` / `jito` only — Firedancer uses XDP natively. Set the per-host
`xdp_*` inventory vars (see Key Variables); `slv v init` prompts for them. When
`xdp_enabled` + a non-empty `xdp_interface` are present, the start script gains
`--experimental-retransmit-xdp-*` flags and the systemd unit gets
`CAP_NET_RAW CAP_NET_ADMIN CAP_BPF CAP_PERFMON`. Needs kernel 6.8+ (igb: 6.14+).
Do not enable `xdp_zero_copy` on `bnxt_en` / `ice` drivers. Flags are
`--experimental-` in current Agave and may be renamed upstream.

### BAM (jito)

Joining BAM needs `bam_url` **plus** `block_engine_url` and
`shred_receiver_address` (BAM runs alongside the block engine, it does not
replace it), and the MEV flags (`--tip-payment-program-pubkey`,
`--tip-distribution-program-pubkey`, `--merkle-root-upload-authority`,
`--commission-bps`) — all already emitted by the jito start template. A relayer
URL is not used with BAM. BAM can be enabled/disabled at runtime without a
restart via `agave-validator -l <ledger> set-bam-config --bam-url [<url>]`.
A validator only connects to BAM while it is in the leader schedule.

## Direct ansible invocation

If you need to bypass the `slv v` wrapper, the playbooks are designed to run
directly via `ansible-playbook` with `extra_vars`:

```bash
ansible-playbook -i inventory.yml mainnet-validator/init.yml \
  -e '{"validator_type":"allnodes-jito","allnodes_jito_version":"3.1.14","snapshot_url":"https://..."}'
```

No `versions.yml` required when you pass everything via `extra_vars`.

## Interactive Deployment Flow

When deploying a new validator, the agent should guide the user through variable collection
in this order. See `AGENT.md` for the full step-by-step flow and `examples/inventory.yml`
for the generated output format.

### Required Variables (must collect)

| Variable | Prompt | Validation |
|---|---|---|
| `server_ip` | "Target server IP?" | Valid IPv4 |
| `network` | "Mainnet or testnet?" | `mainnet` or `testnet` |
| `region` | "Server region? (amsterdam, frankfurt, tokyo, ny, ...)" | String |
| `validator_type` | "Which validator type?" | `jito`, `allnodes-jito`, `agave`, `firedancer-agave`, `firedancer-jito` |
| `solana_version` | "Solana version? (default: 3.1.8)" | Semver |
| `jito_version` | "Jito version?" (if jito/allnodes-jito) | Semver |
| `firedancer_version` | "Firedancer version?" (if firedancer) | String |
| `identity_account` | "Validator identity pubkey? (or generate)" | Base58 pubkey or `generate` |
| `vote_account` | "Vote account pubkey? (or generate)" | Base58 pubkey or `generate` |
| `snapshot_url` | "Snapshot URL? (auto-detected for ERPC nodes)" | URL (cannot be empty for init) |

### Optional Variables (show defaults, confirm)

| Variable | Default | When Required |
|---|---|---|
| `ssh_user` | `solv` (`ubuntu` for fresh servers) | Always |
| `commission_bps` | `0` | Always |
| `dynamic_port_range` | `8000-8025` | Always |
| `limit_ledger_size` | `200000000` | Always |
| `allowed_ssh_ips` | — | Strongly recommended (UFW) |
| `allowed_ips` | — | Optional (UFW) |
| `block_engine_url` | Auto by region | Jito types only |
| `shred_receiver_address` | Auto by region | Jito types only |
| `bam_url` | — | Jito types joining BAM |
| `expected_shred_version` | Epoch-dependent | Testnet only |
| `expected_bank_hash` | — | Cluster restart only (leave unset normally) |
| `wait_for_supermajority` | — | Cluster restart only (leave unset normally) |
| `xdp_enabled` (+ `xdp_interface`, `xdp_cpu_cores`, `xdp_zero_copy`, `xdp_poh_pinned_cpu_core`) | `false` | agave/jito, per-host |

### Optional: Reference RPC

| Variable | Description | Default |
|---|---|---|
| `reference_rpc_url` | Reference RPC endpoint for slot sync comparison (e.g., ERPC) | — |

ERPC API keys are free at https://erpc.global — enables full slot sync monitoring during deployment and updates.

### Pre-flight: Fresh Server Setup

If the target is a new server without a `solv` user:
```bash
ansible-playbook -i inventory.yml cmn/add_solv.yml \
  -e '{"ansible_user":"ubuntu"}' --become
```

### Deployment Command

All paths relative to skill's `ansible/` directory:
```bash
cd /path/to/slv-validator/ansible/
ansible-playbook -i inventory.yml {network}-validator/init.yml \
  -e '{"validator_type":"<type>","solana_version":"<version>","snapshot_url":"<url>"}'
```

### Dry-Run First

Always offer `--check` mode before actual deployment:
```bash
ansible-playbook -i inventory.yml {network}-validator/init.yml \
  -e '{"validator_type":"jito","solana_version":"3.1.8"}' --check
```

## Performance Tuning

`slv v init` automatically applies node-level performance tuning **after the SSH
connection check** and **before** `slv v deploy`. The orchestrator playbook
(`cmn/optimize_node.yml`) chains three sub-skills and reboots the node once
when GRUB / kernel changes require it.

| Sub-skill | Playbook | Description | Reboot |
|---|---|---|---|
| `slv-smt-disable` | `cmn/smt_disable.yml` | Disables Hyper-Threading via GRUB `nosmt=force` | **Yes** (first run) |
| `slv-irq-tuning` | `cmn/irq_tune.yml` | NIC IRQ 1:1 pin + RPS/XPS + ring 8192 + sysctl + ulimit + THP off + NVMe queue=ncores | Only if NVMe GRUB changed |
| `slv-performance-boost` | `cmn/boost_performance.yml` | HWE kernel install + amd_pstate=active + governor=performance + EPP + boost + cpuidle disable + scaling_max_freq | **Yes** if AMD < 6.14 or AMD GRUB updated |

### Idempotency markers

Each sub-skill writes a marker file so subsequent runs no-op safely:

| Marker | Skill |
|---|---|
| `/var/lib/slv/.smt_disabled` | smt_disable |
| `/var/lib/slv/.irq_tuned` | irq_tune |
| `/var/lib/slv/.boost_applied` | boost_performance |

### Reboot Flow

`cmn/optimize_node.yml` runs the three sub-playbooks, collects each one's
`need_reboot` fact, then calls `ansible.builtin.reboot` once if any of them
requested it (waits up to 15 min for the host to come back). No manual user
intervention is required.

### Standalone Usage

```bash
# Run full optimization stack
ansible-playbook -i inventory.yml cmn/optimize_node.yml --limit <host>

# Or run a single sub-skill
ansible-playbook -i inventory.yml cmn/smt_disable.yml --limit <host>
ansible-playbook -i inventory.yml cmn/irq_tune.yml --limit <host>
ansible-playbook -i inventory.yml cmn/boost_performance.yml --limit <host> -e amd_pstate_mode=active
```

### CLI Command Mapping

| CLI Command | Playbook |
|---|---|
| `slv v init` (after SSH check) | `cmn/optimize_node.yml` (auto-reboot if needed) |
| `slv v deploy` | `{net}/init.yml` |
| *(standalone optimization)* | `cmn/optimize_node.yml` |