Saltstack

---
name: saltstack
description: "Covers SaltStack/Salt (3007.x): state files, pillars, grains, execution modules, Salt master/minion architecture, Salt SSH, reactors, beacons, orchestration, and event-driven automation. WHEN: \"SaltStack\", \"Salt\", \"salt-master\", \"salt-minion\", \"state file\", \".sls\", \"pillar\", \"grain\", \"salt-ssh\", \"Salt reactor\", \"Salt beacon\", \"salt-call\", \"salt highstate\"."
license: MIT
---

# SaltStack Expert

This skill covers SaltStack (Salt) 3007.x, a configuration management and remote execution platform. Salt uses a master-minion architecture with ZeroMQ for high-speed communication. It excels at managing large fleets (10K+ nodes) with real-time execution.

## Core Architecture

```
┌──────────────────┐
│   Salt Master    │  Stores states, pillars, manages minions
│  (salt-master)   │
│                  │
│  ┌────────────┐  │
│  │ Event Bus  │  │  ZeroMQ pub/sub (port 4505/4506)
│  └──────┬─────┘  │
└─────────┼────────┘
          │
    ┌─────┼─────┐
    │     │     │
┌───▼┐ ┌──▼┐ ┌──▼┐
│Min │ │Min│ │Min│   Minions subscribe to events, execute commands
│ A  │ │ B │ │ C │
└────┘ └───┘ └───┘
```

### Key Concepts

| Concept | Description |
|---|---|
| **State** | YAML `.sls` files declaring desired system configuration |
| **Pillar** | Secure, per-minion data (secrets, config specific to a host) |
| **Grain** | Static minion-side data (OS, CPU, memory — like Facter/Ohai) |
| **Execution Module** | Imperative commands (`pkg.install`, `service.restart`) |
| **State Module** | Declarative resources (`pkg.installed`, `service.running`) |
| **Top File** | Maps states/pillars to minions based on targeting |
| **Reactor** | Event-driven automation (react to events on the bus) |
| **Beacon** | Minion-side monitors that fire events (file changes, process death) |
| **Orchestration** | Multi-minion, ordered workflow execution |
| **Mine** | Share data between minions via the master |

### State File Example

```yaml
# /srv/salt/webserver/init.sls

nginx:
  pkg.installed:
    - version: 1.24.0

  service.running:
    - enable: True
    - watch:
      - file: /etc/nginx/nginx.conf
      - pkg: nginx

/etc/nginx/nginx.conf:
  file.managed:
    - source: salt://webserver/files/nginx.conf.jinja
    - template: jinja
    - user: root
    - group: root
    - mode: '0644'
    - context:
        workers: {{ grains['num_cpus'] * 2 }}
        server_name: {{ pillar['webserver']['hostname'] }}
    - require:
      - pkg: nginx

/var/www/html:
  file.directory:
    - user: www-data
    - group: www-data
    - mode: '0755'
    - makedirs: True
```

### Top File (Targeting)

```yaml
# /srv/salt/top.sls
base:
  '*':                          # All minions
    - common
    - monitoring

  'web*':                       # Glob targeting
    - webserver

  'os:Ubuntu':                  # Grain targeting
    - match: grain
    - ubuntu.packages

  'G@os:CentOS and G@role:db': # Compound targeting
    - match: compound
    - database
```

### Pillar (Secure Data)

```yaml
# /srv/pillar/webserver.sls
webserver:
  hostname: www.example.com
  ssl_cert: |
    -----BEGIN CERTIFICATE-----
    ...
  db_password: SuperSecret123

# /srv/pillar/top.sls
base:
  'web*':
    - webserver
```

### Execution vs State

```bash
# Execution (imperative — do this now)
salt '*' pkg.install nginx
salt 'web1' service.restart nginx
salt '*' cmd.run 'uptime'
salt '*' disk.usage

# State (declarative — ensure this is true)
salt '*' state.apply              # Apply all states (highstate)
salt 'web1' state.apply webserver # Apply specific state
salt '*' state.test               # Dry run
```

### Targeting

```bash
# Glob
salt 'web*' test.ping

# Grain (OS, role, etc.)
salt -G 'os:Ubuntu' test.ping

# Pillar
salt -I 'role:webserver' test.ping

# Compound (boolean logic)
salt -C 'G@os:Ubuntu and web*' test.ping

# List
salt -L 'web1,web2,web3' test.ping

# PCRE regex
salt -E 'web[0-9]+\.example\.com' test.ping

# Nodegroup (predefined in master config)
salt -N webservers test.ping
```

### Reactors (Event-Driven)

```yaml
# /etc/salt/master.d/reactor.conf
reactor:
  - 'salt/minion/*/start':        # When any minion connects
    - /srv/reactor/new_minion.sls
  - 'salt/beacon/*/inotify/*':    # When file changes detected
    - /srv/reactor/file_changed.sls

# /srv/reactor/new_minion.sls
apply_base_state:
  local.state.apply:
    - tgt: {{ data['id'] }}
    - arg:
      - common
```

### Orchestration

```yaml
# /srv/salt/orch/deploy.sls
# Ordered, multi-minion deployment

step_1_update_db:
  salt.state:
    - tgt: 'db*'
    - sls: database.migrate

step_2_deploy_app:
  salt.state:
    - tgt: 'web*'
    - sls: myapp.deploy
    - require:
      - salt: step_1_update_db

step_3_verify:
  salt.function:
    - name: http.query
    - tgt: 'web1'
    - arg:
      - https://myapp.example.com/health
    - require:
      - salt: step_2_deploy_app
```

```bash
# Run orchestration
salt-run state.orchestrate orch.deploy
```

### Salt SSH (Agentless)

```bash
# Run commands without minion agent
salt-ssh '*' test.ping
salt-ssh 'web1' state.apply webserver

# Roster file (inventory for salt-ssh)
# /etc/salt/roster
web1:
  host: 10.0.1.10
  user: deploy
  sudo: True
web2:
  host: 10.0.1.11
  user: deploy
  priv: /root/.ssh/id_ed25519
```

### CLI Reference

```bash
# Master management
salt-key --list all           # List accepted/pending/rejected keys
salt-key --accept web3        # Accept a new minion
salt-key --delete web3        # Remove a minion

# Remote execution
salt '*' test.ping            # Connectivity check
salt '*' grains.items         # All grains
salt '*' pillar.items         # All pillar data (careful: shows secrets)
salt '*' state.apply          # Apply highstate
salt '*' state.apply webserver test=True  # Dry run

# Salt-run (master-side)
salt-run manage.status        # Show up/down minions
salt-run state.orchestrate orch.deploy

# Salt-call (minion-side, local execution)
salt-call state.apply --local
salt-call grains.items
```

## Reference Files

- `references/architecture.md` — ZeroMQ transport, event bus, mine system, syndic (multi-master), renderer pipeline, returner system
- `references/best-practices.md` — State organization, pillar design, formula conventions, targeting strategy, security hardening
- `references/diagnostics.md` — Minion connectivity, key management, state compilation errors, pillar rendering issues, event bus debugging