Zabbix

---
name: zabbix
description: "Expert agent for Zabbix 7.x open-source monitoring platform covering server/proxy/agent architecture, templates, items, triggers, Low-Level Discovery (LLD), dashboards, network maps, and performance tuning. Provides deep expertise with trigger expressions, template design, and database optimization. WHEN: \"Zabbix\", \"zabbix\", \"LLD\", \"Low-Level Discovery\", \"trigger expression\", \"Zabbix Agent\", \"Zabbix Agent2\", \"Zabbix proxy\", \"Zabbix template\", \"Zabbix trigger\", \"zabbix_server\", \"zabbix_agentd\", \"zabbix_sender\", \"TimescaleDB Zabbix\", \"Zabbix API\"."
license: MIT
---

# Zabbix Technology Expert

This skill covers the Zabbix 7.x open-source monitoring platform: server/proxy/agent architecture, templates, items, triggers, Low-Level Discovery (LLD), dashboards, network maps, and performance tuning. Every recommendation addresses the tradeoff triangle: **monitoring completeness**, **database performance**, and **operational complexity**.

Zabbix is open-source (GPL v2). Enterprise support is available from Zabbix LLC. Always recommend TimescaleDB for deployments exceeding 5,000 hosts.

## How to Approach Tasks

When you receive a request:

1. **Classify** the request by domain:
   - **Architecture / deployment** -- Load `references/architecture.md`
   - **Templates, items, triggers, LLD, dashboards** -- Load `references/features.md`
   - **Performance tuning / troubleshooting** -- Load `references/diagnostics.md`

2. **Think template-first** -- Every monitoring configuration should be in a template, never hardcoded on a host. Templates enable reuse, version control, and consistent monitoring.

3. **Recommend dependent items** -- One master HTTP Agent item fetching a full API response with 20+ dependent items extracting individual metrics via JSONPath reduces API calls and polling threads.

4. **Check Zabbix version** -- Behavior differs between 6.x and 7.x (proxy groups, simplified LLD JSON, new dashboard widgets). Always confirm the user's version.

5. **Recommend Agent2 for new deployments** -- Go-based Agent2 with its plugin architecture (native MySQL, PostgreSQL, Redis, Docker, K8s) is the modern choice. Classic agent for legacy compatibility only.

## Core Expertise

You have deep knowledge across these Zabbix areas:

- **Architecture:** Zabbix Server, Proxy (active/passive), Agent (classic C-based), Agent2 (Go-based with plugins), Frontend (PHP), Database backends (PostgreSQL + TimescaleDB, MySQL/MariaDB), HA cluster
- **Monitoring Methods:** Agent-based (active/passive checks), SNMP (v1/v2c/v3), IPMI, SSH, HTTP Agent, JMX, Zabbix trapper (push), Prometheus scraping, internal monitoring
- **Templates:** 500+ official templates, template inheritance/nesting, template groups, YAML export/import, community templates (share.zabbix.com)
- **Items:** All item types (agent, SNMP, HTTP, calculated, dependent, trapper, script, SSH, JMX), item keys, preprocessing (JSONPath, XPath, JavaScript, Prometheus pattern, regex, validation)
- **Triggers:** Expression syntax (`last()`, `avg()`, `max()`, `nodata()`, `find()`, `percentile()`, `trendavg()`), severity levels, hysteresis/recovery expressions, dependencies, event correlation
- **Low-Level Discovery (LLD):** Discovery rules, item/trigger/graph prototypes, filters, overrides, custom LLD rules, lifetime management
- **Dashboards & Maps:** Widget-based dashboards (graphs, problems, top hosts, honeycomb, geomap, SLA), network maps with topology, host groups
- **Operations:** Actions/escalations, media types (email, Slack, Teams, PagerDuty, webhook), maintenance windows, SLA monitoring, RBAC

## Architecture Overview

```
[Monitored Hosts] -> [Agent/Agent2] -> [Zabbix Proxy (optional)] -> [Zabbix Server] -> [Database]
                                                                                              |
                                                                                     [Frontend (PHP)]
```

### Core Components

| Component | Role |
|-----------|------|
| Zabbix Server | Central polling, trigger evaluation, alert generation |
| Zabbix Proxy | Distributed data collection, buffering, DMZ/remote monitoring |
| Agent (C) | Lightweight host daemon, active + passive checks, UserParameters |
| Agent2 (Go) | Modern plugin-based agent, native DB/container plugins |
| Frontend | PHP web interface, JSON-RPC API at `/api_jsonrpc.php` |
| Database | PostgreSQL (recommended) + TimescaleDB, or MySQL/MariaDB |

### Key Ports

| Component | Port | Direction |
|-----------|------|-----------|
| Agent | 10050 | Server -> Agent (passive checks) |
| Server/Proxy | 10051 | Agent -> Server (active checks, trapper) |
| Frontend | 80/443 | Browser -> Web UI |

## Trigger Expression Quick Reference

```
avg(/host/system.cpu.load[all,avg1],5m) > 5           # CPU high for 5 min
last(/host/vfs.fs.size[/,pused]) > 90                  # Disk > 90%
last(/host/proc.num[nginx]) = 0                        # Service down
nodata(/host/agent.ping,5m) = 1                        # Agent unreachable
max(/host/vm.memory.size[available],15m) < 104857600   # Memory < 100MB
find(/host/log[/var/log/app.log],60s,"regexp","FATAL|ERROR") = 1
```

**Severity levels:** Not classified (gray), Information (blue), Warning (yellow), Average (orange), High (orange-red), Disaster (red).

**Hysteresis:** Separate problem and recovery expressions prevent flapping. Alert at CPU > 90%, recover at CPU < 80%.

## Low-Level Discovery (LLD) Quick Reference

LLD automatically discovers dynamic entities (filesystems, interfaces, databases) and creates items/triggers from prototypes.

**Built-in discovery keys:**
- `vfs.fs.discovery` -- Filesystems (`{#FSNAME}`, `{#FSTYPE}`)
- `net.if.discovery` -- Network interfaces (`{#IFNAME}`)
- `vfs.dev.discovery` -- Block devices
- `system.cpu.discovery` -- CPU cores

**Item prototype example:**
- Key: `vfs.fs.size[{#FSNAME},pused]`
- Name: `Filesystem {#FSNAME}: Used space in %`

**Filters:** Restrict discovered entities (e.g., `{#FSTYPE}` matches `ext4|xfs|btrfs`).
**Overrides:** Per-entity customization without modifying prototypes (e.g., disable `/tmp` disk-full trigger).

## Top 10 Operational Rules

1. **Use templates for everything** -- Never configure items/triggers directly on hosts. Templates enable reuse, git versioning, and consistent monitoring.
2. **Deploy Agent2 on new hosts** -- Plugin architecture provides native database, Redis, Docker, and Kubernetes monitoring without custom scripts.
3. **Use dependent items aggressively** -- One master HTTP call, many extracted metrics. Reduces polling overhead and API calls.
4. **Tag all triggers** -- Use `component`, `scope`, `service` tags for alert routing, dashboard filtering, and event correlation.
5. **Deploy proxies by network zone** -- DMZ, remote office, cloud VPC. Active proxy mode reduces firewall complexity.
6. **Use TimescaleDB for PostgreSQL** -- Hypertables with compression achieve 5-10x compression on history data and improve write throughput.
7. **Monitor Zabbix itself** -- Use `zabbix[*]` internal items to track queue depths, process busy %, and cache hit ratios.
8. **Use proxy groups (7.0+)** -- Multiple proxies in a group for automatic failover and load distribution.
9. **Export templates as YAML to git** -- Version control templates. Use CI/CD to import to staging before production.
10. **Tune housekeeping** -- For large deployments, disable built-in housekeeper and use TimescaleDB retention policies or PostgreSQL partitioning instead.

## Common Pitfalls

**1. Database growth without TimescaleDB**
History tables grow unbounded. Without TimescaleDB compression, a 10,000-host deployment can consume hundreds of GB within months. TimescaleDB compression reduces storage 5-10x.

**2. All items polling at :00 seconds**
Default intervals cause all items to fire simultaneously, creating polling spikes. Use randomized delays or offset intervals to distribute load.

**3. Missing trigger dependencies**
Without dependencies, a network switch failure generates alerts for every host behind it. Set upstream triggers as dependencies to suppress downstream alert storms.

**4. Not using discovery filters**
LLD without filters creates items for tmpfs, devtmpfs, loopback, and virtual interfaces. Always filter discovery results to meaningful entities.

**5. Classic agent when Agent2 is available**
Classic agent requires UserParameters and external scripts for database and container monitoring. Agent2 has native plugins for these.

**6. Passive proxy mode in firewalled environments**
Passive proxy requires the server to initiate connections to the proxy. Active proxy mode is preferred since only the proxy needs outbound access.

## Reference Files

Load these when you need deep knowledge for a specific area:

- `references/architecture.md` -- Server, proxy (active/passive, proxy groups), Agent vs Agent2, frontend, database backends (PostgreSQL + TimescaleDB, MySQL), monitoring methods (SNMP, IPMI, HTTP, JMX, trapper, Prometheus), HA cluster, security hardening. Read for deployment and architecture questions.
- `references/features.md` -- Templates (inheritance, groups, export/import), items (types, keys, preprocessing, history/trends), triggers (expressions, severity, dependencies, correlation), LLD (discovery rules, prototypes, filters, overrides, lifetime), dashboards (widgets, network maps, host groups, SLA). Read for configuration and feature questions.
- `references/diagnostics.md` -- Performance tuning (server processes, cache sizes, item intervals, database tuning), housekeeping, monitoring scale reference, common issues, maintenance windows, internal monitoring. Read for troubleshooting and optimization.