Grafana

---
name: grafana
description: "Covers Grafana 12.x for observability and metrics dashboards: dashboard design, panel types, variables, transformations, provisioning, Unified Alerting, Loki/LogQL, Tempo/TraceQL, data source configuration, RBAC, and performance optimization. Use for \"Grafana\", \"Grafana dashboard\", \"Grafana alerting\", \"Loki\", \"LogQL\", \"Tempo\", \"TraceQL\", \"Grafana panel\", \"Grafana variable\", \"Grafana provisioning\", \"Grafana Cloud\", \"dashboard JSON\", \"Grafana data source\", \"Grafana transformation\", \"Grafana annotation\", \"Grafana RBAC\". Do NOT use for BI/reporting dashboards built on Grafana for business metrics — that's the `grafana` skill in the `analytics` plugin."
license: MIT
---

# Grafana Technology Expert

This skill covers Grafana 12.x: dashboard design, visualization, alerting, Loki log management, Tempo distributed tracing, provisioning, and the LGTM observability stack. Every recommendation addresses the tradeoff between **dashboard usability**, **query performance**, and **operational maintainability**.

## How to Approach Tasks

When you receive a request:

1. **Classify** the request by area:
   - **Architecture** (server, data sources, provisioning, Cloud vs self-hosted) -- Load `references/architecture.md`
   - **Dashboards** (panels, variables, transformations, annotations, JSON model) -- Load `references/dashboards.md`
   - **Alerting** (rules, contact points, notification policies, silences) -- Load `references/alerting.md`
   - **Loki / Tempo** (LogQL, TraceQL, log-to-trace correlation) -- Load `references/loki-tempo.md`

2. **Design for progressive drill-down** -- Dashboards should follow the L1 (overview) > L2 (service) > L3 (diagnostic) hierarchy. Use data links and panel links to connect them.

3. **Recommend provisioning as code** -- Manual dashboard edits in the UI are fragile. Always suggest storing dashboards and data sources as provisioning files in Git.

4. **Include performance context** -- Dashboards with 50+ panels or high-cardinality variables degrade browser performance. Recommend recording rules, sensible time ranges, and panel limits.

## Core Expertise

- **Architecture:** Single Go binary, React frontend, plugin system, database backend (SQLite/MySQL/PostgreSQL), authentication (OAuth2/OIDC, SAML, LDAP, SCIM), versioned API model (12.x), Git Sync (observability as code)
- **Dashboards:** 20+ panel types (time series, stat, gauge, bar, table, heatmap, logs, traces, node graph, geomap, canvas), variables (query, custom, interval, ad hoc), transformations (merge, filter, group by, join, calculate), annotations, dashboard links, data links, JSON model
- **Alerting:** Unified Alerting (Grafana-managed and data source-managed rules), contact points (Slack, PagerDuty, email, webhook, Teams, OpsGenie), notification policies (routing tree), silences, mute timings, active time intervals
- **Loki:** LogQL (stream selectors, filter expressions, parsers, metric queries), derived fields for trace correlation, log volume histograms, live tail
- **Tempo:** TraceQL (span selectors, structural operators, TraceQL metrics), trace-to-log and trace-to-metric correlation, service maps, exemplars
- **12.x Features:** Git Sync, conditional rendering, auto-grid layout (Dynamic Dashboards), tabs, dashboard outline, schema v2, SQL Expressions (private preview), table refactor (97.8% faster)

## Dashboard Design Quick Reference

### Monitoring Methodologies

| Method | Signals | Best For |
|--------|---------|----------|
| **USE** | Utilization, Saturation, Errors | Infrastructure (CPU, memory, disk, network) |
| **RED** | Rate, Errors, Duration | Services (API, microservice) |
| **4 Golden Signals** | Latency, Traffic, Errors, Saturation | SRE teams, SLO tracking |

### Dashboard Hierarchy

| Level | Purpose | Panels | Audience |
|-------|---------|--------|----------|
| **L1 Overview** | All services health | 8-12 stat/gauge panels | On-call, management |
| **L2 Service** | One service RED/USE metrics | 15-20 time series panels | On-call engineer |
| **L3 Diagnostic** | Detailed metrics + logs + traces | 20-30 mixed panels | Debugging engineer |

### Design Rules

- Target 20-30 panels maximum per dashboard
- Use thresholds with consistent colors: green (normal), yellow (warning), red (critical)
- Always specify Y-axis units (requests/s, ms, bytes)
- Use repeat panels (by variable) rather than duplicating manually
- Set panel descriptions explaining what the metric means and what action to take
- Use dashboard tags for discovery (service name, team, environment)
- Set default time range appropriate to the use case (1h for ops, 24h for capacity)
- Set refresh rate to 30s or longer (avoid < 30s)
- Store dashboards in version control; avoid manual-only edits in production

## Top 10 Operational Rules

1. **Provision everything as code** -- Data sources, dashboards, and alert rules in YAML/JSON under Git. Use Grafana 12 Git Sync for bidirectional sync.

2. **Use recording rules for expensive queries** -- Pre-compute `rate()` and `histogram_quantile()` in Prometheus/Mimir. Query recording rules in Grafana.

3. **Set `Min interval` in panel query options** -- Align with Prometheus scrape interval to avoid unnecessary data points.

4. **Use variables for drill-down** -- Chain variables (`namespace` feeds `deployment` feeds `pod`) for progressive filtering.

5. **Connect dashboards with data links** -- Click a data point to navigate to related dashboards, Loki Explore, or external systems.

6. **Configure derived fields in Loki** -- Extract trace IDs from log lines to link directly to Tempo traces.

7. **Use mute timings, not silences** -- Mute timings are recurring and policy-based. Silences are one-time and easily forgotten.

8. **Limit auto-refresh and variable options** -- Auto-refresh < 30s and variables with thousands of options degrade performance.

9. **Organize by folder with RBAC** -- Assign permissions at the folder level (Platform, Services, SLOs). Use naming conventions for discoverability.

10. **Test alert rules in the editor** -- Use the "Test rule" button to see evaluation results before saving. Verify routing with notification policy view.

## Common Pitfalls

**1. Too many panels per dashboard**
Browsers struggle with 50+ panels querying simultaneously. Split into multiple dashboards connected by drill-down links.

**2. High-cardinality variables**
Variables generating thousands of options (e.g., all pod names across all namespaces) slow the dashboard and the data source. Scope variables with chaining.

**3. Manual-only dashboards in production**
UI edits are not version-controlled and cannot be reproduced after data loss. Use provisioning or Git Sync.

**4. Missing `editable: false` on provisioned resources**
Without this, users modify provisioned dashboards in the UI, creating drift from the source of truth.

**5. NoData alerts firing unexpectedly**
Default NoData behavior varies. Set the NoData state to `Normal` if gaps are expected (e.g., scrape interval misalignment).

**6. Not using Go templates in contact points**
Default notification messages lack context. Customize with Go templates to include alert details, runbook URLs, and relevant labels.

## Reference Files

Load these when you need deep knowledge for a specific area:

- `references/architecture.md` -- Grafana server components, data source plugins, provisioning (YAML layout, data source, dashboard provider, alerting), Git Sync, Cloud vs self-hosted comparison. Read for architecture and setup questions.
- `references/dashboards.md` -- Panel types (20+ types with best-use guidance), variables (query, custom, interval, ad hoc, chaining), transformations (merge, filter, group by, join, calculate), annotations, dashboard links, data links, JSON model structure. Read for dashboard design.
- `references/alerting.md` -- Unified Alerting (rules, folders, groups), alert state lifecycle, contact points (12+ integrations), notification policies (routing tree, grouping, timings), silences, mute timings, active time intervals. Read for alerting questions.
- `references/loki-tempo.md` -- Loki data source configuration, LogQL (stream selectors, filter expressions, parsers, metric queries), Tempo data source configuration, TraceQL (span selectors, structural operators, metrics), trace correlations (trace-to-log, trace-to-metric, log-to-trace, exemplars). Read for log/trace questions.