Grafana

---
name: grafana
description: "Grafana for analytics and BI-style dashboarding: dashboard design, data source configuration, alerting, provisioning as code, plugin management, and the LGTM stack, from the dashboarding/reporting angle. Use for building or troubleshooting Grafana dashboards, panels, alerting rules, or provisioning. WHEN: \"Grafana\", \"Grafana dashboard\", \"Grafana panel\", \"Grafana alerting\", \"Grafana data source\", \"Grafana Cloud\", \"Grafana provisioning\", \"Grafana Terraform\", \"PromQL\", \"LogQL\", \"TraceQL\", \"Loki\", \"Tempo\", \"Mimir\", \"Grafana Alloy\", \"LGTM stack\", \"Grafana plugin\", \"Grafana variable\", \"grafana-cli\", \"Grafana alert rule\", \"notification policy\", \"contact point\", \"mute timing\", \"recording rule\", \"Grafana embed\". Do NOT use for Grafana as part of an observability/monitoring stack (SLOs, on-call alert routing, infra metrics strategy) -- that's the `grafana` skill in the monitoring plugin."
license: MIT
---

# Grafana for Analytics

This skill covers Grafana as a dashboarding and visualization tool within the analytics domain -- BI-style panels, provisioning, and alerting mechanics. For Grafana in infrastructure monitoring and observability strategy contexts (SLOs, alert routing philosophy, the broader metrics/logs/traces stack), see the `grafana` skill in the monitoring plugin.

This skill covers:

- Dashboard design (panels, rows, tabs, variables, annotations, conditional rendering, auto grid layouts)
- Data source configuration (Prometheus, Loki, Tempo, Mimir, InfluxDB, Elasticsearch, SQL databases, cloud providers)
- Query languages (PromQL, LogQL, TraceQL, SQL Expressions)
- Alerting engine (alert rules, notification policies, contact points, silences, mute timings, recording rules)
- Provisioning as code (YAML file-based, Terraform provider, Kubernetes Operator, Crossplane, Git Sync)
- Plugin ecosystem (data source, panel, and app plugins; plugin catalog; custom plugin development)
- LGTM stack (Loki for logs, Grafana for visualization, Tempo for traces, Mimir for metrics, Alloy for collection)
- Deployment tiers (OSS/AGPL, Enterprise, Cloud) and architecture (single instance, HA, Kubernetes)
- Embedded analytics and reporting

Grafana follows a continuous release model. The current major version is **Grafana 12.x** (12.0 released May 2025, latest 12.4 as of February 2026). There are no version-specific references -- guidance applies to the current platform.

## How to Approach Tasks

1. **Classify** the request:
   - **Dashboard design / visualization** -- Load `references/best-practices.md` for layout patterns, panel selection, variable usage, conditional rendering
   - **Data source / query optimization** -- Load `references/architecture.md` for data source architecture, query engine internals, and `references/best-practices.md` for PromQL/LogQL optimization
   - **Alerting** -- Load `references/architecture.md` for alerting engine architecture, `references/best-practices.md` for alert rule design and notification routing
   - **Performance / troubleshooting** -- Load `references/diagnostics.md` for slow dashboard diagnosis, data source errors, alerting failures, resource usage
   - **Provisioning / IaC / deployment** -- Load `references/architecture.md` for provisioning methods and deployment architecture, `references/best-practices.md` for Terraform and Git Sync patterns
   - **LGTM stack** -- Load `references/architecture.md` for Loki, Tempo, Mimir, Alloy architecture and cross-signal correlation

2. **Determine scope** -- Identify whether the question is about Grafana OSS, Enterprise, or Cloud. Feature availability differs (e.g., query caching, SAML, reporting, enterprise data sources are Enterprise/Cloud only). Also determine if the question involves the broader LGTM stack or Grafana alone.

3. **Load context** -- Read the relevant reference file for deep technical detail.

4. **Analyze** -- Apply Grafana-specific reasoning. Consider data source capabilities, query language idioms, alerting architecture, and deployment tier constraints.

5. **Recommend** -- Provide actionable guidance with query examples, YAML provisioning snippets, Terraform resources, or `grafana.ini` configuration.

6. **Verify** -- Suggest validation steps (Query Inspector, browser DevTools Network tab, `/metrics` endpoint, `grafana.log`, Grafana Advisor).

## Ecosystem

```
┌───────────────────────────────────────────────────────────────┐
│                     LGTM Observability Stack                  │
│  ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌────────────────┐  │
│  │  Mimir   │ │   Loki   │ │  Tempo   │ │  Grafana Alloy │  │
│  │ (Metrics)│ │  (Logs)  │ │ (Traces) │ │  (Collector)   │  │
│  └─────┬────┘ └────┬─────┘ └────┬─────┘ └───────┬────────┘  │
│        │           │            │                │            │
│        └───────────┴─────┬──────┴────────────────┘            │
│                    ┌─────▼─────┐                              │
│                    │  Grafana  │  Dashboards / Alerting        │
│                    └─────┬─────┘                              │
│         ┌────────────────┼────────────────┐                   │
│  ┌──────▼──────┐  ┌──────▼──────┐  ┌──────▼──────┐          │
│  │  Dashboards │  │   Alerting  │  │  Drilldown  │          │
│  │  (Panels)   │  │   Engine    │  │   Apps      │          │
│  └─────────────┘  └─────────────┘  └─────────────┘          │
└───────────────────────────────────────────────────────────────┘
         │                    │                │
  ┌──────▼──────┐     ┌──────▼──────┐  ┌──────▼──────┐
  │   Web UI    │     │  Terraform  │  │  Kubernetes │
  │   Browser   │     │  Provider   │  │  Operator   │
  └─────────────┘     └─────────────┘  └─────────────┘
```

| Component | Purpose | Deployment |
|---|---|---|
| **Grafana** | Dashboard visualization, alerting, query interface | Self-hosted or Cloud |
| **Mimir** | Long-term, horizontally scalable metrics storage (PromQL-compatible) | Self-hosted or Cloud |
| **Loki** | Horizontally scalable log aggregation (LogQL) | Self-hosted or Cloud |
| **Tempo** | Distributed tracing backend (TraceQL) | Self-hosted or Cloud |
| **Grafana Alloy** | OpenTelemetry-compatible telemetry collector (replaces Grafana Agent) | Self-hosted |
| **Grafana Cloud** | Fully managed SaaS for entire LGTM stack | Managed SaaS |

## Dashboard Architecture

Dashboards are JSON-based collections of panels arranged in rows, with support for variables, annotations, and drilldown navigation.

**Core elements:**
- **Panels** -- Individual visualizations (time series, stat, gauge, bar chart, table, heatmap, geomap, logs, traces, canvas, node graph, etc.)
- **Rows** -- Collapsible containers grouping related panels
- **Tabs** (12.0+) -- Segment dashboard content by context without creating multiple dashboards
- **Variables** -- Template variables enabling dynamic, reusable dashboards (`$env`, `$service`, `$instance`)
- **Annotations** -- Event markers on time-series graphs (deployments, incidents, config changes)
- **Dashboard links** -- Navigation between related dashboards; drilldown workflows
- **Conditional rendering** (12.0+) -- Show/hide panels and rows based on variable selections or data availability
- **Auto grid layout** (12.4+) -- Flexible, responsive panel arrangement

**Panel features:**
- Query Inspector for debugging query performance and response payloads
- Transformations pipeline (filter, join, aggregate, rename, calculate, regression analysis)
- Thresholds and value mappings for color-coded status indication
- Panel overrides for field-level styling
- Visualization actions with custom variables (12.1+)

## Data Sources

Data sources are backend plugins that translate Grafana queries into source-specific query languages.

**Built-in data sources:**
- Prometheus / Mimir (PromQL), Loki (LogQL), Tempo (TraceQL)
- InfluxDB, Graphite, Elasticsearch/OpenSearch
- MySQL, PostgreSQL, Microsoft SQL Server
- CloudWatch, Azure Monitor, Google Cloud Monitoring
- Jaeger, Zipkin (distributed tracing)

**Enterprise/Cloud-only:** Splunk, Oracle, ServiceNow, SAP HANA, Snowflake, Databricks, MongoDB

**Key architecture points:**
- Queries route through the Grafana server (server-side proxy), preventing direct browser-to-database connections
- Mixed data source mode combines queries from multiple sources in a single panel
- SQL Expressions (12.0+) join and combine data from multiple sources using SQL syntax

## Alerting

Grafana Alerting is built on the Prometheus alerting model with unified multi-data-source support.

**Alert rule types:**
- **Grafana-managed rules** -- Evaluated by the Grafana alerting engine; support any data source
- **Data source-managed rules** -- Evaluated by compatible backends (Prometheus/Mimir/Loki ruler)
- **Recording rules** -- Pre-compute expensive queries for faster dashboard loading

**Notification pipeline:**
1. Alert rules define query + condition + `for` duration
2. Notification policies route alerts via label matchers (tree-structured, top-down matching)
3. Contact points deliver notifications (Slack, PagerDuty, OpsGenie, Teams, email, webhooks, Telegram, Discord)
4. Silences suppress one-time; mute timings suppress recurring (nights, weekends, maintenance)
5. Grouping combines related alert instances (`group_by`, `group_wait`, `group_interval`, `repeat_interval`)

**No Data / Error handling:** Configure explicitly per rule -- `Alerting`, `NoData`, `OK`, or `KeepLastState`.

## Provisioning

Grafana supports multiple infrastructure-as-code approaches:

| Method | Scope | Best For |
|---|---|---|
| **YAML file-based** | Data sources, dashboards, alerting, plugins | Simple deployments, Docker Compose |
| **Terraform provider** | All Grafana resources (dashboards, data sources, folders, alerts, orgs) | Multi-environment IaC |
| **Kubernetes Operator** | Grafana instances + resources via CRDs | Kubernetes-native deployments |
| **Crossplane** (alpha) | Kubernetes manifests for Terraform resources | GitOps with ArgoCD/Flux |
| **Git Sync** (12.0+) | Dashboard version control with PR-based workflows | Collaborative dashboard development |
| **REST API** | Programmatic management of all resources | CI/CD pipelines, automation |

**File-based provisioning structure:**
```
/etc/grafana/provisioning/
  datasources/
    datasource.yaml
  dashboards/
    dashboard-provider.yaml
  alerting/
    alert-rules.yaml
    contact-points.yaml
    notification-policies.yaml
    mute-timings.yaml
  plugins/
    plugin.yaml
```

Environment variable substitution: `$ENV_VAR` or `${ENV_VAR}` in provisioning YAML files.

## Deployment Tiers

| Feature | OSS (AGPL) | Enterprise | Cloud |
|---|---|---|---|
| Core dashboards, alerting, plugins | Yes | Yes | Yes |
| Enterprise data sources (Splunk, Oracle, etc.) | No | Yes | Yes |
| SAML / Team Sync / SCIM | No | Yes | Yes |
| Query caching (built-in) | No | Yes | Yes |
| Reporting (scheduled PDF/CSV) | No | Yes | Yes |
| Audit logging | No | Yes | Yes |
| Data source permissions (fine-grained) | No | Yes | Yes |
| White labeling / custom branding | No | Yes | Yes |
| Managed LGTM stack | No | No | Yes |
| Adaptive Metrics / Asserts | No | No | Yes |
| Synthetic Monitoring / Frontend Observability | No | No | Yes |
| Uptime SLA | Self-managed | Self-managed | 99.5% |

### Grafana Cloud Pricing

| Tier | Cost | Included |
|---|---|---|
| **Free** | $0 | 10K metrics series, 50 GB logs/traces/profiles, 3 users, 2-week retention |
| **Pro** | $19/mo + usage | Extended retention (13-month metrics, 30-day logs/traces), support, all features |
| **Advanced/Enterprise** | Custom | SLAs, dedicated support, advanced security |

## Deployment Architecture

| Pattern | Users | Database | Notes |
|---|---|---|---|
| **Single instance** | 5-10 | Embedded SQLite | 200-500 MB memory baseline |
| **High availability** | 10-200+ | PostgreSQL or MySQL | Load balancer, shared storage, session affinity |
| **Kubernetes** | 10-200+ | PostgreSQL | Helm charts, Operator, HPA, ConfigMaps/Secrets |
| **Docker Compose** | 5-50 | PostgreSQL | Development and small production |

**For production:** Use PostgreSQL or MySQL (not SQLite). Multiple Grafana instances behind a load balancer with shared session storage (Redis/Memcached).

## Grafana 12.x Key Features

| Version | Highlights |
|---|---|
| **12.0** (May 2025) | Tabs, conditional rendering, Git Sync (preview), SQL Expressions, Drilldown apps GA, table rebuilt (97.8% faster), Cloud migration GA, SCIM preview |
| **12.1** (Jul 2025) | New alert rule page GA, regression analysis transformation, Grafana Advisor GA, visualization actions with custom variables |
| **12.2** (Sep 2025) | Enhanced ad hoc filtering GA, redesigned table GA, AI-powered SQL expressions (preview), Metrics Drilldown with Alert Integration GA |
| **12.3** (Nov 2025) | Redesigned logs panel, SolarWinds/Honeycomb/OpenSearch data sources, dashboard sharing improvements |
| **12.4** (Feb 2026) | Git Sync GitHub App auth, auto grid layout, suggested dashboards (preview), dashboard templates (DORA metrics), revamped gauge, SCIM GA, RBAC for saved queries |

## LGTM Stack

The LGTM stack provides unified observability across metrics, logs, and traces.

### Cross-Signal Correlation

- Tempo links to Loki (`tracesToLogsV2`) and Mimir (`tracesToMetrics`)
- Loki links to Tempo via derived fields (extract trace IDs from log entries)
- Exemplars link metric data points to specific traces
- Drilldown apps (GA in 12.0) provide code-free navigation between signals

### Grafana Alloy

Replaces Grafana Agent as the recommended telemetry collector:
- OpenTelemetry-compatible pipeline for metrics, logs, and traces
- Declarative, component-based configuration
- Supports Prometheus scraping, OTLP ingestion, and various receivers

### Performance Benchmarks (GrafanaCON 2025)

LGTM stack query P99: 85ms vs ELK stack at 650ms (7x faster).

## Anti-Patterns

| Anti-Pattern | Problem | Correct Approach |
|---|---|---|
| 30+ panels on a single dashboard | Cumulative lag from parallel data fetches; browser unresponsive | Split into focused dashboards or use tabs; keep overview dashboards to 10-15 panels |
| 5-second auto-refresh on complex dashboards | Constant query load on data sources and Grafana server | 30s-60s for operational dashboards; 5m-15m for trends and capacity |
| `{__name__=~".+"}` catch-all PromQL | Scans all metrics; overwhelms Prometheus/Mimir | Use specific metric names and label selectors |
| Querying months of high-resolution data | Massive data transfer; slow rendering | Use per-panel time range overrides; recording rules for long-term trends |
| Per-pod metrics without aggregation | Thousands of time series per query (high cardinality) | Aggregate: `sum by (service)` instead of per-instance |
| SQLite in production | File-level locking under concurrent load; corruption risk | PostgreSQL or MySQL for production |
| Unsigned plugins in production | Security risk from unverified code | Only install signed plugins; use `allow_loading_unsigned_plugins` for dev only |
| Forgetting No Data / Error state config | False `DatasourceError` or `NoData` alerts | Configure `nodata_state` and `error_state` explicitly per alert rule |
| Mute timings expected to inherit | Mute timings are NOT inherited from parent notification policies | Apply mute timings at each relevant policy level |
| No self-monitoring | Grafana performance degrades silently | Scrape `/metrics` endpoint; build self-monitoring dashboard; alert on resource usage |

## Cross-References

- `overview` skill, this plugin -- technology comparison and selection guidance across BI tools
- `grafana` skill, monitoring plugin -- Grafana for infrastructure monitoring and observability use cases

## Reference Files

- `references/architecture.md` -- Dashboard internals, data source architecture, panels, alerting engine, provisioning methods, plugin system, deployment tiers, LGTM stack, cross-signal correlation
- `references/best-practices.md` -- Dashboard design (Z-pattern, RED method, hierarchy), query optimization (PromQL/LogQL), alerting rule design, notification policies, provisioning as code (Terraform, Git Sync, Kubernetes), plugin management
- `references/diagnostics.md` -- Slow dashboard diagnosis (Query Inspector, network tab, server metrics, logs), data source error troubleshooting, alerting failure investigation, resource usage monitoring, self-monitoring setup

## Diagnostic Scripts

Ready-made HTTP API scripts (bash + curl + jq, service-account token) in `scripts/`, numbered by investigation order. All read-only; placeholders are `__UPPER_SNAKE__` tokens.

- `scripts/01-health-and-datasources.sh` -- Instance health plus per-datasource connectivity checks
- `scripts/02-dashboard-inventory.sh` -- Dashboard inventory by folder with stale-content detection
- `scripts/03-alert-rules-status.sh` -- Alert rule health (error/nodata) and currently firing alerts
- `scripts/04-user-access-audit.sh` -- Users by role, last-seen age, service accounts (privilege creep)