Grafana Architect

---
name: grafana-architect
version: 1.0.0
description: Grafana dashboards + alerts — dashboards-as-code (Grizzly), per-service folders, one-question-per-panel, unified alerting with runbooks, low-cardinality discipline. Use when designing dashboards, writing alert rules, or auditing.
---

# Grafana Architecture — Signal Consumption

How an operator *uses* what [observability-architect](../observability-architect/SKILL.md) emits. Dashboards, alerts, data sources, exploration. **Dashboards-as-code via Grizzly** is the default; the Grafana UI is for exploration, not authoring. Layouts, panel reference, and alert YAML in [RECIPES.md](RECIPES.md).

## 1. Dashboards-as-code workflow

Dashboards live in the application repo (or a sibling `observability/` repo), versioned in git, applied via [Grizzly](https://grafana.github.io/grizzly/) (`grr`). Repo layout in [RECIPES § 1](RECIPES.md#1-observability-repo-layout).

- **Sync on merge to main** — staging on every push, prod on tagged releases. Dashboards drift only via reviewed PRs.
- **Never edit in the UI for production dashboards.** Grizzly refuses to apply on drift; resolve by pulling the change into JSON.
- **Exploration is different.** Build in the UI; once it earns a home, export and commit.

## 2. Folder organization — per service

One folder per service. Operators on call find dashboards by what they own, not by which team made them (teams reshuffle; services don't). Folder structure example in [RECIPES § 2](RECIPES.md#2-service-folder-layout).

- **Folder permissions match service ownership.**
- **General / platform dashboards** live in a top-level `Platform` folder owned by SRE.
- **No personal folders for production.**

## 3. Panel design — one question per panel

Every panel answers exactly one question. If the title is "stuff," redesign.

- **Panel title is a question or noun phrase:** `"Request rate (req/s)"`, `"p95 latency by route"`. Not `"Stats"`.
- **Y-axis unit is mandatory** — `seconds`, `bytes`, `req/s`, `percent`. Auto-formatting hides confusion.
- **Time range is consistent across the dashboard** unless the dashboard's purpose is the comparison.
- **Thresholds where they exist.** SLO at 99.9% gets a red line at 99.9%.
- **Legends are bounded.** Templatize the query (`sum by (route)`) to bound it. High natural cardinality → heatmap or top-N.

Panel-type reference (time series / stat / gauge / heatmap / bar / logs / traces / text) in [RECIPES § 3](RECIPES.md#3-panel-type-reference).

## 4. Variables and templating

Variables let one dashboard serve many slices.

- **Standard variables** on every service dashboard: `service`, `environment` (`prod`/`staging`), `region` if multi-region.
- **Variables come from label queries**, not hand-maintained lists: `label_values(up{job="$service"}, environment)`.
- **`All` is dangerous** on high-cardinality metrics — fans out into millions of series. Disable or restrict to a curated regex.
- **Don't split dashboards on a variable.** `Orders Prod` and `Orders Staging` are one dashboard with an `environment` variable.

## 5. Alerting — Grafana unified alerting

Alerts live alongside dashboards in Grizzly. A panel and its alert rule are version-controlled together. Full Grizzly YAML in [RECIPES § 4](RECIPES.md#4-grizzly-alert-rule).

### Alert hygiene

- **Every alert is actionable.** If the response is "I'll look in the morning," it's a dashboard, not an alert.
- **Two-tier severity:** `severity=page` (wakes someone) and `severity=ticket` (queues for next business day). No middle ground.
- **Multi-window, multi-burn-rate for SLOs:** `fast_burn = (2% in 1h)` AND `slow_burn = (5% in 6h)` together. Catches real issues without flapping.
- **`for:` is mandatory.** Minimum `2m` on noisy signals; `5m`+ for slow burns.
- **Annotations include a `runbook_url`.** First thing on-call clicks; if it doesn't exist, the alert is half-built.
- **Group related alerts** under labels (`team`, `service`) so silencing can target a service.

### Alerts NOT to write

- **CPU > 80%.** Useless without context. Use saturation (run queue, GC) or RED (latency/errors) instead.
- **"Disk full" without `for:`** — needs `for: 15m` so log rotation spikes don't page.
- **Per-instance alerts on horizontally scaled services.** Fleet-level signal; one bad pod isn't a page.
- **Static thresholds on metrics that grow with traffic.** Use rate-of-change or anomaly instead.

## 6. Data sources

- **Provisioned via Grizzly**, not the UI. Datasource `.yaml` lives next to dashboards.
- **One data source per signal type per environment:** `prometheus-prod`, `prometheus-staging`, `loki-prod`, `tempo-prod`.
- **Service account API keys**, not personal tokens. Rotate quarterly.
- **Read-only data sources for dashboards.** Write access (Alertmanager mute rules, etc.) goes through dedicated service accounts with audit logging.

## 7. Permissions

- **Editor at the folder level.** A service team gets `Editor` on `Orders/`; everyone else has `Viewer`.
- **Admin sparingly.** Two or three admins per Grafana instance.
- **`Anonymous` access off** in any environment with non-public data — including staging.
- **SSO (SAML/OIDC) for humans**, service accounts for automation. No shared passwords.

## 8. Common pitfalls

Sprawl, unactionable alerts, raw-label dashboards, unit mixups, static thresholds, UI drift, panel overload — full list with diagnostics + fixes in [RECIPES § 5](RECIPES.md#5-common-pitfalls).

## 9. SLO dashboards — the special case

Every service has exactly one **SLO dashboard** showing:

- **Current SLO compliance** — percentage over the rolling window (last 30 days).
- **Error budget remaining** — `(SLO_target - current_failure_rate) * total_requests` as a budget number.
- **Burn rate** — instantaneous burn rate, with fast/slow window thresholds visible as red lines.
- **Top contributors** — table of endpoints/operations driving the failure rate.

This dashboard is the single source of truth during an incident. Link to it from the runbook, the page, and the post-incident review.

## 10. Cross-skill ties

- [observability-architect](../observability-architect/SKILL.md) — produces what Grafana consumes. Naming and cardinality discipline established there must hold for queries here to work.
- [docker-architect §10](../docker-architect/SKILL.md#10-vulnerability-scanning--trivy) — Trivy scan results can be dashboards too (security metric over time).
- [rest-api-architect](../../protocols/rest-api-architect/SKILL.md) — SLO burn rate alerts reference REST status codes (`5xx` → error budget) and latency.
- [improve-codebase-architecture](../../refactoring/improve-codebase-architecture/SKILL.md) — friction visible in dashboards is input to architecture review.
- [grpc-architect §2](../../protocols/grpc-architect/SKILL.md) — gRPC status codes feed equivalent metrics; same SLO mechanics apply.