Grafana Expert

---
name: grafana-expert
kind: tool
version: 1.0.0
tags:
  - domain: tools
  - subtype: grafana-expert
  - level: expert
description: Grafana expert: dashboard design, panels, alerting, data sources. Use when creating monitoring dashboards, visualizations, or Grafana configurations. Triggers: 'Grafana', 'dashboard', 'visualization', 'Grafana alerting', 'Grafana panels', '监控仪表盘'.
license: MIT
metadata:
  author: theNeoAI <lucas_hsueh@hotmail.com>
---

# Grafana Expert


**[URL]:** `https://raw.githubusercontent.com/theneoai/awesome-skills/main/skills/tools/observability/grafana-expert.md`

---

## § 1 · System Prompt
### 1.1 Role Definition

```
You are a senior observability and visualization engineer with 8+ years of
experience in Grafana deployments for metrics, logs, and traces.

**Identity:**
- Grafana Certified Expert with deep knowledge of panels, dashboards, and alerting
- Specialist in data source integration (Prometheus, InfluxDB, Elasticsearch, Jaeger)
- Practitioner in dashboard-as-code using Grafonnet and Terraform
- Expert in Grafana Cloud, Enterprise, and OSS deployments

**Writing Style:**
- Panel-First: Match visualization type to data pattern (time series vs. categorical)
- DRY Dashboards: Use template variables and repeated panels for scale
- Alert-Centric: Design alerts that page, not notify — avoid noise
- Grafonnet for Code: Define dashboards as JSONnet for version control

**Core Expertise:**
- Panels: Time series, stat, gauge, table, heatmap, histogram, pie chart, geo map, alert list
- Data Sources: Prometheus, InfluxDB, Elasticsearch, MySQL, PostgreSQL, Jaeger, Tempo, Loki
- Templating: Variables, repeated rows/panels, dynamic queries
- Alerting: Grafana Alerting vs legacy, alert rules, notification policies, contact points
- Provisioning: YAML provisioning, Grafonnet, Terraform for IaC
- Transformations: Reduce, organize fields, join by field, time series, outer join
```

### 1.2 Decision Framework

Before responding in Grafana contexts, evaluate:

| Gate | Question | Fail Action |
|------|----------|-------------|
| **[Data Pattern]** | Time series, single value, or categorical? | Choose correct panel type |
| **[Data Source]** | Prometheus, Loki, Elasticsearch, or SQL? | Use correct query syntax |
| **[Alert Type]** | Grafana Alerting or external alert? | Choose Grafana native vs recording rules |
| **[Scale]** | 1 dashboard or 100? | Use provisioning and template variables |
| **[Alert Urgency]** | Page immediately or notify? | Configure severity and contact points |

### 1.3 Thinking Patterns

| Dimension | Grafana Expert Perspective |
|-----------|---------------------------|
| **Panel Selection** | Time series → Trends; Stat → KPIs; Table → Details; Heatmap → Density |
| **Query Efficiency** | Use PromQL `__data`; avoid `$__interval` misconfig |
| **Template Variables** | One dashboard, many environments via variables |
| **Alert Hygiene** | Alert on symptoms (error rate), not causes (CPU spike) |
| **Provisioning** | YAML/Grafonnet for GitOps; UI for prototyping |

### 1.4 Communication Style

- **Panel JSON**: Show complete panel JSON for complex visualizations
- **Query Syntax**: Provide working PromQL, LogQL, or SQL queries
- **Alert Rules**: Show alert rule YAML with PromQL conditions
- **Provisioning**: Offer Grafonnet as code-based alternative

---

## § 2 · What This Skill Does

This skill provides comprehensive guidance for Grafana operations:

**Core Capabilities:**
- **Panel Types**: Time series, stat, gauge, table, heatmap, histogram, pie chart, bar chart, geo map, alert list, logs, trace
- **Data Sources**: Prometheus, Loki, Elasticsearch, InfluxDB, MySQL, PostgreSQL, MSSQL, Jaeger, Tempo, CloudWatch, Azure Monitor, Graphite
- **Templating**: Variables, repeated panels/rows, dynamic queries, chaining
- **Alerting**: Grafana Alerting (modern), recording rules, alert instances, notification policies, contact points
- **Provisioning**: YAML provisioning, Grafonnet (JSONnet), Terraform provider_grafana
- **Transformations**: Data processing within panels (reduce, organize, join, time series)
- **Grafana Cloud**: Hosted dashboards, alerting, logs, traces, synthetic monitoring

**Common Use Cases:**
- Building time-series dashboards for infrastructure metrics
- Creating alert rules for service-level objectives
- Setting up log exploration with Loki or Elasticsearch
- Building distributed tracing dashboards with Jaeger/Tempo
- Templating dashboards for multi-environment deployments
- Provisioning dashboards as code for GitOps workflows

---

## § 3 · Risk Disclaimer

| Risk | Severity | Description | Mitigation |
|------|----------|-------------|------------|
| **Dashboard Staleness** | 🟡 Medium | Dashboards show outdated metrics or broken queries | Review dashboards quarterly; use recording rules |
| **Alert Fatigue** | 🔴 High | Too many alerts cause ignored pages | Tune thresholds to baselines; use multi-dimensional alerts |
| **Query Performance** | 🟡 Medium | Complex queries slow dashboard loading | Use `$__rate_interval`; limit query range; use recording rules |
| **Provisioning Conflicts** | 🟡 Medium | Git-managed configs conflict with UI changes | Use provisioning exclusively; disable UI admin |
| **Data Source Misconfig** | 🟡 Medium | Wrong TLS or auth causes missing data | Verify data source health; check proxy settings |
| **Template Variable Injection** | 🟡 Medium | Malicious variable values can break queries | Use scoped variables; avoid `__text`/`__value` from user input |

**⚠️ IMPORTANT:**
- Always use `$__rate_interval` in PromQL for rate queries — prevents gaps
- Set minimum interval on panels to prevent excessive query load
- Use Grafana Alerting (v8+) over legacy alerting — unified alert/repeat

---

## § 4 · Core Philosophy

### 4.1 Panel Selection Guide

| Data Pattern | Best Panel | Alternatives |
|-------------|------------|--------------|
| **Metric over time** | Time series | Bar chart (low cardinality) |
| **Single current value** | Stat | Gauge, Singlestat (deprecated) |
| **Progress toward goal** | Gauge | Time series with threshold |
| **Percentage/ratio** | Stat (colored) | Pie chart, Time series |
| **Distribution/histogram** | Histogram | Heatmap |
| **Density over time** | Heatmap | Time series (heatmap mode) |
| **Categorical comparison** | Bar chart | Table, Pie chart |
| **Text/metadata** | Text panel | Stat panel |
| **Logs** | Logs panel | Table with log data |
| **Geographic data** | Geomap | Worldmap |
| **Traces** | Trace panel | Table with trace IDs |
| **Alert status** | Alert list | Stat panel with alert count |

### 4.2 Alerting Principles

```
Grafana Alerting (v8+):         Legacy Alerting:
┌─────────────────────────┐    ┌─────────────────────────┐
│ Alert Rule               │    │ Alert                   │
│ ├─ Condition (PromQL)    │    │ ├─ Condition            │
│ ├─ Alert Instances       │    │ ├─ For (evaluation time)│
│ └─ Labels                │    │ └─ Notifications        │
├─────────────────────────┤    └─────────────────────────┘
│ Notification Policy      │
│ ├─ Contact Points        │
│ └─ Routing (labels)       │
└─────────────────────────┘
```

**Alert Rule Design:**
1. **Alert on symptoms**: Error rate > 5%, availability < 99%
2. **Avoid causes**: Don't alert on CPU > 80% unless critical
3. **Use multi-dimensional alerts**: `ALERTS{service="checkout", env="prod"}`
4. **Set reasonable `for` duration**: 5m to avoid flapping
5. **Annotate with runbook URL**: Every alert needs documentation

### 4.3 Dashboard Design Principles

1. **Templated, not duplicated**: One dashboard per service with environment variable
2. **SLO-Focused**: Primary view shows SLO metrics prominently
3. **Drill-Down Architecture**: Overview → Service Detail → Instance Detail
4. **Query Efficiency**: Use recording rules for complex queries
5. **Consistent Styling**: Use dashboard variables for colors and thresholds

---

### 5.1 Data Source Configuration

| Data Source | Query Language | Use Case | Key Config |
|------------|---------------|----------|------------|
| **Prometheus** | PromQL | Metrics | `http_url`, `access` (server/proxy/browser) |
| **Loki** | LogQL | Logs | Log labels, derived fields |
| **Elasticsearch** | Elasticsearch DSL | Logs, metrics | Index pattern, time field |
| **InfluxDB** | InfluxQL/Flux | Time series | Database, retention policy |
| **MySQL/PostgreSQL** | SQL | Application data | Host, database, credentials |
| **Jaeger** | Jaeger API | Traces | Collector endpoint |
| **Tempo** | TraceQL | Traces | Endpoint, authentication |
| **CloudWatch** | AWS CloudWatch | AWS metrics | IAM role or access keys |
| **Azure Monitor** | KQL | Azure metrics | Application ID, Tenant ID |
| **Graphite** | Graphite render API | Metrics | URL, version |

### 5.2 Grafana Editions

| Edition | Features | Licensing |
|---------|----------|-----------|
| **OSS** | All panel types, alerting, provisioning | Apache 2.0 |
| **Enterprise** | + LDAP/SSO, Reporting, Extended data sources | Commercial |
| **Cloud** | Hosted, Alerts, Logs, Traces, Synthetic monitoring | Usage-based subscription |
| **Enterprise Data Source** | + Datadog, Splunk, Dynatrace, New Relic | Commercial |

### 5.3 Grafana Variables

| Variable Type | Syntax | Example |
|--------------|--------|---------|
| **Query** | `$<name>` | `label_values(up, job)` → `prometheus`, `alertmanager` |
| **Custom** | `$<name>` | `prod,staging,dev` |
| **Constant** | `$<name>` | `86400` (seconds per day) |
| **Text box** | `${<name>}` | Free-form input |
| **Interval** | `$<name>` | `1m,5m,15m,1h` |
| **Datasource** | `$<name>` | Switch data sources in dashboard |
| **Ad hoc filter** | `$<name>` | Auto-generated from data source |

---

## § 6 · Professional Toolkit

### 6.1 PromQL Query Templates

```promql
[Code block moved to code-block-1.md]
```

### 6.2 Dashboard JSON Examples

```json
[Code block moved to code-block-1.md]
```

### 6.3 Alert Rule YAML (Grafonnet)

```jsonnet
[Code block moved to code-block-2.md]
```

### 6.4 Loki Log Query Examples

```logql
# Basic log query with filters
{service="checkout", environment="production"} |= "error" | json

# Parse JSON logs and extract fields
{service="checkout"} | json | level="error" | latency_ms > 1000

# Count logs by level
sum by (level) (count_over_time({service="checkout"}[5m]))

# Extract and aggregate
{service="checkout"} | json | line_format "{{.timestamp}} - {{.message}}"

# Parse nginx logs with regex
{service="nginx"} | regexp '(?P<ip>\\S+) - \\S+ \\[(?P<timestamp>[^\\]]+)\\] "(?P<request>[^"]+)" (?P<status>\\d+)'

# Live tail
{tervice="checkout"} | json | level="error"

# Metrics from logs (count errors per minute)
sum by (service) (rate({service=~"checkout|payment"}[1m] | json | status >= 500 [1m]))
```

---

## § 7 · Standards & Reference

### 7.1 Dashboard Naming Convention

| Pattern | Example | Use |
|---------|---------|-----|
| `{Service} - Overview` | `Checkout - Overview` | Service entry point |
| `{Service} - SLO` | `Checkout - SLO` | SLO metrics only |
| `{Service} - Infrastructure` | `Checkout - Infrastructure` | Host/container metrics |
| `{Service} - Application` | `Checkout - Application` | App-specific metrics |
| `{Team} - Platform` | `Platform - Kubernetes` | Team-level dashboards |
| `{Environment} - Overview` | `Production - Overview` | Environment overview |

### 7.2 Color Palette

| Theme | Success | Warning | Error | Info |
|-------|---------|---------|-------|------|
| **Light** | Green `#73BF69` | Yellow `#FAD64B` | Red `#FC4E4E` | Blue `#3274D9` |
| **Dark** | Green `#73BF69` | Yellow `#FAD64B` | Red `#FC4E4E` | Blue `#5794F2` |

### 7.3 Threshold Standards

| Metric Type | Warning | Critical | Unit |
|------------|---------|----------|------|
| Error Rate | 1% | 5% | % |
| Latency (p99) | 500ms | 2000ms | ms |
| Availability | 99.5% | 99% | % |
| CPU | 70% | 90% | % |
| Memory | 80% | 95% | % |
| Disk | 80% | 90% | % |
| Request Rate | Anomaly | Baseline deviation | reqps |

---

## § 8 · Troubleshooting

### 8.1 Common Issues and Solutions

| Issue | Cause | Solution |
|-------|-------|----------|
| **No data in panel** | Query error, wrong time range | Check query in Explore; extend time range |
| **Gaps in time series** | `[$__interval]` too large | Change to `[$__rate_interval]` |
| **Panel loads slowly** | Complex query, no recording rule | Use recording rules; reduce time range |
| **Alert not firing** | Query returns no data | Add `> 0`; check `for` duration |
| **Variable not populating** | Data source query error | Test query in variable editor |
| **Templated query broken** | Variable not defined | Ensure variable order; use `${var}` |
| **Provisioned dashboard reverted** | Provisioning overrides UI | Disable `allowUiUpdates` or use provisioning only |

### 8.2 Debug Commands

```bash
# Test Prometheus query in curl
curl "http://prometheus:9090/api/v1/query?query=up" | jq

# Check Grafana health
curl http://localhost:3000/api/health

# Test data source
curl -X POST http://localhost:3000/api/ds/query \
  -H "Content-Type: application/json" \
  -d '{"queries":[{"refId":"A","datasourceId":1,"expr":"up"}]}'
```

---


## § 9 · Scenario Examples

### Scenario 1: Initial Consultation

**Context:** A new client needs guidance on grafana expert.

**User:** "I'm new to this and need help with [problem]. Where do I start?"

**Expert:** Welcome! Let me help you navigate this challenge.

**Assessment:**
- Current experience level?
- Immediate goals and constraints?
- Key stakeholders involved?

**Roadmap:**
1. **Phase 1:** Discovery & Assessment
2. **Phase 2:** Strategy Development
3. **Phase 3:** Implementation
4. **Phase 4:** Review & Optimization

---

### Scenario 2: Problem Resolution

**Context:** Urgent grafana expert issue needs attention.

**User:** "Critical situation: [problem]. Need solution fast!"

**Expert:** Let's address this systematically.

**Triage:**
- Impact: [Critical/High/Medium]
- Timeline: [Immediate/24h/Week]
- Reversibility: [Yes/No]

**Options:**
| Option | Approach | Risk | Timeline |
|--------|----------|------|----------|
| Quick | Immediate fix | High | 1 day |
| Standard | Balanced | Medium | 1 week |
| Complete | Thorough | Low | 1 month |

---

### Scenario 3: Strategic Planning

**Context:** Build long-term grafana expert capability.

**User:** "How do we become world-class in this area?"

**Expert:** Here's an 18-month roadmap.

**Phase 1 (M1-3): Foundation**
- Baseline assessment
- Quick wins identification
- Infrastructure setup

**Phase 2 (M4-9): Acceleration**
- Core system implementation
- Team upskilling
- Process standardization

**Phase 3 (M10-18): Excellence**
- Advanced methodologies
- Innovation pipeline
- Knowledge leadership

**Metrics:**
| Dimension | 6 Mo | 12 Mo | 18 Mo |
|-----------|------|-------|-------|
| Efficiency | +20% | +40% | +60% |
| Quality | -30% | -50% | -70% |

---

### Scenario 4: Quality Assurance

**Context:** Deliverable requires quality verification.

**User:** "Can you review [deliverable] before delivery?"

**Expert:** Conducting comprehensive quality review.

**Checklist:**
- [ ] Requirements aligned
- [ ] Standards compliant
- [ ] Best practices applied
- [ ] Documentation complete

**Gap Analysis:**
| Aspect | Current | Target | Action |
|--------|---------|--------|--------|
| Completeness | 80% | 100% | Add X |
| Accuracy | 90% | 100% | Fix Y |

**Result:** ✓ Ready for delivery

---

## § 10 · Example Interactions

→ **Detailed examples moved to [`references/09-examples.md`](references/09-examples.md)**

| Example | Description |
|---------|-------------|
| **Example 1** | Multi-environment SLO dashboard with template variables |
| **Example 2** | Grafana alerting with PagerDuty integration |
| **Example 3** | GitOps dashboard provisioning |

## § 11 · Edge Cases

→ **Detailed edge case solutions moved to [`references/09-examples.md`](references/09-examples.md)**

| Scenario | Problem | Solution |
|----------|---------|----------|
| **Dynamic Services** | One dashboard for many services | Template variable with `$__all` |
| **Missing Metrics** | Alert not firing | `absent()` function |
| **Correlated Traces** | Metrics + traces together | Mixed datasource |
| **Row Groups** | Too many panels | Collapsible rows with repeat |
| **Percentage Calc** | sum(A) / sum(A+B) | Reduce transformation |
| **Sensitive Data** | Sensitive annotation info | Labels for sensitive data |
| **Multi-Datasource Vars** | Variable from two sources | Mixed datasource |
| **Long Queries** | Dashboard timeout | Recording rules |

---

## § 12 · Related Skills

| Combination | Workflow | Result |
|-------------|----------|--------|
| Grafana + **Prometheus** | Query Prometheus → Grafana dashboards | Full metrics observability |
| Grafana + **Loki** | Log exploration in Grafana | Unified metrics + logs |
| Grafana + **PagerDuty** | Grafana Alerting → PagerDuty | Incident notification |
| Grafana + **Datadog** | Import Datadog metrics via plugin | Datadog in Grafana |
| Grafana + **OpenTelemetry** | Tempo for traces in Grafana | 3 pillars of observability |

---

## § 13 · Change Log

### v3.0.0 (2026-03-20)
- Full upgrade to comprehensive 9.5/10 standard
- Complete rewrite with full System Prompt (role, decision framework, thinking patterns)
- Added complete dashboard JSON examples with time series, stat, and gauge panels
- Added PromQL query templates for common monitoring patterns
- Added Grafonnet alerting examples and GitOps provisioning guide
- Added troubleshooting workflow and 8+ edge case scenarios
- Added Loki LogQL query examples

### v1.0.0 (2024-01-01)
- Initial basic skill creation

---

## § 14 · Contributing

Contributions to improve this skill are welcome. Please:
1. Follow Grafana dashboard naming conventions
2. Include PromQL/LogQL queries with `$__rate_interval`
3. Add working dashboard JSON with proper panel configurations
4. Test all alerting rules before contributing
5. Reference official Grafana documentation for accuracy

---

## § 15 · Final Notes

Grafana provides powerful visualization across metrics, logs, and traces. Always use `$__rate_interval` for PromQL rate queries, design alerts on symptoms rather than causes, and provision dashboards as code for GitOps workflows. Use template variables to build reusable dashboards, and leverage Grafonnet for complex dashboard generation.

---

## § 16 · Install Guide

**Quick Install:**
```
Read https://raw.githubusercontent.com/theneoai/awesome-skills/main/skills/tools/observability/grafana-expert.md and install as skill
```

**Trigger Words:** "Grafana", "dashboard", "visualization", "Grafana alerting", "Grafana panels", "监控仪表盘", "PromQL", "LogQL"

---


## Domain Benchmarks

| Metric | Industry Standard | Target |
|--------|------------------|--------|
| Quality Score | 95% | 99%+ |
| Error Rate | <5% | <1% |
| Efficiency | Baseline | 20% improvement |


### Done Criteria
- All tasks completed per specification
- Quality standards met
- Stakeholder approval received

### Fail Criteria
- Quality defects detected
- Requirements not met
- Timeline/budget overrun