Prometheus

---
name: prometheus
description: >-
  Prometheus instrumentation discipline: right metric type, right name, right labels.
  Invoke whenever task involves any interaction with Prometheus metrics — instrumenting
  application code, writing PromQL queries, defining alerting or recording rules, choosing
  metric types, managing label cardinality, building exporters, or reviewing monitoring
  configuration.
---

# Prometheus

Choose the right metric type, name it clearly, label it sparingly. Prometheus is a pull-based monitoring system built on
a dimensional data model — every metric is a time series identified by a name and key-value label pairs. Getting this
right at instrumentation time prevents rework later.

## References

- **Metric types** — [`${CLAUDE_SKILL_DIR}/references/metric-types.md`] Extended type comparison, histogram bucket
  tuning, summary configuration
- **Naming** — [`${CLAUDE_SKILL_DIR}/references/naming.md`] Full naming examples, base units table, character rules,
  label best practices
- **Instrumentation** — [`${CLAUDE_SKILL_DIR}/references/instrumentation.md`] Code patterns per system type, library
  instrumentation, performance tuning
- **PromQL** — [`${CLAUDE_SKILL_DIR}/references/promql.md`] Full operator catalog, vector matching, over-time
  aggregation, operator precedence
- **Alerting and rules** — [`${CLAUDE_SKILL_DIR}/references/alerting-and-rules.md`] Alert design, recording rule naming,
  aggregation patterns, anti-patterns
- **Exporters** — [`${CLAUDE_SKILL_DIR}/references/exporters.md`] Exporter architecture, collectors, help strings,
  push-based sources

## Metric Type Selection

Choose correctly at instrumentation time — changing later requires migrating dashboards, alerts, and recording rules.

| Question                                                                       | Answer | Type      |
| ------------------------------------------------------------------------------ | ------ | --------- |
| Can the value decrease?                                                        | No     | Counter   |
| Is it a snapshot of current state?                                             | Yes    | Gauge     |
| Observing a distribution needing cross-instance aggregation?                   | Yes    | Histogram |
| Need accurate quantiles from a single instance, known at instrumentation time? | Yes    | Summary   |
| None of the above                                                              | —      | Gauge     |

### Counter

Monotonically increasing value — resets to zero only on restart.

- Use for: requests served, errors occurred, bytes transferred, tasks completed
- API: `inc()`, `inc(v)` where v >= 0
- Always suffix with `_total`: `http_requests_total`
- Always apply `rate()` or `increase()` in queries — raw values are meaningless
- Prometheus handles counter resets automatically in `rate()`
- Never use a counter for values that can decrease — that is a gauge

### Gauge

Value that goes up and down arbitrarily.

- Use for: temperature, memory usage, in-progress requests, queue depth, timestamps
- API: `inc()`, `dec()`, `set(v)`, `set_to_current_time()`
- No `_total` suffix
- Never apply `rate()` to a gauge — use `deriv()` or `delta()`
- **Timestamp pattern:** store Unix epoch seconds as `myapp_last_success_timestamp_seconds`; compute elapsed time with
  `time() - metric` in PromQL
- **Info metric pattern:** `myapp_build_info{version="1.2.3", commit="abc"} 1` — metadata as labels with constant value
  1

### Histogram

Samples observations into configurable buckets. Produces `_bucket{le="..."}`, `_sum`, `_count`.

- Use for: request latencies, response sizes, any distribution needing percentiles or cross-instance aggregation
- API: `observe(v)`
- Buckets are cumulative — `le="0.5"` includes all observations <= 0.5
- Must include `+Inf` bucket (equal to `_count`)
- Choose buckets matching expected value range; place more buckets near SLO boundaries
- Buckets cannot be changed after metric creation
- Use `histogram_quantile()` in PromQL to calculate percentiles
- Aggregatable across instances — the primary advantage over summary

### Summary

Calculates streaming quantiles on the client side. Produces `{quantile="..."}`, `_sum`, `_count`.

- **Cannot be aggregated across instances** — `avg(x{quantile="0.95"})` is statistically invalid
- `_sum` and `_count` without quantiles is a valid and useful configuration

**Use summary over histogram only when ALL of these are true:**

1. You need accurate quantiles (not approximate)
2. From a single instance (no cross-instance aggregation)
3. You know the exact quantiles at instrumentation time
4. You accept that adding new quantiles requires code changes

**Default choice: histogram.** See `${CLAUDE_SKILL_DIR}/references/metric-types.md` for detailed comparison.

## Naming

Format: `<namespace>_<subsystem>_<name>_<unit>_<suffix>`. Not all parts required — minimum is namespace + meaningful
name + unit/suffix.

### Naming Rules

1. Use `snake_case` — lowercase with underscores, matching `[a-zA-Z_:][a-zA-Z0-9_:]*`
2. Colons (`:`) are reserved for recording rules — never use in direct instrumentation
3. Double underscore prefix (`__`) is reserved for Prometheus internals
4. Every metric MUST have a namespace prefix identifying its origin
5. Always use base units — seconds not milliseconds, bytes not megabytes. Let visualization tools handle conversion.
6. Append unit to metric name in plural form: `http_request_duration_seconds`
7. Suffix counters with `_total`, counter-with-unit as `_<unit>_total` (e.g., `process_cpu_seconds_total`)
8. Suffix info metrics with `_info`, timestamps with `_timestamp_seconds`
9. A metric MUST represent the same logical thing across all its label dimensions — `sum()` or `avg()` across all
   dimensions should be meaningful. If nonsensical, split into separate metrics.

See `${CLAUDE_SKILL_DIR}/references/naming.md` for base units table, component ordering, and full examples.

## Labels

Use labels for dimensions you will filter or aggregate by: `http_requests_total{method="GET", status="200"}` — not
separate metrics per status. Do not put label names in metric names.

### When NOT to Use Labels

- Unbounded values — user IDs, email addresses, full URLs, query strings
- High cardinality — anything above ~100 unique values per metric

### Cardinality

Every unique label combination is a new time series — each costs RAM, CPU, disk, and network. **Cardinality math:**
total series = metric cardinality x number of targets.

- `< 10`: Safe for most metrics
- `10-100`: Acceptable, monitor growth
- `100-1000`: Investigate alternatives
- `> 1000`: Move analysis out of Prometheus

### Label Best Practices

1. Start with no labels. Add as concrete use cases emerge.
2. Keep cardinality below 10 per metric as a default target.
3. Initialize all label combinations you know upfront to avoid missing metrics — export 0 for known label sets.
4. Use stable label values. Avoid labels that change frequently.
5. Never include a "total" label value — rely on Prometheus `sum()`.

## Instrumentation Patterns

### Online-Serving Systems (HTTP servers, APIs, databases)

Key metrics: request rate (`_total`), error rate, latency (histogram), in-progress (gauge).

- Count requests at completion (not start) — aligns with error and latency stats
- Always have a total requests counter alongside error counters (for ratio calculation)

### Offline Processing (queues, pipelines, ETL)

Key metrics per stage: items in (`_total`), items out (`_total`), in progress (gauge), last processed timestamp (gauge),
processing duration (histogram).

- Export heartbeat timestamps to detect stalled processing

### Batch Jobs (cron, scheduled tasks)

Key metrics (push to Pushgateway): last success timestamp (gauge), last completion timestamp (gauge), duration (gauge —
single run, not distribution).

- Batch job durations are gauges (single event), not histograms
- Jobs running more often than every 15 minutes should be converted to daemons

### Libraries

Instrument transparently — users get metrics without configuration. Minimum for external resource access: request count
(counter), error count (counter), latency (histogram).

### Subsystem Patterns

- **Logging:** maintain `log_messages_total{level="..."}` counter per log level
- **Failures:** always pair failure counter with total attempts counter for ratio calculation
- **Caches:** `cache_requests_total{result="hit|miss"}`, evictions (counter), size (gauge), lookup latency (histogram).
  Also instrument the downstream system.

See `${CLAUDE_SKILL_DIR}/references/instrumentation.md` for threadpool patterns, custom collectors, and performance
tuning in hot paths.

## PromQL

### Rate and Increase (Counters Only)

- `rate(counter[5m])` — per-second rate. Use for alerts and dashboards.
- `increase(counter[5m])` — total increase. Sugar for `rate() * range_seconds`.
- `irate(counter[5m])` — instant rate from last two samples. Only for graphing volatile counters.
- **`rate()` first, then aggregate:** `sum(rate(x[5m]))` — never `rate(sum(x)[5m])`. Rate must see individual counter
  resets.
- Never `rate()` a gauge — use `deriv()` or `delta()`.

### Histogram Quantiles

- `histogram_quantile(0.95, rate(metric_bucket[5m]))` — single histogram
- When aggregating histogram buckets, always preserve `le` in the `by` clause:
  `histogram_quantile(0.95, sum by (job, le) (rate(metric_bucket[5m])))`
- Average duration: `rate(metric_sum[5m]) / rate(metric_count[5m])`

### PromQL Gotchas

- **Staleness:** most recent sample within lookback period (default 5 min). Series disappears if not scraped within that
  window.
- **Rate window size:** `rate()` needs at least 2 samples. Range should be at least 4x scrape interval. With 15s scrape
  use `rate(x[5m])` or wider.
- **Expensive queries:** bare metric names can expand to thousands of series. Always filter or aggregate before
  graphing. Use recording rules for expensive expressions.

See `${CLAUDE_SKILL_DIR}/references/promql.md` for data types, selectors, aggregation operators, vector matching,
over-time aggregation, binary operators, and operator precedence.

## Alerting Rules

Alert on **symptoms** (user-visible impact), not causes. Use dashboards to pinpoint causes after an alert fires.

- **Online-serving** — High latency, high error rate (user-facing, high in the stack)
- **Offline processing** — Data taking too long to get through the system
- **Batch jobs** — Job has not succeeded recently enough (>= 2x normal cycle)
- **Capacity** — Approaching resource limits that will cause outage without intervention

Only page on latency at one point in the stack — if overall user latency is fine, don't page on a slow sub-component. If
an alert fires and there's nothing to do, remove it.

See `${CLAUDE_SKILL_DIR}/references/alerting-and-rules.md` for alert design, naming conventions, and recording rule
details.

## Recording Rules

Pre-compute frequently used or expensive expressions. Format: `level:metric:operations`.

- Aggregate ratios correctly — aggregate numerator and denominator separately, then divide. Never average a ratio.
- Use `without` for aggregation — preserves all labels except those removed. Prefer over `by`.
- Use recording rules for expensive dashboard queries, expressions used in multiple alerts, or complex multi-step
  aggregations.

See `${CLAUDE_SKILL_DIR}/references/alerting-and-rules.md` for full naming convention and recording rule anti-patterns.

## Exporters

Write an exporter when the target system does not expose Prometheus metrics natively. For your own code, use a client
library directly.

- Prefix all metrics with exporter name: `haproxy_up`, `mysql_global_status_threads_connected`
- Create fresh metric instances per scrape — do NOT use global metric variables updated each scrape (race conditions,
  stale labels)
- Drop pre-computed rates, min/max since start, and stddev from source systems — export raw counters and current values;
  let Prometheus `rate()` handle the rest

See `${CLAUDE_SKILL_DIR}/references/exporters.md` for architecture, collectors, help strings, label rules, and
push-based sources.

## Application

When **writing** Prometheus instrumentation:

- Apply all conventions silently — don't narrate each rule.
- Choose metric types based on the decision criteria above.
- If an existing codebase contradicts a convention, follow the codebase and flag the divergence once.

When **writing** PromQL queries:

- Always wrap counters in `rate()` or `increase()` before further operations.
- Prefer `without` over `by` for aggregation.

When **writing** alerting or recording rules:

- Follow `level:metric:operations` naming.
- Alert on symptoms, not causes.
- Aggregate ratios correctly (numerator and denominator separately).

When **reviewing** Prometheus code:

- Cite the specific violation and show the fix inline.
- State what's wrong and how to fix it.

## Integration

The coding skill governs workflow; this skill governs Prometheus implementation choices.