Slo Reliability Architect

---
name: slo-reliability-architect
description: Derive SLOs from user journeys, not infrastructure — inventory the journeys users depend on, select symptom-based SLIs per journey (availability, latency percentiles, correctness, freshness — measured where users experience them, blind spots named), set targets with error budgets in user-meaningful units, design burn-rate alerting where PAGES fire on symptoms/budget burn and causes (CPU, restarts, queue depth) go to tickets, analyze failure modes against the targets, and define the error-budget policy (what happens to release velocity when the budget is spent) plus per-tenant/noisy-neighbor views and a review cadence. Produces the SLO catalog and alert spec that observability-operator implements. Use when asked to define SLOs/SLIs/error budgets, decide what should page, set reliability targets, or rationalize a noisy alert inventory. Do NOT use to implement alerts/dashboards (observability-operator), author incident procedures (incident-response-runbook), or debug current failures (systematic-debugger).
---

# SLO Reliability Architect

Terms used below: a **service-level indicator (SLI)** measures a user-visible
symptom; a **service-level objective (SLO)** is the target for that indicator;
and a **service-level agreement (SLA)** is an applicable customer commitment.
An **error budget** is the allowed amount of failed or slow experience implied
by an SLO over its window. Infrastructure signals and components such as
central processing unit (CPU) load, Domain Name System (DNS), content
delivery network (CDN), and garbage collection (GC) help diagnosis but do
not replace journey SLIs. The 99th-percentile (p99) latency is the time within
which 99 percent of requests complete, and API means application programming
interface.

## Purpose

Produce a reliability design the business can reason about and on-call can
live with: user-journey-derived SLOs with error budgets, symptom-based SLIs
measured where users experience them, burn-rate alerting that pages only on
what threatens the budget, failure-mode analysis against the targets, and
the budget policy that converts reliability data into release decisions.
The discipline is journeys-before-infrastructure: users do not experience
CPU; they experience "checkout failed" — so checkout, not CPU, is what gets
a target, a budget, and a page.

## Use When

- Use when: asked to define SLOs, SLIs, or error budgets for a product,
  service, or user journey.
- Use when: deciding what should page vs ticket — or rationalizing a noisy
  alert inventory around symptoms.
- Use when: setting reliability targets for a new surface, or revisiting
  targets after incidents or architecture changes.
- Use when: reliability vs release-velocity tension needs a policy (the
  error budget is that policy).
- Use when: a multi-tenant product needs per-tenant reliability visibility
  (noisy neighbors, enterprise-tier commitments).
- Do NOT use when: implementing the resulting alerts, dashboards, or
  instrumentation — `observability-operator` *(manual-only)* executes this
  design.
- Do NOT use when: writing what on-call DOES when paged —
  `incident-response-runbook`.
- Do NOT use when: drafting contractual SLAs — this skill informs the
  commercial decision with achievable numbers; it does not own the
  contract.
- Do NOT use when: diagnosing an active reliability problem —
  `systematic-debugger` *(manual-only)*.

## Inputs to Inspect

1. The user journeys that matter: product flows, `qa-strategy-architect`
   risk inventory / `test-plan-designer` critical journeys if present —
   reliability targets attach to what users depend on.
2. Real traffic and failure history: request volumes, current
   availability/latency distributions, incident history — targets set
   without baseline data are aspirations, labeled as such.
3. The architecture and dependency chains (`architecture-designer` /
   cloud-provider architecture outputs): what each journey traverses, shared
   dependencies, single points of failure.
4. Current alert inventory and its noise profile (what pages today, what
   on-call ignores).
5. Commercial commitments: SLAs already sold, tier promises, enterprise
   contracts — the floor the SLOs must clear with margin.
6. Tenancy shape: pooled/siloed components, tenant size skew — whether one
   tenant's experience can diverge from the aggregate.
7. Deploy/release cadence and the team's appetite for a budget policy with
   teeth.

## Workflow

When an undeliverable service promise requires a human choice, define SLA,
SLO, and the measured gap; explain viable promise changes and architecture
work by reason, user impact, pros, cons, and money/setup/upkeep cost or
uncertainty. Recommend the evidence-supported path before one decision
question. A choice does not revise a contract or authorize infrastructure work.

1. **Inventory the journeys**: enumerate the user journeys worth a
   reliability promise (login, core workflow, checkout/billing, data
   export, API integration paths), each with its user population, business
   impact of failure, and traffic volume. Rank; not everything gets an
   SLO — an SLO catalog nobody can recite protects nothing.
2. **Select symptom-based SLIs per journey**: availability (good events /
   valid events, with "good" and "valid" defined precisely — what counts,
   what is excluded, where measured), latency (percentile thresholds at
   the user-experienced edge, not the backend hop), correctness/freshness
   where the journey's value is data (exports complete, dashboards
   current). Measurement point stated per SLI: load balancer, client,
   synthetic — each with its blind spots named.
3. **Set targets with error budgets**: per SLI, a target justified by
   baseline data, commercial floor, and cost of the next nine — with the
   error budget stated in user-meaningful units (minutes of full outage
   per month, failed requests per million). Targets without baseline data
   are labeled provisional with a review date.
4. **Design burn-rate alerting**: pages fire on budget burn rates
   (fast-burn: page now; slow-burn: page/ticket on sustained burn), causes
   (CPU, GC, restarts, queue depth) go to tickets and dashboards as
   diagnostic context. Every page maps to a budget threat; every
   cause-alert that pages today is explicitly demoted or defended in the
   spec. Alert spec fields: SLI, burn threshold + window, severity,
   owner, runbook link placeholder — handed to `observability-operator` *(manual-only)*
   to implement and to `incident-response-runbook` for the procedures.
5. **Analyze failure modes against targets**: per journey, walk the
   dependency chain — shared dependencies whose failure burns multiple
   budgets, retry/timeout behavior that amplifies, saturation points,
   and the blast radius of each dependency's failure. Findings that
   require architectural change route to `architecture-designer`; this
   skill sizes the reliability impact.
6. **Define the error-budget policy**: what happens when a budget is
   spent — release freeze/slowdown criteria, who decides exceptions, how
   the budget resets, and what "budget spent on planned work vs incidents"
   means. A budget with no consequences is a chart, not a policy.
7. **Add the tenant dimension**: per-tenant SLI views where tenancy skew
   can hide misery in aggregates (one big tenant at 50% error rate inside
   a 99.9% aggregate), noisy-neighbor detection signals, and
   tier-differentiated targets where commercial commitments differ.
8. **Set the review cadence**: SLO review rhythm (post-incident, quarterly
   target review, alert-precision review — pages that led to action vs
   noise), and the staleness triggers (architecture change, traffic-shape
   change) that reopen the design.

## Output Format

```
RELIABILITY DESIGN — <product/scope>
Journey inventory: <journey — users — business impact — volume — ranked;
  explicitly not-covered journeys listed>
Service-level indicator (SLI) definitions: <journey — measured signal —
  good/valid definitions — measurement point + blind spots>
Service-level objective (SLO) targets & error budgets: <SLI — target —
  allowed failure in user-meaningful units — justification
  (baseline/commercial/cost) — provisional flags + review dates>
Alert spec (for observability-operator): <SLI — fast/slow error-budget burn
  thresholds + windows — severity — owner — page/ticket — runbook link placeholder>
Demotions: <today's cause-alerts that stop paging, each with rationale>
Failure-mode analysis: <journey — dependency chain — shared/single points
  — amplification risks — architectural findings routed>
Error-budget policy: <spent-budget consequences — decider — reset — 
  planned-work accounting>
Tenant dimension: <per-tenant views needed — noisy-neighbor signals —
  tier-differentiated targets>
Review cadence: <rhythms + staleness triggers>
Assumptions & open questions: <each with risk-if-wrong / who answers>
Promise choice (if needed): <explained paths, costs/unknowns, pros/cons, recommendation + why>
```

## Validation Checklist

- [ ] A human-facing promise/architecture choice is explained before one
      question; contract and infrastructure authority remain separate.

- [ ] Every SLO traces to a user journey with named business impact — no
      SLO exists because a metric was available.
- [ ] Every SLI defines "good" and "valid" precisely and names its
      measurement point with blind spots.
- [ ] Every target carries a justification (baseline data, commercial
      floor, cost) or a provisional label with a review date.
- [ ] Error budgets are stated in user-meaningful units.
- [ ] Every page in the alert spec maps to budget burn; every currently
      paging cause-alert is explicitly demoted or defended.
- [ ] The budget policy has consequences and a decider.
- [ ] The tenant dimension is addressed (or explicitly n/a for
      single-tenant).
- [ ] Implementation was handed to `observability-operator`, procedures to
      `incident-response-runbook` — neither restated here.

## Gotchas

- Availability measured at the backend misses the CDN, DNS, and client —
  the user's 99.9% and the server's 99.99% differ by real outages; name
  the measurement point's blind spots.
- Percentile latency over a low-traffic window is noise: a p99 on 50
  requests pages on one slow request. Windows and minimum-volume guards
  are part of the SLI definition.
- 100%-minus-epsilon targets are budget-free zones: a 99.99% target on a
  monthly window is ~4 minutes of budget — no room to deploy. The next
  nine must be priced, not defaulted.
- Aggregate SLOs hide tenant misery in skewed populations; if one tenant
  is 40% of traffic, per-tenant views are not optional.
- Retries mask failures into latency: a journey "succeeding" after three
  retries burns latency budget invisibly and amplifies load at the worst
  time — count retried-success distinctly.
- Error budgets spent by planned maintenance vs incidents are different
  signals; a policy that cannot tell them apart freezes releases for the
  wrong reasons.

## Stop Conditions

- No baseline data exists and none can be gathered (new product) →
  provisional targets only, labeled, with instrumentation-first ordering
  (`observability-operator` *(manual-only)*) and a mandatory review date; refuse to
  present aspirations as commitments.
- The commercially promised SLA exceeds what the architecture can
  plausibly deliver → surface the gap to the human as a commercial/
  architecture decision with explained paths and a reasoned recommendation
  (`human-approval-boundary` for the tradeoff);
  do not design SLOs that paper over it.
- Journey inventory cannot be established (nobody can say what users
  depend on) → stop; route to product/`qa-strategy-architect` risk
  inventory first. SLOs on infrastructure metrics alone are refused.
- The error-budget policy has no owner willing to enforce consequences →
  deliver the design with the policy gap named as its top risk.

## Supporting Files

- `references/slo-derivation.md` — SLI menu per journey type,
  burn-rate window/threshold patterns, budget-policy templates.
- `evals/evals.json` — trigger + behavior cases.
- `evals/trigger-evals.json` — discrimination within the reliability
  cluster (`observability-operator`, `incident-response-runbook`) and
  against shipped `qa-strategy-architect` / `saas-cost-architect`.