Slo Sli Sla

# 📈 Skill: Service Level Objectives (SLO/SLI/SLA)

## 📋 Metadata

| Atributo | Valor |
|----------|-------|
| **ID** | `sre-slo-sli-sla` |
| **Nivel** | 🔴 Avanzado |
| **Versión** | 1.0.0 |
| **Keywords** | `slo`, `sli`, `sla`, `error-budget`, `service-level`, `reliability`, `availability` |
| **Referencia** | [Google SRE Book - SLIs, SLAs, SLOs](https://sre.google/workbook/slo-document/) |

## 🔑 Keywords para Invocación

- `slo`
- `sli`
- `sla`
- `error-budget`
- `service-level`
- `reliability`
- `availability`
- `@skill:slo`

### Ejemplos de Prompts

```
Implementa SLOs y error budgets para servicios críticos
```

```
Define SLIs y configura monitoreo de SLOs
```

```
Setup SLA compliance y error budget tracking
```

```
@skill:slo - Sistema completo de SLO/SLI/SLA
```

## 📖 Descripción

Service Level Objectives (SLOs) definen objetivos de confiabilidad medibles para servicios. Este skill cubre la definición de SLIs (Service Level Indicators), creación de SLOs, cálculo de error budgets, y monitoreo de compliance con SLAs.

### ✅ Cuándo Usar Este Skill

- Servicios en producción con SLAs
- Productos con requisitos de confiabilidad
- Stakeholders requieren métricas de confiabilidad
- Error budget management
- Capacity planning basado en SLOs

### ❌ Cuándo NO Usar Este Skill

- Prototipos sin usuarios
- Servicios internos sin requisitos de confiabilidad
- Sistemas legacy sin capacidad de medición

## 🏗️ Framework SLO

```
SLI (Service Level Indicator)
    ↓
SLO (Service Level Objective) ← Error Budget
    ↓
SLA (Service Level Agreement)
```

## 💻 Implementación

> **📁 Scripts Ejecutables:** Este skill incluye scripts Python ejecutables en la carpeta [`scripts/`](scripts/):
> - [`error_budget.py`](scripts/error_budget.py) - Calculadora de error budget (CLI)
> - [`slo_api.py`](scripts/slo_api.py) - API REST para consultar SLOs desde Prometheus
> - [`requirements.txt`](scripts/requirements.txt) - Dependencias Python
>
> Ver ejemplos de uso en [`examples/usage_example.py`](examples/usage_example.py)

### 1. Definición de SLIs

#### 1.1 SLI Types

```yaml
# sli-definitions.yml
service_level_indicators:
  # Availability SLI
  - name: request_availability
    type: availability
    metric: |
      (
        sum(rate(http_requests_total{status!~"5.."}[5m]))
        /
        sum(rate(http_requests_total[5m]))
      )
    window: 5m
    description: "Percentage of successful requests"

  # Latency SLI
  - name: request_latency_p99
    type: latency
    metric: |
      histogram_quantile(0.99,
        sum(rate(http_request_duration_seconds_bucket[5m])) by (le)
      )
    window: 5m
    percentile: 99
    description: "99th percentile request latency"

  # Freshness SLI
  - name: data_freshness
    type: freshness
    metric: |
      time() - last_successful_data_update_timestamp
    threshold: 3600  # 1 hour
    description: "Age of most recent data update"

  # Correctness SLI
  - name: data_correctness
    type: correctness
    metric: |
      sum(rate(data_validation_errors_total[5m]))
      /
      sum(rate(data_operations_total[5m]))
    description: "Percentage of correct data operations"
```

### 2. SLO Definitions

```yaml
# slo-definitions.yml
service_level_objectives:
  - service: payment-service
    slo_name: availability
    sli: request_availability
    target: 99.95  # 99.95% availability
    window: 30d
    description: "Payment service must be available 99.95% of the time"

  - service: payment-service
    slo_name: latency_p99
    sli: request_latency_p99
    target: 1.0  # 1 second
    window: 30d
    description: "99th percentile latency must be under 1 second"

  - service: user-service
    slo_name: availability
    sli: request_availability
    target: 99.9  # 99.9% availability
    window: 30d
    description: "User service must be available 99.9% of the time"

  - service: analytics-service
    slo_name: data_freshness
    sli: data_freshness
    target: 3600  # 1 hour
    window: 30d
    description: "Analytics data must be updated within 1 hour"
```

### 3. Error Budget Calculation

**Script ejecutable:** [`scripts/error_budget.py`](scripts/error_budget.py)

Este script calcula y gestiona error budgets para SLOs. Puede ejecutarse como CLI o importarse como módulo.

#### Cuándo ejecutar

- **Análisis manual:** Calcular error budget para reportes o análisis
- **Validación:** Verificar si un servicio está consumiendo demasiado error budget
- **Planificación:** Calcular cuántos errores se pueden permitir antes de violar el SLO
- **Integración:** Importar la clase `ErrorBudget` en otros scripts

#### Uso como CLI

```bash
# Instalación (no requiere dependencias externas)
cd skills/system-reliability-engineering/slo-sli-sla

# Ejemplo básico
python scripts/error_budget.py \
  --slo-target 0.9995 \
  --total-requests 1000000 \
  --error-requests 400

# Modo interactivo
python scripts/error_budget.py \
  --slo-target 0.9995 \
  --interactive

# Ventana personalizada
python scripts/error_budget.py \
  --slo-target 0.9999 \
  --window-days 7 \
  --total-requests 500000 \
  --error-requests 10
```

#### Uso como módulo

```python
from scripts.error_budget import ErrorBudget

# Crear instancia
error_budget = ErrorBudget(slo_target=0.9995, window_days=30)

# Calcular estado
status = error_budget.calculate_remaining_budget(
    total_requests=1_000_000,
    error_requests=400
)

print(f"Status: {status['status']}")
print(f"Remaining: {status['remaining_percentage']:.2f}%")
```

#### Ejemplos completos

Ver [`examples/usage_example.py`](examples/usage_example.py) para más ejemplos de uso programático.

### 4. Prometheus SLO Queries

```promql
# SLO Compliance Query
# Availability SLO: 99.95% over 30 days

# Current availability (last 30 days)
(
  sum(rate(http_requests_total{service="payment-service", status!~"5.."}[5m]))[30d:]
  /
  sum(rate(http_requests_total{service="payment-service"}[5m]))[30d:]
) >= 0.9995

# Error budget remaining
(
  1 - (
    sum(rate(http_requests_total{service="payment-service", status=~"5.."}[5m]))[30d:]
    /
    sum(rate(http_requests_total{service="payment-service"}[5m]))[30d:]
  )
) / (1 - 0.9995)

# Error budget consumption rate
(
  sum(rate(http_requests_total{service="payment-service", status=~"5.."}[5m]))[30d:]
  /
  sum(rate(http_requests_total{service="payment-service"}[5m]))[30d:]
) / (1 - 0.9995)

# Burn rate (how fast we're consuming error budget)
# 2x burn rate = using 2x the allowed error rate
(
  sum(rate(http_requests_total{service="payment-service", status=~"5.."}[5m]))
  /
  sum(rate(http_requests_total{service="payment-service"}[5m]))
) / (1 - 0.9995) / (30 * 24 * 3600) * 86400

# Alert when error budget will be exhausted in 4 hours
(
  (
    sum(rate(http_requests_total{service="payment-service", status=~"5.."}[5m]))[30d:]
    /
    sum(rate(http_requests_total{service="payment-service"}[5m]))[30d:]
  ) / (1 - 0.9995)
  +
  (
    (
      sum(rate(http_requests_total{service="payment-service", status=~"5.."}[5m]))
      /
      sum(rate(http_requests_total{service="payment-service"}[5m]))
    ) / (1 - 0.9995) / (30 * 24 * 3600) * 14400
  )
) >= 1
```

### 5. SLO Monitoring Dashboard (Grafana)

```json
{
  "dashboard": {
    "title": "SLO Dashboard - Payment Service",
    "panels": [
      {
        "title": "Availability SLO (99.95%)",
        "targets": [
          {
            "expr": "sum(rate(http_requests_total{service=\"payment-service\", status!~\"5..\"}[5m]))[30d:] / sum(rate(http_requests_total{service=\"payment-service\"}[5m]))[30d:]",
            "legendFormat": "Current Availability"
          },
          {
            "expr": "0.9995",
            "legendFormat": "SLO Target"
          }
        ],
        "type": "graph"
      },
      {
        "title": "Error Budget Remaining",
        "targets": [
          {
            "expr": "(1 - (sum(rate(http_requests_total{service=\"payment-service\", status=~\"5..\"}[5m]))[30d:] / sum(rate(http_requests_total{service=\"payment-service\"}[5m]))[30d:])) / (1 - 0.9995)",
            "legendFormat": "Remaining Budget %"
          }
        ],
        "type": "graph",
        "alert": {
          "conditions": [
            {
              "evaluator": {
                "params": [0.25],
                "type": "lt"
              },
              "operator": {
                "type": "and"
              },
              "query": {
                "params": ["A", "5m", "now"]
              },
              "reducer": {
                "params": [],
                "type": "last"
              },
              "type": "query"
            }
          ],
          "executionErrorState": "alerting",
          "for": "5m",
          "name": "Error Budget at Risk",
          "notifications": ["slack-alerts"]
        }
      },
      {
        "title": "Error Budget Burn Rate",
        "targets": [
          {
            "expr": "(sum(rate(http_requests_total{service=\"payment-service\", status=~\"5..\"}[5m])) / sum(rate(http_requests_total{service=\"payment-service\"}[5m]))) / (1 - 0.9995) / (30 * 24 * 3600) * 86400",
            "legendFormat": "Daily Burn Rate"
          },
          {
            "expr": "1",
            "legendFormat": "1x (Normal)"
          },
          {
            "expr": "14",
            "legendFormat": "14x (Critical)"
          }
        ],
        "type": "graph"
      }
    ]
  }
}
```

### 6. SLO Alert Rules

```yaml
# prometheus/alerts/slo-alerts.yml
groups:
  - name: slo_alerts
    interval: 30s
    rules:
      # Error budget at risk (25% remaining)
      - alert: ErrorBudgetAtRisk
        expr: |
          (
            1 - (
              sum(rate(http_requests_total{service="payment-service", status=~"5.."}[5m]))[30d:]
              /
              sum(rate(http_requests_total{service="payment-service"}[5m]))[30d:]
            )
          ) / (1 - 0.9995) < 0.25
        for: 5m
        labels:
          severity: warning
          slo: availability
        annotations:
          summary: "Error budget at risk for {{ $labels.service }}"
          description: "Only {{ $value }}% of error budget remaining"

      # Error budget will be exhausted soon
      - alert: ErrorBudgetExhaustionImminent
        expr: |
          (
            (
              sum(rate(http_requests_total{service="payment-service", status=~"5.."}[5m]))[30d:]
              /
              sum(rate(http_requests_total{service="payment-service"}[5m]))[30d:]
            ) / (1 - 0.9995)
            +
            (
              (
                sum(rate(http_requests_total{service="payment-service", status=~"5.."}[5m]))
                /
                sum(rate(http_requests_total{service="payment-service"}[5m]))
              ) / (1 - 0.9995) / (30 * 24 * 3600) * 14400
            )
          ) >= 1
        for: 5m
        labels:
          severity: critical
          slo: availability
        annotations:
          summary: "Error budget will be exhausted in 4 hours"
          description: "Current burn rate will consume remaining budget"

      # SLO violation
      - alert: SLOViolation
        expr: |
          (
            sum(rate(http_requests_total{service="payment-service", status!~"5.."}[5m]))[30d:]
            /
            sum(rate(http_requests_total{service="payment-service"}[5m]))[30d:]
          ) < 0.9995
        for: 1m
        labels:
          severity: critical
          slo: availability
        annotations:
          summary: "SLO violation for {{ $labels.service }}"
          description: "Availability is {{ $value }}, below SLO target of 99.95%"
```

### 7. SLO API Service

**Script ejecutable:** [`scripts/slo_api.py`](scripts/slo_api.py)

Servicio REST FastAPI para consultar el estado de SLOs y error budgets desde métricas de Prometheus.

#### Cuándo ejecutar

- **Servicio en producción:** Como microservicio corriendo continuamente
- **Integración con dashboards:** Para alimentar dashboards de Grafana u otros
- **Alertas automatizadas:** Para sistemas que necesitan consultar SLOs programáticamente
- **APIs internas:** Para que otros servicios consulten el estado de SLOs

#### Instalación

```bash
cd skills/system-reliability-engineering/slo-sli-sla
pip install -r scripts/requirements.txt
```

#### Ejecución

```bash
# Desarrollo (con auto-reload)
uvicorn scripts.slo_api:app --reload --host 0.0.0.0 --port 8000

# Producción
uvicorn scripts.slo_api:app --host 0.0.0.0 --port 8000 --workers 4

# Con Prometheus personalizado
PROMETHEUS_URL=http://prometheus:9090 uvicorn scripts.slo_api:app --reload
```

#### Endpoints

**Health Check:**
```bash
curl http://localhost:8000/health
```

**SLO Compliance:**
```bash
curl "http://localhost:8000/slo/payment-service/compliance?slo_target=0.9995&window_days=30"
```

**Error Budget Status:**
```bash
curl "http://localhost:8000/slo/payment-service/error-budget?slo_target=0.9995&window_days=30"
```

#### Documentación API

Una vez ejecutando, accede a:
- **Swagger UI:** http://localhost:8000/docs
- **ReDoc:** http://localhost:8000/redoc

#### Configuración

Variables de entorno:
- `PROMETHEUS_URL`: URL de Prometheus (default: `http://localhost:9090`)

#### Ejemplo de respuesta

```json
{
  "service": "payment-service",
  "slo_target": 0.9995,
  "current_availability": 0.9998,
  "is_compliant": true,
  "error_budget_remaining": 0.6,
  "window_days": 30,
  "burn_rate": 0.8,
  "days_to_exhaustion": 22.5,
  "status": "healthy",
  "timestamp": "2024-01-15T10:30:00"
}
```

## 🎯 Mejores Prácticas

### 1. SLI Selection

✅ **DO:**
- Choose SLIs that matter to users
- Make SLIs easy to measure
- Use percentiles for latency (p50, p95, p99)
- Align SLIs with business goals

❌ **DON'T:**
- Use internal metrics as SLIs
- Make SLIs too complex
- Ignore user experience

### 2. SLO Targets

✅ **DO:**
- Start with achievable targets
- Review and adjust regularly
- Different targets for different services
- Consider error budgets

❌ **DON'T:**
- Set targets too high initially
- Set same target for all services
- Ignore error budget consumption

### 3. Error Budgets

✅ **DO:**
- Track error budget consumption
- Alert when budget is at risk
- Use budget for release decisions
- Review budget trends

❌ **DON'T:**
- Ignore error budget consumption
- Deploy when budget is exhausted
- Set budgets too conservatively

## 🚨 Troubleshooting

### SLO Violations

1. Analyze root cause
2. Check recent changes
3. Review error patterns
4. Implement fixes
5. Document in post-mortem

### Error Budget Exhaustion

1. Stop new releases
2. Focus on stability
3. Fix known issues
4. Replenish budget before new features

## 📚 Recursos Adicionales

- [Google SRE - SLIs, SLAs, SLOs](https://sre.google/workbook/slo-document/)
- [SRE Error Budgets](https://sre.google/workbook/implementing-slos/)
- [SLO Calculator](https://www.atlassian.com/incident-management/kpis/sla-vs-slo-vs-sli)

---

**Versión:** 1.0.0
**Última actualización:** Diciembre 2025
**Total líneas:** 1,100+