1525 07 Operations Runbook 5edc8070

# 📖 Operations Runbook: terraform-e2e

![Step](https://img.shields.io/badge/Step-7c-blue?style=for-the-badge)
![Status](https://img.shields.io/badge/Status-Draft-orange?style=for-the-badge)
![Agent](https://img.shields.io/badge/Agent-As--Built-purple?style=for-the-badge)

<details open>
<summary><strong>📑 Runbook Contents</strong></summary>

- [⚡ Quick Reference](#-quick-reference)
- [📋 1. Daily Operations](#-1-daily-operations)
- [🚨 2. Incident Response](#-2-incident-response)
- [🔧 3. Common Procedures](#-3-common-procedures)
- [🕐 4. Maintenance Windows](#-4-maintenance-windows)
- [📞 5. Contacts & Escalation](#-5-contacts--escalation)
- [📝 6. Change Log](#-6-change-log)
- [References](#references)

</details>

> Generated by as-built agent | 2026-02-26

| ⬅️ Previous                                    | 📑 Index            | Next ➡️                                              |
| ---------------------------------------------- | ------------------- | ---------------------------------------------------- |
| [07-design-document.md](07-design-document.md) | [README](README.md) | [07-resource-inventory.md](07-resource-inventory.md) |

**Version**: 1.0
**Date**: 2026-02-26
**Environment**: dev
**Region**: swedencentral

---

## ⚡ Quick Reference

| Item | Value |
| ---- | ----- |
| **Primary Region** | swedencentral |
| **Resource Group** | rg-terraform-e2e-dev |
| **Frontend Endpoint** | https://app-terraform-e2e-fe-dev-3hpu.azurewebsites.net |
| **Backend Endpoint** | https://app-terraform-e2e-dev-3hpu.azurewebsites.net |
| **Support Contact** | team-terraform |
| **Escalation Path** | On-call engineer → Platform lead → Subscription owner |

### Critical Resources

| Resource | Name | Resource Group | Severity |
| -------- | ---- | -------------- | -------- |
| App Service (Frontend) | app-terraform-e2e-fe-dev-3hpu | rg-terraform-e2e-dev | 🟠 P2 |
| App Service (Backend) | app-terraform-e2e-dev-3hpu | rg-terraform-e2e-dev | 🔴 P1 |
| SQL Database | sqldb-terraform-e2e-dev | rg-terraform-e2e-dev | 🔴 P1 |
| Key Vault | kv-tfe2dev-3hpu | rg-terraform-e2e-dev | 🟠 P2 |

---

## 📋 1. Daily Operations

### 1.1 Health Checks

**Morning Health Check:**

1. ✅ Verify frontend and backend return HTTP response
2. ✅ Confirm App Services are `Running`
3. ✅ Confirm SQL server state is `Ready` and DB state is `Online`
4. ✅ Check last 24h smart detector alerts for anomalies

**KQL Query - System Health Overview:**

```kusto
requests
| where timestamp > ago(24h)
| summarize
    RequestCount = count(),
    FailedRequests = countif(success == false),
    P95DurationMs = percentile(duration, 95)
  by cloud_RoleName, bin(timestamp, 1h)
| order by timestamp desc
```

### 1.2 Log Review

**Priority Logs to Review:**

| Log Source | Query Focus | Action Threshold |
| ---------- | ----------- | ---------------- |
| Application Insights | Failed requests and dependencies | >2% failures over 15 min |
| App Service platform logs | Availability and restart events | Any repeated restart event |
| Azure SQL diagnostics | Connectivity/authentication issues | >5 failures in 15 min |

---

## 🚨 2. Incident Response

### 2.1 Severity Definitions

| Severity | Definition | Response Time |
| -------- | ---------- | ------------- |
| 🔴 P1 | Backend unavailable or SQL outage | 15 min |
| 🟠 P2 | Partial degradation (high latency, intermittent errors) | 30 min |
| 🟢 P3 | Non-critical issue (single feature impact) | 4 hours |

### 2.2 Runbooks by Alert

| Alert | Runbook | Owner |
| ----- | ------- | ----- |
| Failure Anomalies smart detector | Triage App Insights failures, then app logs and SQL connectivity | Platform on-call |
| App unavailable | Validate app runtime state, restart app if needed, rollback recent config | Platform on-call |
| SQL connectivity failures | Validate SQL server status and AAD auth configuration | Platform + DBA |

---

## 🔧 3. Common Procedures

### 3.1 Restart Services

```bash
az webapp restart --name app-terraform-e2e-dev-3hpu --resource-group rg-terraform-e2e-dev
az webapp restart --name app-terraform-e2e-fe-dev-3hpu --resource-group rg-terraform-e2e-dev
```

### 3.2 Validate Runtime State

```bash
az webapp show --name app-terraform-e2e-dev-3hpu --resource-group rg-terraform-e2e-dev --query "{name:name,state:state,httpsOnly:httpsOnly,host:defaultHostName}" -o table
az webapp show --name app-terraform-e2e-fe-dev-3hpu --resource-group rg-terraform-e2e-dev --query "{name:name,state:state,httpsOnly:httpsOnly,host:defaultHostName}" -o table
az sql server show --name sql-terraform-e2e-dev-3hpu --resource-group rg-terraform-e2e-dev --query "{name:name,state:state,publicNetworkAccess:publicNetworkAccess,version:version}" -o table
```

### 3.3 Terraform Drift Check

```bash
cd infra/terraform/terraform-e2e
terraform plan -detailed-exitcode
```

### 3.4 Secret/RBAC Validation

```bash
az keyvault show --name kv-tfe2dev-3hpu --resource-group rg-terraform-e2e-dev --query "{name:name,enableRbacAuthorization:properties.enableRbacAuthorization,publicNetworkAccess:properties.publicNetworkAccess}" -o table
az role assignment list --scope /subscriptions/00858ffc-dded-4f0f-8bbf-e17fff0d47d9/resourceGroups/rg-terraform-e2e-dev/providers/Microsoft.KeyVault/vaults/kv-tfe2dev-3hpu -o table
```

---

## 🕐 4. Maintenance Windows

| Task | Schedule | Duration |
| ---- | -------- | -------- |
| Terraform patching/updates | Saturdays 09:00 UTC | 1 hour |
| App dependency refresh | Monthly, first business Monday | 1 hour |
| DR rehearsal | Quarterly | 2 hours |

---

## 📞 5. Contacts & Escalation

| Role | Contact | Phone | On-Call Rotation |
| ---- | ------- | ----- | ---------------- |
| L1 On-Call Engineer | team-terraform | N/A | Weekly |
| L2 Platform Lead | team-terraform lead | N/A | Weekly backup |
| L3 Subscription Owner | subscription admin | N/A | Escalation only |

### Escalation Path

1. L1 triages within SLA window.
2. If unresolved in 30 minutes (P1/P2), escalate to L2.
3. If platform-wide or policy-related blocker persists, escalate to L3.

---

## 📝 6. Change Log

| Date | Change | Author |
| ---- | ------ | ------ |
| 2026-02-26 | Initial as-built runbook generated from deployed Terraform state and deployment summary | as-built agent |

---

## References

| Topic | Link |
| ----- | ---- |
| Azure Monitor Alerts | [Alerting Best Practices](https://learn.microsoft.com/azure/azure-monitor/best-practices-alerts) |
| KQL Queries | [KQL Reference](https://learn.microsoft.com/azure/azure-monitor/logs/get-started-queries) |
| Azure Service Health | [Overview](https://learn.microsoft.com/azure/service-health/overview) |
| App Service Operations | [Operations guide](https://learn.microsoft.com/azure/app-service/overview) |

---

<div align="center">

| ⬅️ [07-design-document.md](07-design-document.md) | 🏠 [Project Index](README.md) | ➡️ [07-resource-inventory.md](07-resource-inventory.md) |
| ------------------------------------------------- | ----------------------------- | ------------------------------------------------------- |

</div>