Notion Incident Runbook

---
name: notion-incident-runbook
description: "| Execute Notion incident response procedures with triage, mitigation, and postmortem. Use when responding to Notion API outages, investigating errors, or running post-incident reviews for Notion integration failures. Trigger with phrases like \"notion incident\", \"notion outage\", \"notion down\", \"notion on-call\", \"notion emergency\", \"notion broken\"."
allowed-tools: "Read, Bash(kubectl:*), Bash(curl:*)"
category: engineering-core
source_repo: jeremylongshore/claude-code-plugins-plus-skills
source_path: "skills/.curated/notion-incident-runbook/SKILL.md"
source_url: https://github.com/jeremylongshore/claude-code-plugins-plus-skills/blob/HEAD/skills/.curated/notion-incident-runbook/SKILL.md
---

# Notion Incident Runbook

## Overview

Rapid incident response for Notion API failures. This runbook drives a sub-5-minute
triage that classifies the failure as Notion-side vs. integration-side, then routes to
per-error-type mitigation, cached fallback, and a structured postmortem.

Deep material — full triage scripts, remediation code, and templates — lives in
`references/`; this file is the high-level flow you follow live.

## Prerequisites

- Access to application monitoring dashboards and log aggregator
- `NOTION_TOKEN` environment variable set for diagnostic API calls
- `curl` and `jq` installed for quick CLI triage
- Python alternative: `notion-client` (`pip install notion-client`)
- Communication channels configured (Slack webhook, PagerDuty, etc.)

## Instructions

### Step 1: Quick Triage (Under 5 Minutes)

At first alert, decide whether the fault is Notion's or yours. Check the platform
status page, then test your own auth. This one call is enough to start:

```bash
curl -sf -o /dev/null -w "%{http_code}" \
  https://api.notion.com/v1/users/me \
  -H "Authorization: Bearer ${NOTION_TOKEN}" \
  -H "Notion-Version: 2022-06-28"   # 2022-06-28 = pinned Notion API version
```

Map the result: `200` → integration-side, `401` → token expired/revoked, `429` →
rate limited, `000` → network/DNS. If `status.notion.so` shows an active incident, it
is Notion-side regardless of your auth code.

Read [references/triage.md](references/triage.md) for the full `notion-triage.sh`
diagnostic (status page + auth + DB-query latency + auto-classification) and a
TypeScript `triageNotionHealth()` equivalent for in-app health checks.

### Step 2: Decision Tree and Mitigation

Route the classification to a remediation path:

- **Notion-side outage** — enable cached/fallback mode, notify users, monitor the
  status page. Do NOT restart or rotate tokens.
- **401 token expired/revoked** — regenerate at `notion.so/my-integrations`, update the
  secret manager, restart the app.
- **429 rate limited** — you are exceeding the 3 req/s average; find runaway loops or
  webhook storms, drop concurrency to 1, add exponential backoff.
- **404 on known resources** — pages unshared or trashed; re-share via the Connections menu.
- **400 validation errors** — the database schema changed in the UI; re-fetch with
  `databases.retrieve` and update property mappings.

Read [references/mitigation.md](references/mitigation.md) for the full decision tree,
token-rotation commands (AWS/GCP Secret Manager + `kubectl` restart), the
`queryWithFallback()` cached-fallback pattern, and `detectSchemaChanges()`.

### Step 3: Communication and Postmortem

Post an internal Slack update on every state change (INVESTIGATING → MITIGATING →
RESOLVED), an external status-page notice if users are impacted, and file a structured
postmortem once resolved. Copy-paste templates for all three are in
[references/communication-and-postmortem.md](references/communication-and-postmortem.md).

## Output

- Automated triage script classifying incidents in under 5 minutes
- Decision tree mapping HTTP status codes to root causes
- Per-error-type mitigation procedures with real code
- Cached fallback mode for Notion outages
- Schema change detection for 400 validation errors
- Communication templates for internal and external stakeholders
- Postmortem template with timeline and action items

## Error Handling

| Scenario | Triage Signal | Immediate Action |
| ---------- | -------------- | ------------------ |
| Notion platform outage | status.notion.so incident | Enable fallback mode, notify users |
| Token expired/revoked | All requests return 401 | Rotate token in secret manager, restart |
| Rate limited | 429 errors spiking | Reduce concurrency to 1, check for loops |
| Schema changed | 400 on specific operations | Run `databases.retrieve`, update mappings |
| Network/DNS issue | Timeouts, no HTTP response | Check firewall, DNS resolution, proxy config |
| Pages unshared | 404 on previously working pages | Re-share via Connections menu in Notion |

## Examples

### One-Line Health Check

```bash
curl -sf https://api.notion.com/v1/users/me \
  -H "Authorization: Bearer ${NOTION_TOKEN}" \
  -H "Notion-Version: 2022-06-28" \
  | jq '{name: .name, type: .type}' \
  || echo "UNHEALTHY: Notion API unreachable or auth failed"
```

### Python Quick Triage

```python
from notion_client import Client, APIResponseError
import os

def quick_triage():
    try:
        client = Client(auth=os.environ["NOTION_TOKEN"], timeout_ms=10_000)
        me = client.users.me()
        print(f"OK: Connected as {me['name']}")
    except APIResponseError as e:
        print(f"ERROR: {e.code} (HTTP {e.status}): {e.message}")
    except Exception as e:
        print(f"NETWORK ERROR: {e}")

quick_triage()
```

For the full `notion-triage.sh` diagnostic and the TypeScript in-app variant, see
[references/triage.md](references/triage.md).

## Resources

- [Notion Status Page](https://status.notion.so) — real-time platform status
- [Notion API Error Codes](https://developers.notion.com/reference/errors) — full error reference
- [Notion Request Limits](https://developers.notion.com/reference/request-limits) — 3 req/s average
- [Statuspage API](https://www.atlassianstatuspage.io/api) — programmatic status checks
- For data handling and privacy compliance, see the `notion-data-handling` skill.

---

**Source:** [`jeremylongshore/claude-code-plugins-plus-skills`](https://github.com/jeremylongshore/claude-code-plugins-plus-skills) → `skills/.curated/notion-incident-runbook/SKILL.md`

**Also appears in:** `jeremylongshore/claude-code-plugins-plus-skills/plugins/saas-packs/notion-pack/skills/notion-incident-runbook/SKILL.md`