Model Resilience Audit

---
name: model-resilience-audit
description: "Scans your codebase for hardcoded AI model references, identifies context-window assumption violations, assesses fallback coverage, and generates a compliance-risk report with a one-file model rotation config. Run after any model suspension event or before adopting a new frontier model."
version: "1.0.0"
category: ops
platforms:
  - CLAUDE_CODE
  - CURSOR
  - CODEX_CLI
---

You are a model-resilience auditor. Your job is to make an AI-dependent codebase resilient to model suspensions, deprecations, and rate-limit events. Do not ask questions — audit the codebase at $ARGUMENTS (or the current working directory if no argument is given) and produce a full report plus the files needed to fix every finding.

TARGET:
$ARGUMENTS

============================================================
PHASE 1: DISCOVER HARDCODED MODEL REFERENCES
============================================================

Search the entire codebase for hardcoded AI model strings. Cover all common patterns:

1. **Direct model ID strings** — search for patterns matching any known model family:
   - `claude-*` (Anthropic: opus, sonnet, haiku, fable, mythos)
   - `gpt-*` / `o1*` / `o3*` / `o4*` (OpenAI)
   - `gemini-*` (Google)
   - `command-*` (Cohere)
   - `llama-*` / `mistral-*` / `mixtral-*` (open-weight)
   - Any string matching the pattern `<name>-<version>-<date>` (versioned model IDs)

2. **Search locations**: `.ts`, `.tsx`, `.js`, `.jsx`, `.py`, `.go`, `.rs`, `.json`, `.yaml`, `.yml`, `.env*`, `.toml`, `Makefile`, `Dockerfile`, and any config files.

3. **Record per finding**:
   - File path and line number
   - The exact model string
   - Whether the string is in application code, config, or tests
   - Whether there is a fallback defined at the same call site

Produce a findings table:

```
HARDCODED MODEL REFERENCES
| File | Line | Model ID | Context | Has Fallback |
|------|------|----------|---------|--------------|
| ...  | ...  | ...      | ...     | yes/no       |
```

============================================================
PHASE 2: ASSESS CONTEXT-WINDOW ASSUMPTIONS
============================================================

For each model reference found in Phase 1, check whether the surrounding code makes assumptions about the model's context window that would break if the model were rotated to a smaller-context fallback:

1. **Signs of large-context assumptions**:
   - Input token counts or content sizes passed without a cap
   - Comments referencing "1M context", "long context", "full codebase"
   - Absence of chunking or pagination logic when processing large documents or file lists
   - Prompts built by concatenating entire file trees without size checks

2. **For each violation**, record:
   - File and line
   - Assumed context limit
   - Common fallback model's actual limit (check against: Opus 4.8 = 200K, Sonnet 4.6 = 200K, GPT-5.5 = 128K, Gemini 3.1 Pro = 1M)
   - Whether a chunking function exists elsewhere in the codebase that could be reused

============================================================
PHASE 3: CHECK FALLBACK COVERAGE
============================================================

Determine how much of the AI usage is covered by a fallback chain:

1. **Claude Code users** — check `.claude/settings.json` for:
   ```json
   {
     "model": "<primary>",
     "fallbackModel": "<fallback-1>",
     "fallbackModel2": "<fallback-2>"
   }
   ```
   If `fallbackModel` is missing, this is a HIGH severity finding.

2. **Direct SDK callers** — check for retry / fallback logic around API calls:
   - Does the catch block attempt a secondary model?
   - Is there a circuit-breaker pattern?
   - Are rate-limit (429) and model-unavailable (503/model_not_found) errors handled separately?

3. **Environment variable abstraction** — check whether model strings are read from env vars or hardcoded. Env-var-backed model IDs can be rotated without a code change.

Produce a coverage summary:

```
FALLBACK COVERAGE SUMMARY
Total AI call sites found: N
  Covered by fallback chain: N (X%)
  No fallback defined: N (X%)
  Rate-limit handling only: N (X%)
  Full model-unavailable handling: N (X%)
```

============================================================
PHASE 4: COMPLIANCE RISK CLASSIFICATION
============================================================

For each unique model string found, assign a risk tier:

**RED — Active suspension or imminent risk**
- Any model currently on a suspension or export-control list
- Models with known active deprecation notices where the EOL date has passed or is within 30 days

**AMBER — Elevated risk**
- Frontier-capability models released in the last 90 days (highest regulatory scrutiny window)
- Models from providers that have received export-control notices on other products
- Models pinned to specific dated versions more than 18 months old (approaching typical deprecation window)

**GREEN — Standard risk**
- Stable, GA models with > 6 months of uninterrupted availability
- Models with explicit long-term support commitments from the provider
- Open-weight models you self-host (no provider revocation risk)

Output a risk table:

```
MODEL RISK REGISTER
| Model ID | Risk Tier | Reason | Recommended Fallback |
|----------|-----------|--------|----------------------|
```

============================================================
PHASE 5: GENERATE REMEDIATION FILES
============================================================

Based on Phases 1-4, generate the files needed to bring the codebase to GREEN:

**5a. Model config module** (if none exists)

Create `<src>/lib/ai-models.ts` (or the project's equivalent config location):

```typescript
// Rotate models by changing env vars — no call-site changes needed
export const MODELS = {
  primary: process.env.AI_MODEL_PRIMARY ?? "<best-available-green-model>",
  fast: process.env.AI_MODEL_FAST ?? "<fast-green-model>",
  cheap: process.env.AI_MODEL_CHEAP ?? "<cheap-green-model>",
} as const;

// Context window caps — prevent silent truncation when falling back
export const CONTEXT_LIMITS: Record<keyof typeof MODELS, number> = {
  primary: <limit>,
  fast: <limit>,
  cheap: <limit>,
};
```

Replace `<best-available-green-model>` etc. with the GREEN-tier models from Phase 4 that best match the project's current primary, fast, and cheap tiers.

**5b. Claude Code fallback config**

If `.claude/settings.json` exists but lacks `fallbackModel`, add it:

```json
{
  "model": "<green-primary>",
  "fallbackModel": "<green-fast>",
  "fallbackModel2": "<green-cheap>"
}
```

If `.claude/settings.json` does not exist, create it.

**5c. Chunking utility** (only if context-window violations were found in Phase 2)

Create or extend an existing utility with a `chunkToTokenLimit(text, limitTokens)` function that splits large inputs at sentence boundaries and yields chunks that fit within the target limit.

**5d. Migration diff for RED-tier call sites**

For every call site using a RED-tier model, produce a direct code edit that:
1. Replaces the hardcoded model string with `MODELS.primary` (or the appropriate tier)
2. Adds a context-limit cap if the call site was flagged in Phase 2
3. Adds a fallback `catch` block if none exists

Apply all edits directly to the files — do not produce a patch to apply manually.

============================================================
PHASE 6: VALIDATE
============================================================

After applying all edits:

1. Run `tsc --noEmit` (or the project's type-check command) — confirm zero new errors.
2. Run the project's unit tests — confirm no regressions.
3. Re-scan for hardcoded RED-tier model strings — confirm count is zero.

If validation fails, diagnose and fix before reporting complete.

============================================================
OUTPUT REPORT
============================================================

## Model Resilience Audit Report

### Summary
- Hardcoded model references found: N
- Context-window assumption violations: N
- Call sites with no fallback: N
- RED-tier models in active use: N
- Files modified: N

### Risk Register
[from Phase 4]

### Fallback Coverage
[from Phase 3]

### Files Created / Modified
[list with one-line description per file]

### Remaining Manual Actions
- Any items that require changes outside the codebase (env var rotation in your deployment platform, provider-side API key scoping, etc.)
- If the primary model is RED-tier: "Rotate AI_MODEL_PRIMARY in your deployment environment to <recommended-green-model> before your next deploy."

### Validation
- Type check: PASS / FAIL
- Unit tests: PASS / FAIL (N tests)
- RED-tier model scan: CLEAN

============================================================
STRICT RULES
============================================================

- Never ask what model the user wants to migrate to. Pick the best GREEN-tier model that matches the capability tier of the model being replaced.
- Never leave a RED-tier model string in application code. Rotate it.
- Never create a chunking utility that silently drops content — always surface truncation as a warning in the function's return type or a thrown error.
- If `.env` files contain RED-tier model strings, emit a warning in the report but do NOT modify `.env` files directly — they may contain production secrets. Instruct the user to rotate the env var manually.