Audit Codemod Safety

---
name: audit-codemod-safety
description: >
  Read-only audit of a codemod or bulk mechanical transform for
  behavior-preservation — compiles/lints is not same-behavior. Use when "did
  this codemod break anything", "audit this bulk refactor", or before merging
  a mass find-replace. Diff quality → audit-code-review. SQL → plan-data-integrity.
license: MIT
---

# audit-codemod-safety — Compiles and lints is not behaves the same

**Degree of freedom: MIXED** — Phases 0–2 `[HIGH freedom]`; Phase 3
high-risk hand-trace `[LOW freedom — run exactly]`. Do not re-run the
transform.

Read-only. A codemod's danger is scale plus plausibility: it touches hundreds
of files and each diff looks trivially correct, so review fatigue waves it
through. A mechanical transform proves **syntax, not semantics**.

Present findings. Do not re-run the transform or edit the diffs. Hand fixes
to a follow-up session.

## This skill vs neighbors

| Skill | Owns |
|---|---|
| **audit-codemod-safety** (this) | Source-transform correctness at scale — behavior, completeness, consistency |
| `audit-code-review` | General PR / diff quality on a named change set |
| `plan-data-integrity` | Data / SQL / destructive migrations — not JS/TS source mods |
| `audit-code-quality` | Repo-wide anti-patterns, not one mechanical pass |
| `burndown-full` | Drive a leftover MATCH pattern to zero — this *audits* whether the pass was safe |
| `audit-gate-logic` | If size defeated review and syntax defeated linters, that is also a gate-logic finding |

## How to reason

1. **Observe** — quote the rule, the before/after AST or text, and a concrete diff hunk
2. **Interpret** — does the swap change runtime meaning, or only spelling?
3. **Classify** — silent behavior change / incomplete migration / consistent-as-is / needs-a-probe
4. **Severity** — silent runtime change on a hot path = critical

## Worked example

> **Observe:** codemod rewrote `value || fallback` → `value ?? fallback` in
> `pricing.ts`. Tests still pass.
> **Interpret:** `0` and `''` now keep the left side; price `0` no longer
> falls back. Syntax-clean; semantics changed.
> **Classify:** silent behavior change (truthiness vs nullish).
> **Severity:** critical if money path; else major.
> **Finding:** `pricing.ts:41` | `||`→`??` | critical | `0` no longer defaults

---

## Phase 0 — Characterize the transform  [HIGH freedom]

Establish what was done and how:

- Tool: jscodeshift / ts-morph / ast-grep / comby / regex `sed` / AI bulk edit
- Rule in plain terms ("replace X API with Y", "rename Z", "new import path")
- Blast radius (files touched, LOC)
- AST-based (structure-aware) vs text/regex (no scope awareness — higher risk)

Get the diff **and** the codemod script if it exists. Auditing the *rule*
often finds the gap faster than reading 300 diffs.

---

## Phase 1 — Behavior-preservation  [HIGH freedom]

**Semantic-equivalence traps** — syntactic swap, different runtime meaning:

- Operator / precedence shifts (replacement needed parens and did not get them)
- Truthiness vs nullish (`||` → `??` or reverse changes `0` / `''` / `false`)
- Async/await vs `.then` that drop error propagation or change ordering
- Reference vs value, mutate-in-place vs copy, shallow vs deep
- Import swaps with different defaults, side effects, or tree-shaking
- Type-only changes that alter *runtime* (`enum` → `const`, `interface` →
  `type` where declaration merging mattered)

**Comment / string / dynamic false hits** (regex sweeps especially) — pattern
matched inside strings, comments, template literals, or dynamic property
access. Text mods cannot tell code from a string that looks like code.

**Formatting-masked logic** — a reformat (Prettier bump, quote style) bundled
into the same commit as the codemod, so a real behavior change hides inside
thousands of whitespace diffs. Flag if logic and formatting were not
separated.

---

## Phase 2 — Coverage and consistency  [HIGH freedom]

**Missed cases** — matched the common shape, not the variants: alias,
re-export, dynamic import, HOC-wrapped, JSX vs call expression, tests,
`.vue` / `.svelte` / `.mdx` the parser never reached. Grep the *old*
pattern's semantic footprint — not just the exact string. Survivors mean a
half-done migration (two conventions live at once), which is worse than none.

**Inconsistent application** — same construct transformed one way here,
another way there (passes drifted, or the AI drifted). Cross-check
`audit-code-quality`.

**Generated / vendored / ignored** — touched generated code that will be
overwritten, or skipped files the runtime still loads because the ignore
list the codemod respected is not the runtime's.

**Orphans** — old helpers / imports / types the transform was supposed to
remove but left dangling; or new ones it referenced but never created.

---

## Phase 3 — Verify, don't trust  [LOW freedom — hand-trace high-risk files]

- **Test behavior, not compilation.** Did tests exercise the transformed
  paths, or do they pass because coverage there is thin? Cross-check
  `plan-test-coverage`. A codemod over untested code is unverified — say so.
- **Pair with visual / behavioral diffing** where UI is involved
  (`test-visual-regression`). Mechanical JSX / style mods are prime
  silent-break territory.
- **Gate interaction.** Did this large mechanical change slip a real bug
  past CI because its size defeated review and its syntax defeated the
  linters? If so, that is also a finding for `audit-gate-logic`.
- **Spot-probe.** Trace the highest-risk transformed files (auth, money,
  data mutation) by hand end-to-end. Scale review effort by blast radius,
  not evenly.

---

## Definition of Done

- [ ] Transform characterized: tool, rule, blast radius, AST vs text
- [ ] Codemod script / rule reviewed where available
- [ ] Semantic-equivalence traps checked (precedence, truthiness, async, refs, import semantics, type-affecting-runtime)
- [ ] Regex / text mods checked for matches inside strings / comments / dynamic access
- [ ] Formatting confirmed separated from logic (or the coupling flagged)
- [ ] Old-pattern footprint grepped for survivors → migration completeness assessed
- [ ] Inconsistent application across the repo surfaced
- [ ] Generated / vendored / ignored file handling verified
- [ ] Orphaned / dangling symbols found
- [ ] Transformed paths confirmed actually tested; high-risk files hand-traced
- [ ] Gate-bypass interaction noted for `audit-gate-logic` if the mod hid a bug past CI
- [ ] Read-only — findings presented, code not edited

## Self-critique before reporting  [LOW freedom — do not skip]

1. **Evidenced** — quoted hunk or surviving old-pattern grep, not "looks fine"
2. **Semantics not syntax** — compiles/lints is not a pass
3. **Severity justified** — critical = silent runtime change
4. **Right owner** — CI waved it through → also `audit-gate-logic`
5. **No re-run** — findings only; do not re-apply the transform

## Output format

1. **Transform summary** — tool | rule | files touched | AST or text | tests covering the paths?
2. **Behavior-change findings** — file:line | trap type | severity (critical = silent runtime change) | why the diff looks fine but isn't
3. **Coverage / consistency findings** — surviving old-pattern instances | inconsistent conversions | orphans
4. **Verification log** — high-risk files traced | result; test-coverage verdict on transformed paths
5. **Fix plan** — silent behavior changes first, then migration completeness, then consistency; handoffs to `test-visual-regression`, `plan-test-coverage`, `audit-gate-logic`, a fix session. Present and stop.