Explainability

---
name: explainability
description: >
  Produce or audit the interpretability/explainability analysis of a medical-imaging model —
  Grad-CAM / Grad-CAM++ / attention-rollout / saliency / integrated-gradients — so it clears the rigor
  bar a reviewer expects: mandatory Adebayo sanity checks (model- and data-randomisation), a
  quantitative localisation metric against ground truth (IoU / pointing game / Dice) instead of eyeballed
  examples, a cohort-level result rather than cherry-picked cases, and attribution framing rather than
  "proof the model is correct". Emits an explainability-report manifest and a deterministic rigor gate.
  Integrates captum / pytorch-grad-cam; it does not reimplement them, and never runs a model on real
  patient data.
triggers: explainability, interpretability, saliency, saliency map, grad-cam, gradcam, grad-cam++, attention map, attention rollout, integrated gradients, captum, pytorch-grad-cam, heatmap, class activation map, CAM, feature attribution, sanity check, Adebayo, model randomization, localization metric, pointing game, IoU with ground truth, XAI, explainable AI, model looks at, faithfulness
tools: Read, Write, Edit, Bash, Grep, Glob
model: inherit
---

# Explainability Skill

## Purpose

A saliency / Grad-CAM heat-map is the **most over-interpreted artifact** in medical-imaging AI: a
colourful map over the lesion is routinely presented as proof the model "looks at the right thing."
Adebayo et al. (*NeurIPS* 2018) showed many saliency methods produce visually convincing maps that are
**independent of the model's learned weights and of the labels** — so they explain nothing. This skill
produces an explainability analysis that clears the rigor bar, and audits an existing one, so the map
is trustworthy before it reaches a manuscript (CLAIM 2024 / TRIPOD+AI interpretability items).

It sits alongside evaluation in the lane: `/architecture-zoo` → `/preprocess-imaging` →
`/model-scaffold` → `/model-validation` → `/model-evaluation` + **explainability** →
`/write-paper` + `/check-reporting`. It **integrates** captum / pytorch-grad-cam (referenced in the
plan); it does not reimplement them and never runs a model on real patient data.

## When to use
- You produced (or are about to produce) saliency / Grad-CAM / attention maps and want them reported
  to the standard a reviewer expects.
- You want to audit an explainability analysis for the four failure modes below.

## When NOT to use
- Discrimination / calibration metrics → `/model-evaluation` then `/analyze-stats`.
- Split or preprocessing leakage → `/model-validation` / `/preprocess-imaging`.
- LLM/MLLM faithfulness & hallucination → `/mllm-eval`.
- Reimplementing captum / pytorch-grad-cam → out of scope (this skill wires and audits them).

## The four failure modes (what the gate enforces)
1. **Saliency as validation.** A map is *attribution*, not proof the model is correct or that the
   relationship is causal. Frame it as "where signal is attributed", never as "the model is right".
2. **No sanity check.** Run the Adebayo **model-randomisation** and **data-randomisation** tests. A map
   that survives neither is uninterpretable; both axes are the minimum bar.
3. **No quantitative localisation.** If you claim the map localises the finding, measure it — IoU /
   pointing game / Dice against ground-truth masks — do not eyeball a few examples.
4. **Cherry-picked examples.** Report a cohort-level result, not a handful of hand-picked cases.

## Workflow

### Phase 1 — Produce the maps (integrate, don't reimplement)
Choose the method for the architecture (`references/explainability_guide.md`): Grad-CAM / Grad-CAM++
for CNNs, attention-rollout for ViTs, integrated-gradients / SHAP for attribution. Wire captum or
pytorch-grad-cam; do not write a new CAM implementation.

### Phase 2 — Sanity-check and quantify
- Run the **model-parameter randomisation** and **data (label) randomisation** tests (Adebayo 2018);
  a faithful map degrades when the model/labels are randomised.
- Compute a **quantitative localisation metric** against ground-truth masks (IoU / pointing game /
  Dice) over the cohort — not a visual impression.

### Phase 3 — Emit the explainability-report manifest
```json
{
  "method": "grad-cam++",
  "n_examples": 200,
  "cohort_level": true,
  "localization_metric": "iou",
  "localization_value": 0.63,
  "sanity_checks": ["model_randomization", "data_randomization"],
  "interpretation": "localization"
}
```
`interpretation`: `attribution` / `localization` / `faithfulness` (descriptive) — never
`validation` / `causal` (overclaim).

### Phase 4 — Gate the report (deterministic)
```bash
python3 scripts/check_explainability_report.py --manifest explainability_report.json --strict
```
Verdicts: `SALIENCY_AS_VALIDATION`, `NO_SANITY_CHECK`, `NO_LOCALIZATION_METRIC` (Major);
`INSUFFICIENT_SANITY`, `CHERRY_PICKED_EXAMPLES`, `MISSING_METHOD` (Minor). The verdict is reproduced
by rule on the manifest, never asserted from prose.

## Integration
- **`/model-evaluation`** — explainability accompanies the held-out metrics as a secondary analysis.
- **`/self-review`** `ai_overclaiming` / `image_synthesis` probes audit saliency overclaiming in a
  finished manuscript; this skill *produces* the rigorous analysis they look for.
- **`/check-reporting`** — the manifest documents the CLAIM 2024 / TRIPOD+AI interpretability items.

## Anti-Hallucination

- **Never fabricate saliency maps, localisation metrics, or sanity-check results.** Every value in the
  manifest comes from the researcher's executed XAI code — never invented. This skill designs and
  audits the analysis; it does not run a model on real patient data.
- **Never present a saliency map as proof of model correctness or causation.** A map is attribution;
  claiming it validates the model is the overclaim this skill exists to prevent (`SALIENCY_AS_VALIDATION`).
- **Never report an explainability-audit "pass" without running `check_explainability_report.py`.** The
  rigor verdict is reproduced deterministically, never asserted from prose.
- **Integrate, don't reimplement.** Reference captum / pytorch-grad-cam; do not write a new CAM /
  attribution implementation or claim results for one.

## Reproducible challenge
`scripts/check_explainability_report_challenge/` ships a synthetic weak/strong report pair with a
network-free `verify.sh` wired into the skill's validation commands.