Patch Selectivity Eval

---
name: patch-selectivity-eval
description: Evaluates a model's ability to ignore out-of-context patches (patch selectivity) and maintain classification accuracy under simulated occlusion and spatial permutation attacks. Use when the user wants to benchmark on ImageNet-1K val, SMD, NVD, ROD, or asks about evaluating this task. Reports Top-1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.17848
  bibtex_key: lee2023hardwiring
  confidence: high
---

# patch-selectivity-eval

> Hardwiring ViT Patch Selectivity into CNNs using Patch Mixing — Lee et al. (2023) (arXiv:2306.17848, 2023)

## What this evaluates

Evaluates a model's ability to ignore out-of-context patches (patch selectivity) and maintain classification accuracy under simulated occlusion and spatial permutation attacks.

## Datasets

- **ImageNet-1K val** — total ?; splits: val (-1)
- **SMD** — total ?; splits: test (-1)
- **NVD** — total ?; splits: val (-1)
- **ROD** — total ?; splits: test (-1)

## Metrics

- `Top-1 accuracy` **(primary)** — range: percent
  - Standard classification accuracy: the fraction of correctly predicted top-1 classes out of total evaluated instances.
- `Top-5 accuracy` — range: percent
  - Fraction of instances where the true label appears in the model's top-5 predicted classes.
- `Inverse patch selectivity` — range: [0, 1]
  - Sum of Softmax-normalized c-RISE importance values over out-of-context patches. Lower values indicate better selectivity.

## Input / output format

**Input**: Natural images from benchmark datasets, optionally subjected to patch replacement, grid shuffling, or occlusion attacks during evaluation.

**Output**: Class predictions (top-1 or top-5 ranked classes) and, for explainability analysis, c-RISE importance heatmaps normalized via Softmax.

## Scoring recipe

```python
def compute_accuracy(preds, gold):
    correct = sum(1 for p, g in zip(preds, gold) if p == g)
    return correct / len(gold)

def compute_inverse_patch_selectivity(rise_maps, ooc_mask):
    norm_maps = softmax(rise_maps, dim=0)
    return sum(norm_maps[ooc_mask])
```

## Common pitfalls

- Confusing the training-time Patch Mixing augmentation with the evaluation-time patch replacement/shuffle attacks.
- Mixing up Top-1 and Top-5 accuracy metrics, as the paper reports Top-1 for IN/SMD but Top-5 for NVD/ROD.
- Assuming ViTs inherently lack occlusion robustness without testing under controlled information loss.

## Evidence (verbatim from paper)

> Table 1 presents a summary of the results for different network architectures tested on three datasets: ImageNet-1K val (IN) top-1, SMD top-1 (avg. over 10 - 30% occlusion), NVD [25] simulated occlusion validation top-5, and ROD top-5.

## Citation

```bibtex
@misc{lee2023hardwiring,
  title={Hardwiring ViT Patch Selectivity into CNNs using Patch Mixing},
  author={Lee et al. (2023)},
  year={2023},
  note={arXiv:2306.17848}
}
```

- arXiv: 2306.17848