Modad Eval

---
name: modad-eval
description: Evaluates a model's ability to mitigate spurious correlations (bias) in image classification by measuring performance on both overall test sets and specifically on bias-conflicting samples where the spurious attribute contradicts the true label. Use when the user wants to benchmark on Corrupted CIFAR-10, BAR, BFFHQ, Waterbirds, or asks about evaluating this task. Reports Average Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.17449
  bibtex_key: pastore2024modad
  confidence: high
---

# modad-eval

> Looking at Model Debiasing through the Lens of Anomaly Detection — Pastore et al. (2024) (arXiv:2407.17449, 2024)

## What this evaluates

Evaluates a model's ability to mitigate spurious correlations (bias) in image classification by measuring performance on both overall test sets and specifically on bias-conflicting samples where the spurious attribute contradicts the true label.

## Datasets

- **Corrupted CIFAR-10** — total 60000; splits: train (-1), val (-1), test (-1)
- **BAR** — total 2595; splits: train (1941), test (654)
- **BFFHQ** — total 21200; splits: train (19200), val (1000), test (1000)
- **Waterbirds** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `Average Accuracy` **(primary)** — range: percent
  - The fraction of correctly classified samples over the entire test set, which typically contains a balanced number of bias-aligned and conflicting samples.
- `Conflicting Accuracy` — range: percent
  - The fraction of correctly classified samples restricted strictly to the bias-conflicting subset of the test set.

## Input / output format

**Input**: RGB image tensor paired with a ground-truth semantic class label.

**Output**: Predicted semantic class label.

## Scoring recipe

```python
def compute_metrics(predictions, gold_labels, is_conflicting_mask):
    avg_acc = (predictions == gold_labels).mean() * 100
    conflicting_acc = (predictions[is_conflicting_mask] == gold_labels[is_conflicting_mask]).mean() * 100
    return avg_acc, conflicting_acc
```

## Common pitfalls

- BAR dataset lacks ground-truth bias annotations, so Conflicting Accuracy cannot be computed for it; only Average Accuracy is reported.
- Test sets for BFFHQ and Waterbirds are explicitly balanced (50% bias-aligned, 50% bias-conflicting), unlike the highly skewed training sets, which heavily influences Average Accuracy.
- Conflicting Accuracy must be computed strictly on samples where the spurious attribute contradicts the true semantic label; misidentifying bias-aligned samples as conflicting will artificially deflate this metric.

## Evidence (verbatim from paper)

> We employ two quantitative metrics in our results: Average Accuracy and Conflicting Accuracy. Average Accuracy is the accuracy over the whole test set, which typically contains a balanced number of bias-aligned and conflicting samples, except otherwise stated. Conflicting Accuracy refers to the average accuracy computed only on the bias-conflicting data.

## Citation

```bibtex
@misc{pastore2024modad,
  title={Looking at Model Debiasing through the Lens of Anomaly Detection},
  author={Pastore et al. (2024)},
  year={2024},
  note={arXiv:2407.17449}
}
```

- arXiv: 2407.17449