Medfair Eval

---
name: medfair-eval
description: Evaluates group fairness and bias mitigation in medical imaging models across multiple datasets and modalities. It probes whether models trained with Empirical Risk Minimization (ERM) or explicit bias mitigation algorithms exhibit performance disparities across sensitive subgroups, and how model selection strategies impact worst-case group performance. Use when the user wants to benchmark on MEDFAIR Benchmark Suite, or asks about evaluating this task. Reports worst-case AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2210.01725
  bibtex_key: zong2022medfair
  confidence: high
---

# medfair-eval

> MEDFAIR: Benchmarking Fairness for Medical Imaging — Zong et al. (2022) (arXiv:2210.01725, 2022)

## What this evaluates

Evaluates group fairness and bias mitigation in medical imaging models across multiple datasets and modalities. It probes whether models trained with Empirical Risk Minimization (ERM) or explicit bias mitigation algorithms exhibit performance disparities across sensitive subgroups, and how model selection strategies impact worst-case group performance.

## Datasets

- **MEDFAIR Benchmark Suite** — total ?; splits: train (-1), test (-1); repo https://github.com/ys-zong/MEDFAIR

## Metrics

- `worst-case AUC` **(primary)** — range: [0, 1]
  - The Area Under the Receiver Operating Characteristic Curve computed on the sensitive subgroup with the lowest performance. It serves as the headline fairness metric to ensure no group is left behind.
- `AUC gap` — range: [0, 1]
  - The difference between the maximum and minimum AUC across all sensitive subgroups (max(AUC) - min(AUC)). A smaller gap indicates better group fairness.
- `overall AUC` — range: [0, 1]
  - The standard Area Under the ROC Curve computed across the entire dataset without subgroup stratification, used as a baseline for overall diagnostic performance.
- `underdiagnosis rate` — range: [0, 1]
  - Defined as the False Negative Rate (FNR) for malignant labels, or the False Positive Rate (FPR) for 'No Finding' labels, calculated per subgroup to capture clinical error direction.

## Input / output format

**Input**: Medical images paired with diagnostic labels and sensitive demographic attributes (e.g., age, sex, race).

**Output**: Continuous prediction scores or probabilities for the diagnostic task, aggregated per sensitive subgroup for metric computation.

## Scoring recipe

```python
def compute_metrics(preds, labels, subgroups):
    subgroup_aucs = [auc_score(preds[s], labels[s]) for s in subgroups]
    worst_auc = min(subgroup_aucs)
    auc_gap = max(subgroup_aucs) - min(subgroup_aucs)
    overall_auc = auc_score(preds, labels)
    underdiag = fnr(preds, labels) if label=='malignant' else fpr(preds, labels)
    return worst_auc, auc_gap, overall_auc, underdiag
```

## Common pitfalls

- Using standard overall-AUC model selection instead of worst-case or Pareto selection, which masks severe subgroup disparities and leads to poor worst-group performance.
- Misinterpreting underdiagnosis rate: FNR is used for malignant labels, but FPR is used for 'No Finding' labels, reversing the typical error direction depending on the task.
- Drawing conclusions from insufficient dataset counts; the paper emphasizes using Nemenyi post-hoc tests across many datasets to establish statistical significance for fairness claims.

## Evidence (verbatim from paper)

> For each dataset and sensitive attribute, we calculate the maximum and minimum AUC and underdiagnosis rate among subgroups, where we use FNR for the malignant label and FPR for "No Finding" label as the underdiagnosis rate.

## Citation

```bibtex
@misc{zong2022medfair,
  title={MEDFAIR: Benchmarking Fairness for Medical Imaging},
  author={Zong et al. (2022)},
  year={2022},
  note={arXiv:2210.01725}
}
```

- arXiv: 2210.01725