Clip Cxr Fairness Eval

---
name: clip-cxr-fairness-eval
description: Evaluates zero-shot classification performance of CLIP-based vision-language models on chest X-rays, assessing fairness across demographic subgroups (age, sex, race) and robustness to spurious correlations (presence of chest drains in pneumothorax cases). Use when the user wants to benchmark on MIMIC-CXR, or asks about evaluating this task. Reports AUPRCadj.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.21291
  bibtex_key: sourget2025fairnessrobustnessclip
  confidence: high
---

# clip-cxr-fairness-eval

> Fairness and Robustness of CLIP-Based Models for Chest X-rays — Sourget et al. (2025) (arXiv:2507.21291, 2025)

## What this evaluates

Evaluates zero-shot classification performance of CLIP-based vision-language models on chest X-rays, assessing fairness across demographic subgroups (age, sex, race) and robustness to spurious correlations (presence of chest drains in pneumothorax cases).

## Datasets

- **MIMIC-CXR** — total ?; splits: test (-1); repo https://github.com/TheoSourget/clip_cxr_fairness

## Metrics

- `AUC` — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve. Measures the model's ability to discriminate between positive and negative classes across all classification thresholds.
- `AUPRCadj` **(primary)** — range: other
  - Adjusted Area Under the Precision-Recall Curve. Computed relative to a random classifier baseline; negative values indicate performance worse than random. Accounts for class imbalance in medical imaging datasets.

## Input / output format

**Input**: Chest X-ray images and text prompts for zero-shot classification of 14 radiological findings.

**Output**: Predicted probability scores for each pathology class.

## Scoring recipe

```python
def compute_metrics(predictions, gold, subgroups=None):
    auc = compute_auc(gold, predictions)
    auprc_adj = compute_auprc(gold, predictions) - baseline_random_auprc
    results = {'AUC': auc, 'AUPRCadj': auprc_adj}
    if subgroups:
        for group in subgroups:
            mask = gold['subgroup'] == group
            results[group] = compute_metrics(predictions[mask], gold[mask])
    return results
```

## Common pitfalls

- AUPRCadj can yield negative values when model performance falls below the random classifier baseline, which may be misinterpreted as invalid scores.
- Small subgroup sample sizes (e.g., Asian patients or 18-25 age group) cause high variance and wide confidence intervals, potentially skewing fairness assessments.
- PCA visualizations of embeddings may appear to show no bias, while linear probing reveals sensitive attributes are actually encoded in the feature space.

## Evidence (verbatim from paper)

> Table[1] shows the AUC and AUPRCadj of the different models on the MIMIC-CXR test set. While the results vary across the models and subgroups, we can still see a similar pattern with gaps across patient ages. Fig.[4(a)] and [4(b)] show the results on chest X-ray with and without chest drains. We can see that all models except CXR-CLIP obtain better adjusted AUPRC on images with chest drains compared to X-rays without drains (ranging from +0.09 to +0.30)

## Citation

```bibtex
@misc{sourget2025fairnessrobustnessclip,
  title={Fairness and Robustness of CLIP-Based Models for Chest X-rays},
  author={Sourget et al. (2025)},
  year={2025},
  note={arXiv:2507.21291}
}
```

- arXiv: 2507.21291