Chestxray14 Bias Eval

---
name: chestxray14-bias-eval
description: Evaluates the effectiveness of attribute-neutralization models in removing demographic bias (sex and age) from chest X-ray images while preserving diagnostic utility for 15 disease findings. It probes the trade-off between demographic leakage suppression and clinical performance across varying edit intensities. Use when the user wants to benchmark on ChestX-ray14, or asks about evaluating this task. Reports AI-Judge AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.15490
  bibtex_key: solomon2026hybrid
  confidence: high
---

# chestxray14-bias-eval

> Hybrid Vision Transformer_GAN Attribute Neutralizer for Mitigating Bias in Chest X_Ray Diagnosis — Jobeal Solomon, Ali Mohammed Mansoor Alsahag, Seyed Sahand Mohammadi Ziabari (2026) (arXiv:2601.15490, 2026)

## What this evaluates

Evaluates the effectiveness of attribute-neutralization models in removing demographic bias (sex and age) from chest X-ray images while preserving diagnostic utility for 15 disease findings. It probes the trade-off between demographic leakage suppression and clinical performance across varying edit intensities.

## Datasets

- **ChestX-ray14** — total ?; splits: test (1000)

## Metrics

- `AI-Judge AUC` **(primary)** — range: [0, 1]
  - Area under the receiver operating characteristic curve computed between the ground-truth demographic attribute (sex or age) and the AI-Judge's predicted probability on the neutralized image. Measures attribute leakage.
- `Macro ROC-AUC` — range: [0, 1]
  - Mean of the ROC-AUC scores computed independently for each of the 15 disease findings. Measures overall diagnostic performance on neutralized images.
- `Worst-case subgroup AUC` — range: [0, 1]
  - Minimum ROC-AUC across demographic subgroups (e.g., male/female, <60/≥60) for each disease finding, then median across findings. Measures fairness and subgroup performance.
- `SSIM` — range: [0, 1]
  - Structural Similarity Index measuring pixel-level similarity between original and neutralized images. Used to quantify the magnitude of image edits.

## Input / output format

**Input**: Chest X-ray radiographs annotated with patient sex, age, and 15 disease findings. Images are processed through attribute neutralizers at varying edit intensities ($\alpha$ from 0.0 to 1.0).

**Output**: Neutralized chest X-ray images, AI-Judge predictions (probability of sex/age), and disease diagnosis predictions (15 binary findings).

## Scoring recipe

```python
def compute_metrics(original_img, neutralized_img, ai_judge, diagnosis_model, y_true_attr, y_true_disease):
    y_pred_attr = ai_judge.predict(neutralized_img)
    y_pred_disease = diagnosis_model.predict(neutralized_img)
    
    # Leakage
    leakage_auc = roc_auc_score(y_true_attr, y_pred_attr)
    
    # Diagnosis
    macro_auc = np.mean([roc_auc_score(y_true_disease[:, i], y_pred_disease[:, i]) for i in range(15)])
    
    # Fairness
    subgroup_aucs = [roc_auc_score(y_true_disease, y_pred_disease) for group in demographic_groups]
    worst_auc = min(subgroup_aucs)
    
    return leakage_auc, macro_auc, worst_auc
```

## Common pitfalls

- Confusing edit-intensity parameter $\alpha$ with model capability; higher $\alpha$ does not linearly improve neutrality and can collapse diagnostic performance.
- Overlooking class imbalance when interpreting per-finding ROC-AUC distributions, which can mask significant performance drops on rare findings.
- Comparing CNN and ViT neutralizers without noting the CNN baseline showed almost no leakage reduction across all $\alpha$ levels, making direct $\alpha$-sweep comparisons misleading.

## Evidence (verbatim from paper)

> Leakage was assessed at three $\alpha$-levels (0.0, 0.5, 1.0) across three generator variants. Table 2 shows that at the turning-point edit ($\alpha=0.5$), the ViT neutralizer lowers the AI-Judge’s AUC by $\geq 0.15$ relative to both Hu et al.’s U-Net and our own CNN retrain, while the CNN variant hardly changes.

## Citation

```bibtex
@misc{solomon2026hybrid,
  title={Hybrid Vision Transformer_GAN Attribute Neutralizer for Mitigating Bias in Chest X_Ray Diagnosis},
  author={Jobeal Solomon, Ali Mohammed Mansoor Alsahag, Seyed Sahand Mohammadi Ziabari (2026)},
  year={2026},
  note={arXiv:2601.15490}
}
```

- arXiv: 2601.15490