Chexpert Atelectasis Eval

---
name: chexpert-atelectasis-eval
description: Evaluates the diagnostic accuracy of predictive algorithms and human radiologists on chest X-ray images for detecting atelectasis. It specifically probes whether human expertise provides actionable, non-redundant information on input subsets where algorithms are algorithmically indistinguishable. Use when the user wants to benchmark on CheXpert, or asks about evaluating this task. Reports Matthews Correlation Coefficient (MCC).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.00793
  bibtex_key: alur2024humanexpertise
  confidence: high
---

# chexpert-atelectasis-eval

> Human Expertise in Algorithmic Prediction — Alur et al. (2024) (arXiv:2402.00793, 2024)

## What this evaluates

Evaluates the diagnostic accuracy of predictive algorithms and human radiologists on chest X-ray images for detecting atelectasis. It specifically probes whether human expertise provides actionable, non-redundant information on input subsets where algorithms are algorithmically indistinguishable.

## Datasets

- **CheXpert** — total 224816; splits: train (224316), test (500); HF `stanfordmlgroup/chexpert`

## Metrics

- `Matthews Correlation Coefficient (MCC)` **(primary)** — range: [-1, 1]
  - Rescaled covariance between predicted and true binary labels. Computed as (TP*TN - FP*FN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN)).

## Input / output format

**Input**: Chest X-ray radiograph image

**Output**: Binary label: 1 (presence of atelectasis) or 0 (absence)

## Scoring recipe

```python
def compute_mcc(preds, gold):
    tp = sum(p == 1 and g == 1 for p, g in zip(preds, gold))
    tn = sum(p == 0 and g == 0 for p, g in zip(preds, gold))
    fp = sum(p == 1 and g == 0 for p, g in zip(preds, gold))
    fn = sum(p == 0 and g == 1 for p, g in zip(preds, gold))
    num = (tp * tn) - (fp * fn)
    den = math.sqrt((tp + fp) * (tp + fn) * (tn + fp) * (tn + fn))
    return num / den if den > 0 else 0.0
```

## Common pitfalls

- Evaluating only average performance obscures the heterogeneity where humans significantly outperform algorithms on specific indistinguishable subsets.
- Bootstrap confidence intervals for algorithmic predictors within conditional subsets are not strictly valid because subsets are selected based on the predictions themselves.

## Evidence (verbatim from paper)

> Following Rajpurkar et al., (2021), we use the Matthew’s Correlation Coefficient (MCC) as a standard measure of binary classification accuracy (Chicco and Jurman, 2020). The MCC is simply the rescaled covariance between each prediction and the outcome, which corresponds to our definition of indistinguishability (Definition 3.1).

## Citation

```bibtex
@misc{alur2024humanexpertise,
  title={Human Expertise in Algorithmic Prediction},
  author={Alur et al. (2024)},
  year={2024},
  note={arXiv:2402.00793}
}
```

- arXiv: 2402.00793