Cxrlt 2026 Eval

---
name: cxrlt-2026-eval
description: Evaluates robust multi-label classification under long-tailed class distributions and open-world zero-shot generalization to unseen rare diseases in chest X-rays. Use when the user wants to benchmark on PadChest + NIH, or asks about evaluating this task. Reports mAP.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.15555
  bibtex_key: dong2026cxrlt
  confidence: high
---

# cxrlt-2026-eval

> CXR-LT 2026 Challenge: Multi-Center Long-Tailed and Zero Shot Chest X-ray Classification — Dong et al. (2026) (arXiv:2604.15555, 2026)

## What this evaluates

Evaluates robust multi-label classification under long-tailed class distributions and open-world zero-shot generalization to unseen rare diseases in chest X-rays.

## Datasets

- **PadChest + NIH** — total 145000; splits: train (-1), val (-1), test (-1)

## Metrics

- `mAP` **(primary)** — range: [0, 1]
  - Mean Average Precision across all disease classes. Computed as the average of per-class Average Precision scores, which measure the area under the precision-recall curve for each class.
- `AUROC` — range: [0, 1]
  - Area Under the Receiver Operating Characteristic curve, measuring ranking performance across all classification thresholds.
- `F1 score` — range: [0, 1]
  - Harmonic mean of precision and recall at a fixed decision threshold.
- `ECE` — range: [0, 1]
  - Expected Calibration Error, measuring the discrepancy between predicted confidence and actual accuracy. Reported as 1-ECE to align directionality with other metrics.
- `precision` — range: [0, 1]
  - Ratio of true positive predictions to all positive predictions at a fixed threshold.
- `recall` — range: [0, 1]
  - Ratio of true positive predictions to all actual positives at a fixed threshold.

## Input / output format

**Input**: Chest X-ray radiograph images.

**Output**: Multi-label probability scores or logits for each disease class.

## Scoring recipe

```python
def compute_mAP(preds, gold, num_classes):
    ap_scores = []
    for c in range(num_classes):
        p = preds[:, c]
        g = gold[:, c]
        order = np.argsort(-p)
        p, g = p[order], g[order]
        ap = average_precision_score(g, p)
        ap_scores.append(ap)
    return np.mean(ap_scores)
```

## Common pitfalls

- High AUROC does not guarantee good threshold-dependent metrics (F1/recall), as teams used different confidence thresholds.
- Calibration (ECE) is often overlooked; high ranking metrics can coexist with poorly calibrated probabilities, which is critical for clinical use.
- Task 2 zero-shot evaluation uses unseen classes split from the same training pool (24 seen, 6 unseen), requiring careful handling of data leakage during prompt/prototype construction.

## Evidence (verbatim from paper)

> Model performance is primarily evaluated using mean Average Precision (mAP) for ranking, while additional metrics (AUROC, F1 score, ECE, precision, and recall) are reported to provide complementary perspectives on model behavior. Confidence intervals are estimated using 1,000 bootstrap samples.

## Citation

```bibtex
@misc{dong2026cxrlt,
  title={CXR-LT 2026 Challenge: Multi-Center Long-Tailed and Zero Shot Chest X-ray Classification},
  author={Dong et al. (2026)},
  year={2026},
  note={arXiv:2604.15555}
}
```

- arXiv: 2604.15555