Camchex Eval

---
name: camchex-eval
description: Evaluates a multimodal framework's ability to classify thoracic diseases at the study level by jointly modeling multi-view chest X-rays, clinical indications, and vital signs. It probes the model's capacity to integrate heterogeneous clinical data for accurate multi-label diagnosis across head, body, and tail disease categories. Use when the user wants to benchmark on MIMIC-CXR, CXR-LT 2023, CXR-LT 2024, or asks about evaluating this task. Reports mAP.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.09581
  bibtex_key: sloan2025camchex
  confidence: high
---

# camchex-eval

> Clinically-aligned Multi-modal Chest X-ray Classification — Sloan et al. (2025) (arXiv:2511.09581, 2025)

## What this evaluates

Evaluates a multimodal framework's ability to classify thoracic diseases at the study level by jointly modeling multi-view chest X-rays, clinical indications, and vital signs. It probes the model's capacity to integrate heterogeneous clinical data for accurate multi-label diagnosis across head, body, and tail disease categories.

## Datasets

- **MIMIC-CXR** — total 377110; splits: test (-1)
- **CXR-LT 2023** — total ?; splits: test (-1)
- **CXR-LT 2024** — total ?; splits: test (-1)

## Metrics

- `mAP` **(primary)** — range: [0, 1]
  - Mean Average Precision: computes the average of the Average Precision scores across all classes, averaging over varying decision thresholds. It measures performance across thresholds and does not degrade under class imbalance.
- `AUROC (macro)` — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve (macro): calculates the AUC for each class independently and averages them. Can be disproportionately influenced by long-tailed class imbalance.

## Input / output format

**Input**: Per study: frontal and lateral chest X-ray images (1024×1024), clinical indication text, and structured vital signs (temperature, heart rate, respiration rate, blood pressure).

**Output**: Multi-label probability vector over 14 categories (MIMIC-CXR) or 26 categories (CXR-LT) representing common thoracic diseases and findings.

## Scoring recipe

```python
def compute_mAP(y_true, y_pred, num_classes):
    aps = [average_precision_score(y_true[:, c], y_pred[:, c]) for c in range(num_classes)]
    return np.mean(aps)

def compute_AUROC(y_true, y_pred, num_classes):
    aucs = [roc_auc_score(y_true[:, c], y_pred[:, c]) for c in range(num_classes)]
    return np.mean(aucs)
```

## Common pitfalls

- AUROC is heavily skewed by long-tail class imbalance, making mAP the preferred primary metric for CXR-LT benchmarks.
- Evaluation is performed at the study level (aggregating frontal and lateral views), not at the individual image level.
- MIMIC-CXR labels are derived using the CheXpert rule-based extractor, which may introduce noise or differ from manual ground truth.

## Evidence (verbatim from paper)

> In line with the CXR-LT benchmarks, we evaluate CaMCheX with commonly used metrics, mAP and AUC-ROC (macro). The primary evaluation metric is mAP since AUC-ROC can be disproportionately influenced due to long tailed class imbalance. In contrast, mAP measures performance across decision thresholds and does not degrade under class imbalance.

## Citation

```bibtex
@misc{sloan2025camchex,
  title={Clinically-aligned Multi-modal Chest X-ray Classification},
  author={Sloan et al. (2025)},
  year={2025},
  note={arXiv:2511.09581}
}
```

- arXiv: 2511.09581