Crossmed Eval

---
name: crossmed-eval
description: Evaluates compositional generalization in medical vision-language models across a structured Modality–Anatomy–Task (MAT) schema. It probes zero-shot cross-task transfer, generalization to novel MAT combinations, and robustness under low-data regimes using a unified visual question answering interface. Use when the user wants to benchmark on CrossMed, or asks about evaluating this task. Reports top-1 classification accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.11034
  bibtex_key: singh2025crossmed
  confidence: high
---

# crossmed-eval

> CrossMed: A Multimodal Cross-Task Benchmark for Compositional Generalization in Medical Imaging — Pooja Singh et al. (2025) (arXiv:2511.11034, 2025)

## What this evaluates

Evaluates compositional generalization in medical vision-language models across a structured Modality–Anatomy–Task (MAT) schema. It probes zero-shot cross-task transfer, generalization to novel MAT combinations, and robustness under low-data regimes using a unified visual question answering interface.

## Datasets

- **CrossMed** — total 20200; splits: train (-1), val (-1), test (-1)

## Metrics

- `top-1 classification accuracy` **(primary)** — range: percent
  - Fraction of correctly predicted class labels out of total test instances, multiplied by 100.
- `segmentation class-wise Intersection-over-Union (cIoU)` — range: [0, 1]
  - Average Intersection-over-Union computed across all segmentation classes: IoU = |prediction ∩ ground_truth| / |prediction ∪ ground_truth|.

## Input / output format

**Input**: Medical imaging modality (chest X-ray, brain MRI, or chest CT) paired with a structured visual question answering (VQA) prompt following the Modality–Anatomy–Task schema.

**Output**: For classification: a discrete multi-choice class label. For segmentation: a selected mask option from a VQA-style multi-choice head.

## Scoring recipe

```python
def score(predictions, gold):
    # Classification accuracy
    correct = sum(1 for p, g in zip(predictions['class'], gold['class']) if p == g)
    acc = (correct / len(gold['class'])) * 100
    
    # Segmentation cIoU
    ious = []
    for pred_mask, gold_mask in zip(predictions['seg'], gold['seg']):
        inter = np.logical_and(pred_mask, gold_mask).sum()
        union = np.logical_or(pred_mask, gold_mask).sum()
        ious.append(inter / union if union > 0 else 0.0)
    cIoU = np.mean(ious)
    return acc, cIoU
```

## Common pitfalls

- Misinterpreting MAT evaluation splits: 'Related' shares two triplet elements, 'Unrelated' shares at most one, and 'Zero-Overlap' shares none; performance drops drastically between them.
- Confusing evaluation heads: The benchmark's primary protocol uses a VQA-style multi-choice head for segmentation to enable cross-task comparison, not a standard continuous U-Net decoder.
- Assuming standard random splits: The benchmark requires leave-one-triplet-out and zero-overlap splits that strictly factorize Modality, Anatomy, and Task, rather than random image-level splits.

## Evidence (verbatim from paper)

> We present a comprehensive evaluation of the CrossMed benchmark under five key axes: in-domain multi-task learning, compositional generalization, low-data regimes, cross-task transfer, and architectural comparisons. Our work follows a unified VQA formulation, testing generalization across Related/Unrelated MAT splits and measuring top-1 classification accuracy and segmentation class-wise Intersection-over-Union( cIoU).

## Citation

```bibtex
@misc{singh2025crossmed,
  title={CrossMed: A Multimodal Cross-Task Benchmark for Compositional Generalization in Medical Imaging},
  author={Pooja Singh et al. (2025)},
  year={2025},
  note={arXiv:2511.11034}
}
```

- arXiv: 2511.11034