M3da Eval

---
name: m3da-eval
description: This benchmark evaluates unsupervised domain adaptation (UDA) methods for 3D medical image segmentation across eight practical domain shifts, including inter-modality changes (MRI-CT), scanner parameters, contrast presence, and radiation dose. It measures how well models trained on a source domain can segment target domain volumes without target labels, highlighting the robustness of adaptation techniques to real-world imaging variability. Use when the user wants to benchmark on AMOS, LIDC, BraTS, CC359, or asks about evaluating this task. Reports multiclass Dice score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.17029
  bibtex_key: shirokikh2025m3da
  confidence: high
---

# m3da-eval

> M3DA: Benchmark for Unsupervised Domain Adaptation in 3D Medical Image Segmentation — Boris Shirokikh et al. (arXiv:2502.17029, 2025)

## What this evaluates

This benchmark evaluates unsupervised domain adaptation (UDA) methods for 3D medical image segmentation across eight practical domain shifts, including inter-modality changes (MRI-CT), scanner parameters, contrast presence, and radiation dose. It measures how well models trained on a source domain can segment target domain volumes without target labels, highlighting the robustness of adaptation techniques to real-world imaging variability.

## Datasets

- **AMOS** — total ?; splits: (unstated)
- **LIDC** — total ?; splits: (unstated)
- **BraTS** — total ?; splits: (unstated)
- **CC359** — total ?; splits: (unstated)

## Metrics

- `multiclass Dice score` **(primary)** — range: [0, 1]
  - Intersection over Union between predicted and ground truth segmentation masks, averaged across all foreground classes. The background label is explicitly excluded from quantification.
- `percentage of performance gap closed` — range: percent
  - 100 * (Method_Dice - Baseline_Dice) / (Oracle_Dice - Baseline_Dice). Measures how much of the performance gap between a zero-shot baseline and an Oracle model (trained on target domain) is closed by the adaptation method.

## Input / output format

**Input**: 3D medical image volumes (CT, MRI, LDCT, etc.) from a source domain with corresponding segmentation masks for training; target domain volumes for evaluation.

**Output**: Multiclass segmentation masks (excluding background) for each target domain volume.

## Scoring recipe

```python
def dice_score(pred, gt):
    intersection = np.sum(pred[gt == 1])
    union = np.sum(pred) + np.sum(gt)
    return 2 * intersection / union if union > 0 else 0.0

def compute_metrics(preds, gold, baseline_preds, oracle_preds):
    dice = np.mean([dice_score(p, g) for p, g in zip(preds, gold)])
    b_dice = np.mean([dice_score(p, g) for p, g in zip(baseline_preds, gold)])
    o_dice = np.mean([dice_score(p, g) for p, g in zip(oracle_preds, gold)])
    gap = 100 * (dice - b_dice) / (o_dice - b_dice) if (o_dice - b_dice) != 0 else 0.0
    return dice, gap
```

## Common pitfalls

- The background label is explicitly excluded from the Dice score calculation; including it will artificially lower scores.
- The 'Oracle' setup refers to training and validating directly on the target domain, not a separate pre-trained model.
- Gap closed scores can be negative if an adaptation method performs worse than the zero-shot baseline.
- Results report case-wise standard deviations in parentheses, indicating multiple runs or cross-validation folds rather than a single deterministic evaluation.

## Evidence (verbatim from paper)

> We evaluated various DA methods on the M3DA benchmark (Table 5) using multi-class Dice score and the percentage of performance gap closed between the Baseline and Oracle setups: $100 \times \frac{\text{Method}_{\text{Dice}} - \text{Baseline}_{\text{Dice}}}{\text{Oracle}_{\text{Dice}} - \text{Baseline}_{\text{Dice}}}$ .

## Citation

```bibtex
@misc{shirokikh2025m3da,
  title={M3DA: Benchmark for Unsupervised Domain Adaptation in 3D Medical Image Segmentation},
  author={Boris Shirokikh et al.},
  year={2025},
  note={arXiv:2502.17029}
}
```

- arXiv: 2502.17029