Multimodal Medical Seg Eval

---
name: multimodal-medical-seg-eval
description: Evaluates the ability of self-supervised multimodal pretraining to learn modality-agnostic representations for downstream medical image segmentation and survival prediction tasks, particularly under conditions of non-registered data and low-data regimes. Use when the user wants to benchmark on BraTS (Multimodal Brain Tumor Image Segmentation Benchmark), Medical Segmentation Decathlon (Prostate), CHAOS (Liver), or asks about evaluating this task. Reports Dice coefficient (segmentation), Concordance index (survival).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1912.05396
  bibtex_key: taleb2019multimodal
  confidence: medium
---

# multimodal-medical-seg-eval

> Multimodal Self-Supervised Learning for Medical Image Analysis — Taleb et al. (2019) (arXiv:1912.05396, 2019)

## What this evaluates

Evaluates the ability of self-supervised multimodal pretraining to learn modality-agnostic representations for downstream medical image segmentation and survival prediction tasks, particularly under conditions of non-registered data and low-data regimes.

## Datasets

- **BraTS (Multimodal Brain Tumor Image Segmentation Benchmark)** — total 351; splits: train (285), val (66)
- **Medical Segmentation Decathlon (Prostate)** — total 48; splits: train (32), test (16)
- **CHAOS (Liver)** — total 40; splits: train (20), test (20)

## Metrics

- `Dice coefficient (segmentation), Concordance index (survival)` **(primary)** — range: [0, 1]
  - Dice coefficient measures spatial overlap between predicted and ground-truth segmentation masks: 2|A∩B|/(|A|+|B|). Concordance index evaluates ranking consistency for survival predictions. Exact formulas are not specified in the excerpt but follow standard medical imaging conventions.

## Input / output format

**Input**: Multimodal medical imaging volumes or patches (e.g., T1, T1Gd, T2, T2-FLAIR MRI; CT; ADC maps) from registered or non-registered patient scans.

**Output**: Pixel-wise segmentation masks for target anatomical regions (brain tumor, prostate zones, liver) or predicted survival days.

## Scoring recipe

```python
# For segmentation tasks
pred_mask = model.predict(input_volumes)
dice = 2 * np.sum(pred_mask * gt_mask) / (np.sum(pred_mask) + np.sum(gt_mask))
# For survival prediction
survival_pred = model.predict(input_volumes)
c_index = concordance_index(survival_times, survival_pred, event_observed)
```

## Common pitfalls

- Non-registered modalities in CHAOS require modality-agnostic handling rather than simple pixel-wise alignment.
- Low-data regime evaluation is highly sensitive to fine-tuning epochs and learning rates.
- Survival prediction uses continuous days, so regression or ranking metrics are required instead of classification accuracy.

## Evidence (verbatim from paper)

> The BraTS challenge involves two different tasks: i) brain tumor segmentation, and ii) number of survival days prediction. The prostate dataset consists of 48 multimodal MRI cases, from which 32 cases are used for training, and 16 are used for testing. The CHAOS dataset consists of 40 multimodal cases, from which 20 cases are used for training, and 20 for testing.

## Citation

```bibtex
@misc{taleb2019multimodal,
  title={Multimodal Self-Supervised Learning for Medical Image Analysis},
  author={Taleb et al. (2019)},
  year={2019},
  note={arXiv:1912.05396}
}
```

- arXiv: 1912.05396