Brats Challenge Eval

---
name: brats-challenge-eval
description: Evaluates machine learning models on three clinical neuro-oncology tasks using multi-modal MRI data: multi-compartment brain tumor segmentation, tumor progression assessment, and overall patient survival prediction. Use when the user wants to benchmark on BraTS Challenge, or asks about evaluating this task. Reports Dice score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1811.02629
  bibtex_key: bakas2018brats
  confidence: high
---

# brats-challenge-eval

> Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge — Spyridon Bakas et al. (arXiv:1811.02629, 2018)

## What this evaluates

Evaluates machine learning models on three clinical neuro-oncology tasks using multi-modal MRI data: multi-compartment brain tumor segmentation, tumor progression assessment, and overall patient survival prediction.

## Datasets

- **BraTS Challenge** — total ?; splits: test (-1)

## Metrics

- `Dice score` **(primary)** — range: [0, 1]
  - Computes the overlap between predicted and ground-truth segmentation masks. Formula: 2 * |A ∩ B| / (|A| + |B|). Evaluated separately for Whole Tumor (WT), Tumor Core (TC), and Active Tumor (AT).
- `95% Hausdorff distance` — range: mm
  - Measures the maximum boundary discrepancy between predictions and ground truth, computed as the 95th percentile of pairwise distances between surface voxels to reduce outlier sensitivity.
- `Accuracy` — range: [0, 1]
  - Proportion of correctly classified patient survival outcomes among three classes.

## Input / output format

**Input**: 3D MRI volumes per patient.

**Output**: Per instance: (1) Three binary segmentation masks (WT, TC, AT) or (2) One of three survival class labels.

## Scoring recipe

```python
def dice_score(pred_mask, gold_mask):
    intersection = np.sum(pred_mask * gold_mask)
    return 2.0 * intersection / (np.sum(pred_mask) + np.sum(gold_mask))

def accuracy(pred_labels, gold_labels):
    return np.mean(pred_labels == gold_labels)

# For each tumor compartment (WT, TC, AT), compute Dice score.
# For survival task, compute Accuracy across all patients.
```

## Common pitfalls

- Dice score is highly sensitive to small tumor volumes (e.g., AT), causing average Dice to be skewed by outliers compared to median Dice.
- For the 3-class survival task, a random baseline accuracy is 0.33, not 0.0 or 0.5, which must be accounted for when interpreting performance.
- Median vs. average Dice can diverge significantly due to skewness in the distribution of results across teams.

## Evidence (verbatim from paper)

> Specifically, when evaluating WT, TC, and AT segmentation, with respect to Dice score and Hausdorff distance... The 95% Hausdorff distance metric is used to characterize the levels of robustness of the automated results... Overall, the top-5 approaches obtained an accuracy around 0.6, while the rest of teams obtained an accuracy in the range of [0.15-0.55]. We should clarify that the random choice should be considered the 0.33 since this is a 3-class classification.

## Citation

```bibtex
@misc{bakas2018brats,
  title={Identifying the Best Machine Learning Algorithms for Brain Tumor Segmentation, Progression Assessment, and Overall Survival Prediction in the BRATS Challenge},
  author={Spyridon Bakas et al.},
  year={2018},
  note={arXiv:1811.02629}
}
```

- arXiv: 1811.02629