Qu Brats Score

---
name: qu-brats-score
description: Evaluates the calibration of voxel-wise uncertainty estimates in brain tumor segmentation. It measures how effectively uncertainty thresholds filter out incorrect predictions while preserving correct ones, rewarding high confidence in accurate regions and penalizing the loss of correct predictions when filtering uncertain voxels. Use when the user has predictions and gold and needs to compute QU-BraTS unified score.
metadata:
  skill_kind: metric
  source_arxiv: 2112.10074
  bibtex_key: mehta2021qu_brats
  confidence: high
---

# qu-brats-score

> QU-BraTS: MICCAI BraTS 2020 Challenge on Quantifying Uncertainty in Brain Tumor Segmentation - Analysis of Ranking Scores and Benchmarking Results — Mehta et al. (2021) (arXiv:2112.10074, 2021)

## What this evaluates

Evaluates the calibration of voxel-wise uncertainty estimates in brain tumor segmentation. It measures how effectively uncertainty thresholds filter out incorrect predictions while preserving correct ones, rewarding high confidence in accurate regions and penalizing the loss of correct predictions when filtering uncertain voxels.

## Datasets

- **BraTS** — total ?; splits: train (228), val (57), test (50); repo https://github.com/RagMeh11/QU-BraTS

## Metrics

- `QU-BraTS unified score` **(primary)** — range: [0, 1]
  - Computes the Area Under the Curve (AUC) for three metrics across uncertainty thresholds τ: Dice Similarity Coefficient (DSC), Filtered True Positives ratio (FTP), and Filtered True Negatives ratio (FTN). FTP = (TP_100 - TP_τ) / TP_100, FTN = (TN_100 - TN_τ) / TN_100. The final score is (AUC_DSC + (1 - AUC_FTP) + (1 - AUC_FTN)) / 3.
- `DSC` — range: [0, 1]
  - Dice Similarity Coefficient measuring voxel-wise overlap between predicted and ground truth segmentation masks.

## Input / output format

**Input**: 3D MRI volumes with ground truth binary masks for tumor entities (ET, TC, WT) and predicted voxel-wise uncertainty maps normalized to [0, 100].

**Output**: Predicted segmentation masks and corresponding voxel-wise uncertainty values for each tumor entity.

## Scoring recipe

```python
def compute_qu_brats_score(pred_masks, gt_masks, uncertainty_maps, thresholds=[25, 50, 75, 100]):
    dsc_vals, ftp_vals, ftm_vals = [], [], []
    for tau in thresholds:
        filtered_pred = mask_where_uncertainty_lt(pred_masks, uncertainty_maps, tau)
        dsc_vals.append(dice_score(filtered_pred, gt_masks))
        tp_100 = count_true_positives(pred_masks, gt_masks)
        tp_tau = count_true_positives(filtered_pred, gt_masks)
        tn_100 = count_true_negatives(pred_masks, gt_masks)
        tn_tau = count_true_negatives(filtered_pred, gt_masks)
        ftp_vals.append((tp_100 - tp_tau) / tp_100)
        ftm_vals.append((tn_100 - tn_tau) / tn_100)
    auc_dsc = np.trapz(dsc_vals, thresholds)
    auc_ftp = np.trapz(ftp_vals, thresholds)
    auc_ftn = np.trapz(ftm_vals, thresholds)
    return (auc_dsc + (1 - auc_ftp) + (1 - auc_ftn)) / 3
```

## Common pitfalls

- Filtering out correct predictions (TPs/TNs) to artificially inflate DSC is heavily penalized by the FTP/FTN ratios.
- Uncertainty maps must be strictly normalized to the 0-100 range before applying thresholds.
- Evaluation is performed separately for each tumor entity (ET, TC, WT) due to clinical relevance and class imbalance.

## Evidence (verbatim from paper)

> Finally, the resulting uncertainty measures for each team are ranked according to a unified score which combines the area under three curves: 1) DSC vs τ, 2) FTP vs τ, and 3) FTN vs τ, for different values of τ. The unified score is calculated as follows: score_tumor_entity = (AUC1 + (1 - AUC2) + (1 - AUC3)) / 3.

## Citation

```bibtex
@misc{mehta2021qu_brats,
  title={QU-BraTS: MICCAI BraTS 2020 Challenge on Quantifying Uncertainty in Brain Tumor Segmentation - Analysis of Ranking Scores and Benchmarking Results},
  author={Mehta et al. (2021)},
  year={2021},
  note={arXiv:2112.10074}
}
```

- arXiv: 2112.10074