Odelia Breast Mri Eval

---
name: odelia-breast-mri-eval
description: Evaluates a model's ability to classify breast MRI lesions into three clinical categories (no lesion, benign, malignant) using real-world, multi-center imaging data with high heterogeneity in scanners and protocols. It probes robustness to domain shift by comparing in-distribution cross-validation performance against an out-of-distribution test set from unseen centers. Use when the user wants to benchmark on ODELIA Breast MRI Dataset, or asks about evaluating this task. Reports Macro AUC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.00474
  bibtex_key: mullerfranzes2025odelia
  confidence: high
---

# odelia-breast-mri-eval

> A European Multi-Center Breast Cancer MRI Dataset — Müller-Franzes et al. (2025) (arXiv:2506.00474, 2025)

## What this evaluates

Evaluates a model's ability to classify breast MRI lesions into three clinical categories (no lesion, benign, malignant) using real-world, multi-center imaging data with high heterogeneity in scanners and protocols. It probes robustness to domain shift by comparing in-distribution cross-validation performance against an out-of-distribution test set from unseen centers.

## Datasets

- **ODELIA Breast MRI Dataset** — total 741; splits: test (-1), ood_test (-1); HF `ODELIA-AI/ODELIA-Challenge-2025`; repo https://github.com/mueller-franzes/odelia_breast_mri

## Metrics

- `Macro AUC` **(primary)** — range: percent
  - Area under the receiver operating characteristic curve, computed per class and averaged (macro). Values are reported as percentages.
- `Micro AUC` — range: percent
  - Area under the ROC curve computed globally across all classes by pooling predictions.
- `Sensitivity at 90% Specificity` — range: percent
  - True positive rate evaluated at a fixed true negative rate of 90%.
- `Specificity at 90% Sensitivity` — range: percent
  - True negative rate evaluated at a fixed true positive rate of 90%.

## Input / output format

**Input**: Multi-parametric breast MRI scans (images) acquired from heterogeneous clinical scanners and protocols across six European institutions.

**Output**: Three-class classification: 'No Lesion', 'Benign Lesion', or 'Malignant Lesion'.

## Scoring recipe

```python
def compute_metrics(y_true, y_prob, n_classes=3):
    # y_prob: shape (N, n_classes), y_true: shape (N,)
    auc_macro = roc_auc_score(y_true, y_prob, average='macro', multi_class='ovr')
    auc_micro = roc_auc_score(y_true, y_prob, average='micro', multi_class='ovr')
    
    sens_at_90_spec = []
    spec_at_90_sens = []
    for c in range(n_classes):
        fpr_c, tpr_c, _ = roc_curve(y_true == c, y_prob[:, c])
        idx_spec = np.argmin(np.abs(fpr_c - 0.10))
        sens_at_90_spec.append(tpr_c[idx_spec])
        idx_sens = np.argmin(np.abs(tpr_c - 0.90))
        spec_at_90_sens.append(1 - fpr_c[idx_sens])
    
    return {
        'macro_auc': np.mean(auc_macro) * 100,
        'micro_auc': np.mean(auc_micro) * 100,
        'sens_90_spec': np.mean(sens_at_90_spec) * 100,
        'spec_90_sens': np.mean(spec_at_90_sens) * 100
    }
```

## Common pitfalls

- Sensitivity and specificity are reported at fixed thresholds (90%) rather than at the optimal operating point, which differs from standard clinical reporting and requires careful threshold selection during evaluation.
- The OOD test set shows significantly degraded performance (e.g., Specificity drops to 26.7%), highlighting strong domain shift that must be explicitly accounted for when comparing models.
- Evaluation relies on 5-fold cross-validation for in-distribution data due to the small dataset size (n=741), making variance across folds a critical reporting metric.

## Evidence (verbatim from paper)

> A summary of the model's performance in classifying breast lesions is presented in Table 2. The receiver operating characteristic (ROC) curve and confusion matrix for the In-Distribution evaluation are shown in Figure 5, while those for the Out-of-Distribution test set are depicted in Figure 6. Table 2: Model's lesion classification performance. Macro AUC, Micro AUC, Sensitivity*, Specificity* ... *Sensitivity at 90% Specificity and vice versa. All values are expressed as a percentage.

## Citation

```bibtex
@misc{mullerfranzes2025odelia,
  title={A European Multi-Center Breast Cancer MRI Dataset},
  author={Müller-Franzes et al. (2025)},
  year={2025},
  note={arXiv:2506.00474}
}
```

- arXiv: 2506.00474