Dino V2 Radiology Eval

---
name: dino-v2-radiology-eval
description: Evaluates the cross-task generalizability of the DINOv2 vision foundation model on medical image analysis tasks, specifically disease classification and organ segmentation across X-ray, CT, and MRI modalities. Use when the user wants to benchmark on NIH Chest X-ray, CheXpert, SARS-CoV-2, Brain Tumor, Montgomery County (MC), AMOS, MSD Heart, MSD Hipp, MSD Spleen, or asks about evaluating this task. Reports AUROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2312.02366
  bibtex_key: baharoon2023dino
  confidence: high
---

# dino-v2-radiology-eval

> Evaluating General Purpose Vision Foundation Models for Medical Image Analysis: An Experimental Study of DINOv2 on Radiology Benchmarks — Baharoon et al. (2023) (arXiv:2312.02366, 2023)

## What this evaluates

Evaluates the cross-task generalizability of the DINOv2 vision foundation model on medical image analysis tasks, specifically disease classification and organ segmentation across X-ray, CT, and MRI modalities.

## Datasets

- **NIH Chest X-ray** — total ?; splits: test (-1)
- **CheXpert** — total ?; splits: test (-1)
- **SARS-CoV-2** — total ?; splits: test (-1)
- **Brain Tumor** — total ?; splits: test (-1)
- **Montgomery County (MC)** — total ?; splits: test (-1)
- **AMOS** — total ?; splits: test (-1)
- **MSD Heart** — total ?; splits: test (-1)
- **MSD Hipp** — total ?; splits: test (-1)
- **MSD Spleen** — total ?; splits: test (-1)

## Metrics

- `AUROC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive and false positive rates across classification thresholds.
- `Dice/Jaccard average` — range: [0, 1]
  - The arithmetic mean of the Dice coefficient and Jaccard index (Intersection over Union), measuring overlap between predicted and ground-truth segmentation masks.

## Input / output format

**Input**: 2D or 3D medical images (X-ray, CT, MRI) resized to 224×224, 518×518, or 1024×1024 pixels depending on the model and task.

**Output**: Classification: predicted probabilities or class labels. Segmentation: pixel-wise binary or multi-class masks.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    if is_classification(gold):
        return roc_auc_score(gold, predictions)
    else:
        dice = dice_coefficient(gold, predictions)
        jaccard = jaccard_index(gold, predictions)
        return (dice + jaccard) / 2
```

## Common pitfalls

- kNN evaluation underperforms due to domain shift between natural image pre-training and medical images.
- DINOv2's positional encoding interpolation degrades performance on 224×224 inputs compared to its native 518×518 pre-training size.
- SAM classification requires averaging patch embeddings as a CLS token substitute, which is non-standard.

## Evidence (verbatim from paper)

> We used the area under the operating receiver curve (AUROC) as a performance metric for classification tasks, and the average of the dice and jaccard scores as a metric for segmentation.

## Citation

```bibtex
@misc{baharoon2023dino,
  title={Evaluating General Purpose Vision Foundation Models for Medical Image Analysis: An Experimental Study of DINOv2 on Radiology Benchmarks},
  author={Baharoon et al. (2023)},
  year={2023},
  note={arXiv:2312.02366}
}
```

- arXiv: 2312.02366