Benchread Eval

---
name: benchread-eval
description: Evaluates retinal anomaly detection models across fundus photography and OCT modalities, testing their ability to distinguish normal from abnormal images and generalize to unseen anomalies under varying supervision levels. Use when the user wants to benchmark on Fundus Benchmark, OCT Benchmark, or asks about evaluating this task. Reports AUC-ROC.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.10492
  bibtex_key: lian2025benchread
  confidence: high
---

# benchread-eval

> BenchReAD: A systematic benchmark for retinal anomaly detection — Lian et al. (2025) (arXiv:2507.10492, 2025)

## What this evaluates

Evaluates retinal anomaly detection models across fundus photography and OCT modalities, testing their ability to distinguish normal from abnormal images and generalize to unseen anomalies under varying supervision levels.

## Datasets

- **Fundus Benchmark** — total ?; splits: train (39749), val (200), test (3027); repo https://github.com/DopamineLcy/BenchReAD
- **OCT Benchmark** — total ?; splits: train (79542), val (200), test (3606); repo https://github.com/DopamineLcy/BenchReAD

## Metrics

- `AUC-ROC` **(primary)** — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, computed by plotting the true positive rate against the false positive rate across all classification thresholds on the test set.

## Input / output format

**Input**: Raw retinal images (fundus photography or OCT scans) provided as input to the anomaly detection model.

**Output**: Continuous anomaly scores for each image, which are thresholded to produce binary normal/abnormal predictions.

## Scoring recipe

```python
def compute_auc(predictions, labels):
    fpr, tpr, _ = roc_curve(labels, predictions)
    return auc(fpr, tpr)
```

## Common pitfalls

- Models trained on both normal and abnormal samples tend to overfit to seen anomaly features, leading to poor generalization on unseen anomalies.
- Evaluation must separately report performance on seen versus unseen anomaly categories to properly assess generalization capabilities.
- Strict partitioning of training data into labeled and unlabeled subsets is required for fair comparison across supervision levels.

## Evidence (verbatim from paper)

> Figure 2: ROC curves of distinguishing normal samples against all abnormal ones on test sets. Corresponding AUCs (%) are marked alongside 95% confidence intervals.

## Citation

```bibtex
@misc{lian2025benchread,
  title={BenchReAD: A systematic benchmark for retinal anomaly detection},
  author={Lian et al. (2025)},
  year={2025},
  note={arXiv:2507.10492}
}
```

- arXiv: 2507.10492