Nova Eval

---
name: nova-eval
description: Evaluates large vision-language models' ability to detect, localize, and reason about rare brain MRI anomalies under extreme clinical and semantic distribution shifts. It probes zero-shot generalization across localization, descriptive captioning, and diagnostic classification without closed-set assumptions. Use when the user wants to benchmark on NOVA, or asks about evaluating this task. Reports Top-1 accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.14064
  bibtex_key: bercea2025nova
  confidence: high
---

# nova-eval

> NOVA: A Benchmark for Anomaly Localization and Clinical Reasoning in Brain MRI — Bercea et al. (2025) (arXiv:2505.14064, 2025)

## What this evaluates

Evaluates large vision-language models' ability to detect, localize, and reason about rare brain MRI anomalies under extreme clinical and semantic distribution shifts. It probes zero-shot generalization across localization, descriptive captioning, and diagnostic classification without closed-set assumptions.

## Datasets

- **NOVA** — total 906; splits: test (-1)

## Metrics

- `Top-1 accuracy` **(primary)** — range: [0, 1]
  - Fraction of cases where the model's single highest-confidence diagnostic label matches the ground truth label.
- `mAP@50` — range: [0, 1]
  - Mean Average Precision at Intersection over Union (IoU) threshold of 0.50, averaging precision across all recall levels for detected anomalies.
- `Clinical Term F1` — range: [0, 1]
  - F1 score computed over the presence of clinically relevant diagnostic terms in generated captions compared to ground truth reports.
- `BLEU` — range: [0, 1]
  - Bilingual Evaluation Understudy score measuring n-gram overlap between model-generated captions and reference reports.
- `Top-5 accuracy` — range: [0, 1]
  - Fraction of cases where the ground truth label appears within the model's top 5 highest-confidence diagnostic predictions.

## Input / output format

**Input**: Single brain MRI scan image; for diagnostic reasoning, optionally accompanied by clinical history/metadata.

**Output**: Bounding box coordinates for localization; free-text clinical caption for description; single or ranked diagnostic label(s) for reasoning.

## Scoring recipe

```python
def top1_accuracy(predictions, gold_labels):
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
    return correct / len(gold_labels)
```

## Common pitfalls

- Models frequently generate false-positive bounding boxes on normal anatomical structures (e.g., misinterpreting the orbital cavity as a lesion).
- Diagnostic predictions collapse onto a compressed label space (~30% coverage), failing to recognize rare pathologies present in the ground truth.
- Proprietary models may have encountered similar cases in training data, meaning reported scores represent an upper bound on true zero-shot generalization.

## Evidence (verbatim from paper)

> Models were evaluated using standard object detection metrics (mAP@30, mAP@50, and mAP@[50:95]), as summarized in Table 1. Performance was assessed via Top-1 and Top-5 classification accuracy (Table 3).

## Citation

```bibtex
@misc{bercea2025nova,
  title={NOVA: A Benchmark for Anomaly Localization and Clinical Reasoning in Brain MRI},
  author={Bercea et al. (2025)},
  year={2025},
  note={arXiv:2505.14064}
}
```

- arXiv: 2505.14064