Medsg Bench Eval

---
name: medsg-bench-eval
description: This benchmark evaluates sequential visual grounding in medical imaging, specifically testing a model's ability to perform cross-image semantic alignment, detect differences between sequential scans, and identify consistent regions across time. Use when the user wants to benchmark on MedSG-Bench, or asks about evaluating this task. Reports average IoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.11852
  bibtex_key: yue2025medsgbench
  confidence: high
---

# medsg-bench-eval

> MedSG-Bench: A Benchmark for Medical Image Sequences Grounding — Jingkun Yue et al. (2025) (arXiv:2505.11852, 2025)

## What this evaluates

This benchmark evaluates sequential visual grounding in medical imaging, specifically testing a model's ability to perform cross-image semantic alignment, detect differences between sequential scans, and identify consistent regions across time.

## Datasets

- **MedSG-Bench** — total 9630; splits: test (9630)

## Metrics

- `average IoU` **(primary)** — range: [0, 1]
  - Intersection over Union between the predicted grounding region (bounding box or mask) and the ground truth region, averaged across all samples.
- `ACC@0.5` — range: [0, 1]
  - Accuracy thresholded at IoU=0.5, representing the fraction of samples where the predicted region achieves an IoU of at least 0.5 with the ground truth.

## Input / output format

**Input**: A sequence of medical images (multiple modalities/views) paired with a text prompt/question requiring spatial localization or difference/consistency detection.

**Output**: Predicted bounding box coordinates or segmentation mask indicating the grounded region in the image sequence.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    ious = []
    for pred, gold in zip(predictions, golds):
        intersection = len(pred & gold)
        union = len(pred | gold)
        ious.append(intersection / union if union > 0 else 0.0)
    avg_iou = sum(ious) / len(ious)
    acc_05 = sum(1 for i in ious if i >= 0.5) / len(ious)
    return avg_iou, acc_05
```

## Common pitfalls

- Evaluations must be strictly zero-shot; providing in-context examples or fine-tuning on the benchmark violates the protocol.
- Medical-domain specialized models often underperform general-purpose models on this task due to catastrophic forgetting of spatial grounding capabilities during domain-specific instruction tuning.
- Model scale and release recency do not guarantee better grounding performance; many recent models are optimized for high-level semantic tasks rather than fine-grained spatial alignment.

## Evidence (verbatim from paper)

> In this study, we evaluate model performance under a zero-shot setting, where the models were prompted to perform inference without access to in-context examples. We use average Intersection over Union (IoU) and ACC@0.5 as the evaluation metric.

## Citation

```bibtex
@misc{yue2025medsgbench,
  title={MedSG-Bench: A Benchmark for Medical Image Sequences Grounding},
  author={Jingkun Yue et al. (2025)},
  year={2025},
  note={arXiv:2505.11852}
}
```

- arXiv: 2505.11852