Temmed Bench Eval

---
name: temmed-bench-eval
description: Evaluates large vision-language models' ability to perform temporal reasoning on medical images by analyzing condition changes across multiple clinical visits. It probes capabilities in visual question answering, longitudinal clinical report generation, and selecting relevant image pairs based on temporal context. Use when the user wants to benchmark on TemMed-Bench, or asks about evaluating this task. Reports Avg..
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.25143
  bibtex_key: zhang2025temmedbench
  confidence: high
---

# temmed-bench-eval

> TemMed-Bench: Evaluating Temporal Medical Image Reasoning in Vision-Language Models — Zhang et al. (2025) (arXiv:2509.25143, 2025)

## What this evaluates

Evaluates large vision-language models' ability to perform temporal reasoning on medical images by analyzing condition changes across multiple clinical visits. It probes capabilities in visual question answering, longitudinal clinical report generation, and selecting relevant image pairs based on temporal context.

## Datasets

- **TemMed-Bench** — total 17000; splits: test (-1)

## Metrics

- `VQA Accuracy` — range: percent
  - Percentage of instances where the model's predicted answer exactly matches the ground-truth answer.
- `VQA F1` — range: percent
  - Token-level or exact-match F1 score between the predicted and ground-truth answers.
- `Report Generation BLEU` — range: percent
  - Standard BLEU n-gram overlap score between the generated report and the ground-truth clinical report.
- `Report Generation ROUGE-L` — range: percent
  - ROUGE-L recall/precision score based on the longest common subsequence between generated and ground-truth reports.
- `Report Generation METEOR` — range: percent
  - METEOR score incorporating synonymy, stemming, and paraphrase matching between generated and ground-truth reports.
- `Image Selection Accuracy` — range: percent
  - Percentage of instances where the model correctly selects the target image pair based on temporal condition changes.
- `Avg.` **(primary)** — range: percent
  - Arithmetic mean of VQA Accuracy, Report Generation BLEU, ROUGE-L, METEOR, and Image Selection Accuracy.

## Input / output format

**Input**: A pair of medical images (historical and current visit) accompanied by a text prompt or question.

**Output**: Text answer (VQA), generated clinical report (Report Generation), or a selected image pair (Image Selection).

## Scoring recipe

```python
def score(predictions, golds):
    vqa_acc = sum(1 for p, g in zip(predictions['vqa'], golds['vqa']) if p == g) / len(golds['vqa'])
    vqa_f1 = compute_f1(golds['vqa'], predictions['vqa'])
    rep_bleu = compute_bleu(golds['report'], predictions['report'])
    rep_rouge = compute_rouge_l(golds['report'], predictions['report'])
    rep_meteor = compute_meteor(golds['report'], predictions['report'])
    img_acc = sum(1 for p, g in zip(predictions['img'], golds['img']) if p == g) / len(golds['img'])
    avg = (vqa_acc + rep_bleu + rep_rouge + rep_meteor + img_acc) / 5
    return {'vqa_acc': vqa_acc, 'vqa_f1': vqa_f1, 'rep_bleu': rep_bleu, 'rep_rouge': rep_rouge, 'rep_meteor': rep_meteor, 'img_acc': img_acc, 'avg': avg}
```

## Common pitfalls

- Random-guess baselines are surprisingly high (33.3% for VQA and Image Selection), so models scoring near this threshold are effectively failing at temporal reasoning.
- Retrieval augmentation can degrade performance on the image-pair selection task due to attention splitting and retrieval noise when aligning with multiple target pairs.
- Medical fine-tuning can erode general reasoning capabilities, causing specialized medical LVLMs to underperform general-domain models on this benchmark.

## Evidence (verbatim from paper)

> For VQA, we use accuracy and F1 score as metrics. For report generation, following Jing et al. (2018) and Xia et al. (2025), we use BLEU (Papineni et al., 2002), ROUGE-L (Lin, 2004), and METEOR (Banerjee & Lavie, 2005). For image-pair selection, accuracy is used.

## Citation

```bibtex
@misc{zhang2025temmedbench,
  title={TemMed-Bench: Evaluating Temporal Medical Image Reasoning in Vision-Language Models},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2509.25143}
}
```

- arXiv: 2509.25143