Vqa Biomedical Eval

---
name: vqa-biomedical-eval
description: Evaluates multimodal models on biomedical visual question answering tasks. It probes the model's ability to interpret medical images (e.g., X-rays, pathology slides) and generate accurate answers to clinical or radiological questions in both open-ended and closed-ended formats. Use when the user wants to benchmark on VQA-RAD, SLAKE, PathVQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.02900
  bibtex_key: xie2024medtrinity25m
  confidence: high
---

# vqa-biomedical-eval

> MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine — Yunfei Xie et al. (2024) (arXiv:2408.02900, 2024)

## What this evaluates

Evaluates multimodal models on biomedical visual question answering tasks. It probes the model's ability to interpret medical images (e.g., X-rays, pathology slides) and generate accurate answers to clinical or radiological questions in both open-ended and closed-ended formats.

## Datasets

- **VQA-RAD** — total ?; splits: test (-1)
- **SLAKE** — total ?; splits: test (-1)
- **PathVQA** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted answers out of the total number of questions. Calculated separately for open-ended and closed-ended questions, and averaged across both.

## Input / output format

**Input**: A medical image (e.g., radiograph or pathology slide) paired with a natural language question.

**Output**: A natural language answer string (for open-ended) or a selected option/class label (for closed-ended).

## Scoring recipe

```python
correct = 0
total = 0
for pred, gold in zip(predictions, gold_answers):
    if normalize_text(pred) == normalize_text(gold):
        correct += 1
    total += 1
return (correct / total) * 100
```

## Common pitfalls

- Open-ended answers are often evaluated via exact string matching, which may penalize valid paraphrases or minor formatting differences.
- Closed-ended questions are sometimes treated as classification over the training vocabulary, which can overestimate generalizability as noted in the paper's footnote.
- Fine-tuning is strictly limited to 15 epochs per dataset; deviating from this may yield different results.

## Evidence (verbatim from paper)

> We benchmark LLaVA-Tri on three biomedical Visual Question Answering (VQA) datasets, VQA-RAD*(Lau et al., [2018a])*, SLAKE*(Liu et al., [2021])*, and PathVQA*(He et al., [2020a])*, to assess the efficacy of aligning the model using MedTrinity-25M. Following LLaVA-Med, The model is fine-tuned for 15 epochs on each of the three VQA datasets and evaluated accordingly. As shown in [Table 3], LLaVA-Tri achieved state-of-the-art results in all of the three VQA benchmarks, with 75.0% accuracy on VQA-RAD, 87.8% on SLAKE, and 65.3% on PathVQA.

## Citation

```bibtex
@misc{xie2024medtrinity25m,
  title={MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine},
  author={Yunfei Xie et al. (2024)},
  year={2024},
  note={arXiv:2408.02900}
}
```

- arXiv: 2408.02900