Biomed Vqa Eval

---
name: biomed-vqa-eval
description: Evaluates the domain-adaptive post-training of multimodal large language models on biomedical visual question answering tasks, measuring how well models generalize to specialized medical domains using both open and closed evaluation splits. Use when the user wants to benchmark on SLAKE, PathVQA, VQA-RAD, PMC-VQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.19930
  bibtex_key: cheng2024domainadaptiveposttraining
  confidence: medium
---

# biomed-vqa-eval

> On Domain-Adaptive Post-Training for Multimodal Large Language Models — Cheng et al. (2024) (arXiv:2411.19930, 2024)

## What this evaluates

Evaluates the domain-adaptive post-training of multimodal large language models on biomedical visual question answering tasks, measuring how well models generalize to specialized medical domains using both open and closed evaluation splits.

## Datasets

- **SLAKE** — total ?; splits: open (-1), closed (-1)
- **PathVQA** — total ?; splits: open (-1), closed (-1)
- **VQA-RAD** — total ?; splits: open (-1), closed (-1)
- **PMC-VQA** — total ?; splits: open (-1), closed (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Standard VQA accuracy: percentage of predicted answers that exactly match or semantically align with the ground-truth answers. Reported as percentage scores in Table 2.

## Input / output format

**Input**: Image paired with a question (instruction) in a conversational format.

**Output**: Text answer (precise or informative response) generated by the model.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_ans in zip(predictions, gold):
        if normalize(pred) in [normalize(g) for g in gold_ans]:
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Confusing the 'open' and 'closed' evaluation splits, which have different data distributions and baseline scores.
- Failing to account for potential data contamination when using synthetic domain-adaptive training data that may overlap with test sets.

## Evidence (verbatim from paper)

> Table 2: Biomedicine Task Performance of general MLLMs and MLLMs after domain-adaptive post-training. The image-caption sources for AdaMLLM from PMC^Raw and AdaMLLM from PMC^Refined are PMC^Raw and PMC^Refined, respectively.

## Citation

```bibtex
@misc{cheng2024domainadaptiveposttraining,
  title={On Domain-Adaptive Post-Training for Multimodal Large Language Models},
  author={Cheng et al. (2024)},
  year={2024},
  note={arXiv:2411.19930}
}
```

- arXiv: 2411.19930