Med Vqa Eval

---
name: med-vqa-eval
description: Evaluates multimodal models on medical visual question answering across diverse imaging modalities. It probes intrinsic visual reasoning capabilities and extrinsic biomedical knowledge grounding, while measuring the model's ability to minimize clinical hallucinations. Use when the user wants to benchmark on VQA-RAD, SLAKE, ProbMed, or asks about evaluating this task. Reports accuracy/recall (closed/open-ended).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.02328
  bibtex_key: wang2025amanda
  confidence: high
---

# med-vqa-eval

> AMANDA: Agentic Medical Knowledge Augmentation for Data-Efficient Medical Visual Question Answering — Wang et al. (2025) (arXiv:2510.02328, 2025)

## What this evaluates

Evaluates multimodal models on medical visual question answering across diverse imaging modalities. It probes intrinsic visual reasoning capabilities and extrinsic biomedical knowledge grounding, while measuring the model's ability to minimize clinical hallucinations.

## Datasets

- **VQA-RAD** — total ?; splits: test (-1)
- **SLAKE** — total ?; splits: test (-1)
- **ProbMed** — total ?; splits: test (-1)

## Metrics

- `accuracy/recall (closed/open-ended)` **(primary)** — range: percent
  - Accuracy is computed as the percentage of correct answers for closed-ended questions. Recall is computed for open-ended questions to measure the overlap between generated and reference answers.

## Input / output format

**Input**: A medical image paired with a natural language question.

**Output**: A natural language text answer.

## Scoring recipe

```python
def compute_metrics(predictions, golds, question_types):
    correct = 0
    total = 0
    for pred, gold, qtype in zip(predictions, golds, question_types):
        if qtype == 'closed':
            if pred.strip().lower() == gold.strip().lower():
                correct += 1
        elif qtype == 'open':
            # Recall metric (exact match or token overlap variant not specified)
            if match_recall(pred, gold):
                correct += 1
        total += 1
    return (correct / total) * 100
```

## Common pitfalls

- Using fixed-iteration reasoning instead of the proposed adaptive refinement mechanism degrades performance and increases computational cost.
- Treating open-ended and closed-ended questions identically; the protocol explicitly requires recall for open-ended and accuracy for closed-ended questions.
- Assuming more in-context examples always improve performance; the paper shows benefits plateau beyond an optimal point, making example quality more critical than quantity.

## Evidence (verbatim from paper)

> Following prior work, we use accuracy for closed-ended questions and recall for open-ended questions.

## Citation

```bibtex
@misc{wang2025amanda,
  title={AMANDA: Agentic Medical Knowledge Augmentation for Data-Efficient Medical Visual Question Answering},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2510.02328}
}
```

- arXiv: 2510.02328