Slake Med Vqa Eval

---
name: slake-med-vqa-eval
description: Evaluates medical visual question answering capabilities by testing a model's ability to reason over radiology images (CT/MRI/X-ray) to answer vision-only and knowledge-based questions in English and Chinese. It probes multimodal fusion, semantic segmentation utilization, and external medical knowledge graph integration for clinical reasoning. Use when the user wants to benchmark on SLAKE, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2102.09542
  bibtex_key: liu2021slake
  confidence: high
---

# slake-med-vqa-eval

> SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering — Liu et al. (2021) (arXiv:2102.09542, 2021)

## What this evaluates

Evaluates medical visual question answering capabilities by testing a model's ability to reason over radiology images (CT/MRI/X-ray) to answer vision-only and knowledge-based questions in English and Chinese. It probes multimodal fusion, semantic segmentation utilization, and external medical knowledge graph integration for clinical reasoning.

## Datasets

- **SLAKE** — total ?; splits: train (-1), test (-1)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted answers out of the total number of questions. Calculated separately for vision-only (open-ended and closed-ended) and knowledge-based questions, and further broken down by language (English, Chinese).

## Input / output format

**Input**: A radiology image (CT/MRI/X-ray) paired with a bilingual question (English or Chinese). For knowledge-based tasks, question-related embeddings from a medical knowledge graph are also provided.

**Output**: For closed-ended questions: a selection from multiple-choice options. For open-ended questions: free-form text answers.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Open-ended questions require careful string normalization or exact matching, as free-form answers are inherently harder to evaluate than multiple-choice selections.
- The dataset stratifies questions into vision-only and knowledge-based categories; evaluating them together without stratification masks significant performance differences.
- Bilingual evaluation requires separate tokenization and embedding pipelines for English and Chinese, which can cause cross-lingual leakage if not strictly isolated.

## Evidence (verbatim from paper)

> We report the results for vision-only and knowledge-based questions in Table 4 and Table 5 respectively. Answers of “closed-ended” questions are limited multiple-choice options, while answers of “open-ended” questions are free-form texts. Open-ended questions are generally harder to answer than closed-ended ones. In Table 4, we report the results in accuracy for vision-only questions in both English and Chinese.

## Citation

```bibtex
@misc{liu2021slake,
  title={SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering},
  author={Liu et al. (2021)},
  year={2021},
  note={arXiv:2102.09542}
}
```

- arXiv: 2102.09542