Medqa Usmle Eval

---
name: medqa-usmle-eval
description: Evaluates medical multiple-choice question answering capability of 4B-parameter LLMs, specifically comparing the impact of domain fine-tuning versus retrieval-augmented generation (RAG) on accuracy. Use when the user wants to benchmark on MedQA-USMLE, or asks about evaluating this task. Reports Majority-vote accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.23801
  bibtex_key: [REDACTED]
  confidence: high
---

# medqa-usmle-eval

> Domain Fine-Tuning vs. Retrieval-Augmented Generation for Medical Multiple-Choice Question Answering: A Controlled Comparison at the 4B-Parameter Scale — Avi-ad Avraam Buskila et al. (2026) (arXiv:2604.23801, 2026)

## What this evaluates

Evaluates medical multiple-choice question answering capability of 4B-parameter LLMs, specifically comparing the impact of domain fine-tuning versus retrieval-augmented generation (RAG) on accuracy.

## Datasets

- **MedQA-USMLE** — total 1273; splits: test (1273)

## Metrics

- `Majority-vote accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly answered questions out of the total test set. Answers are generated across three decoding repetitions at temperature T=0.1, and the most frequent option is selected as the final prediction.

## Input / output format

**Input**: Medical multiple-choice questions with four options. For RAG setups, relevant medical textbook passages are injected as context.

**Output**: A single letter corresponding to the chosen option. The pipeline allows up to two retries if the initial output does not match the expected answer schema.

## Scoring recipe

```python
# predictions: list of majority-vote answers per question (length N)
# golds: list of correct option letters (length N)
correct = sum(1 for p, g in zip(predictions, golds) if p == g)
accuracy = correct / len(golds)
```

## Common pitfalls

- RAG does not improve performance and may slightly decrease accuracy for domain-tuned models; the effect is not statistically significant.
- Single-pass decoding is insufficient due to stochasticity; the protocol requires three repetitions at T=0.1 to compute majority-vote accuracy.
- Outputs must strictly match the answer schema; parse failures trigger automatic retries, which affects the effective success rate.

## Evidence (verbatim from paper)

> Majority-vote accuracy and Wilson 95% confidence intervals for the four setups appear in Table 2 and Figure 1. The domain-tuned backbone (MedGemma 4B) leads both with and without RAG, and the best non-domain setup (Gemma 3 4B + RAG) does not catch up to the worst domain setup (MedGemma 4B + RAG).

## Citation

```bibtex
@misc{buskila2026domain,
  title={Domain Fine-Tuning vs. Retrieval-Augmented Generation for Medical Multiple-Choice Question Answering: A Controlled Comparison at the 4B-Parameter Scale},
  author={Avi-ad Avraam Buskila et al. (2026)},
  year={2026},
  note={arXiv:2604.23801}
}
```

- arXiv: 2604.23801