Medical Reasoning Benchmarks Eval

---
name: medical-reasoning-benchmarks-eval
description: Evaluates large language models' ability to perform medical reasoning across multiple-choice clinical questions, specialist-level board exams, and general-domain medical subsets. It probes factual knowledge integration, diagnostic accuracy, and reasoning under uncertainty in safety-critical settings. Use when the user wants to benchmark on MedQA (USMLE), MedMCQA (Validation), PubMedQA, GPQA, JMED, ReDis-QA, MedXpertQA, MMLU-Pro, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.08559
  bibtex_key: ren2026medicalreasoning
  confidence: high
---

# medical-reasoning-benchmarks-eval

> Medical Reasoning with Large Language Models: A Survey and MR-Bench — Ren et al. (2026) (arXiv:2604.08559, 2026)

## What this evaluates

Evaluates large language models' ability to perform medical reasoning across multiple-choice clinical questions, specialist-level board exams, and general-domain medical subsets. It probes factual knowledge integration, diagnostic accuracy, and reasoning under uncertainty in safety-critical settings.

## Datasets

- **MedQA (USMLE)** — total 1273; splits: test (1273); HF `bigbio/med_qa`
- **MedMCQA (Validation)** — total 4183; splits: val (4183); HF `openlifescienceai/medmcqa`
- **PubMedQA** — total 1000; splits: test (1000); HF `qiaojin/PubMedQA`
- **GPQA** — total 448; splits: test (448); HF `Idavidrein/gpqa`
- **JMED** — total 1000; splits: test (1000); HF `jdh-algo/JMED`
- **ReDis-QA** — total 1360; splits: test (1360); HF `guan-wang/ReDis-QA`
- **MedXpertQA** — total 2450; splits: test (2450); HF `TsinghuaC3I/MedXpertQA`
- **MMLU-Pro** — total 12032; splits: test (12032); HF `TIGER-Lab/MMLU-Pro`

## Metrics

- `accuracy` **(primary)** — range: percent
  - Proportion of correctly predicted options out of the total number of instances. Calculated as (correct predictions / total instances) × 100.

## Input / output format

**Input**: Multiple-choice clinical questions with a stem and several options (typically A-D or A-E), sometimes accompanied by biomedical literature or patient context.

**Output**: A single letter corresponding to the selected option (e.g., 'A', 'B', 'C', 'D').

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().upper() == g.strip().upper())
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Static question-answering format lacks dynamic evidence and uncertainty handling present in real clinical settings.
- Models may exploit shortcut learning or memorization from training data rather than genuine reasoning.
- Open-ended benchmarks rely on imperfect automated metrics (BLEU/ROUGE) or LLM judges that suffer from preference bias.

## Evidence (verbatim from paper)

> Accuracy remains the standard metric for multiple-choice benchmarks such as MedQA and MMLU, while n-gram overlap measures (e.g., BLEU and ROUGE) are commonly used for open-ended generation and question answering tasks. The primary advantages of this paradigm are its objectivity, low cost, and high reproducibility, enabling consistent comparison across models and studies.

## Citation

```bibtex
@misc{ren2026medicalreasoning,
  title={Medical Reasoning with Large Language Models: A Survey and MR-Bench},
  author={Ren et al. (2026)},
  year={2026},
  note={arXiv:2604.08559}
}
```

- arXiv: 2604.08559