Medical Qa Eval

---
name: medical-qa-eval
description: This benchmark evaluates the medical reasoning and question-answering capabilities of language models across multiple-choice and open-ended clinical tasks. It probes the model's ability to retrieve relevant medical knowledge, perform stepwise reasoning, and select or generate correct answers based on clinical guidelines and literature. Use when the user wants to benchmark on MedQA, MedMCQA, MMLU-Med, DDXPlus, AgentClinicNEJM, AgentClinicMedQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.11474
  bibtex_key: yun2025medprm
  confidence: high
---

# medical-qa-eval

> Med-PRM: Medical Reasoning Models with Stepwise, Guideline-verified Process Rewards — Yun et al. (2025) (arXiv:2506.11474, 2025)

## What this evaluates

This benchmark evaluates the medical reasoning and question-answering capabilities of language models across multiple-choice and open-ended clinical tasks. It probes the model's ability to retrieve relevant medical knowledge, perform stepwise reasoning, and select or generate correct answers based on clinical guidelines and literature.

## Datasets

- **MedQA** — total ?; splits: test (-1)
- **MedMCQA** — total ?; splits: validation (-1)
- **MMLU-Med** — total ?; splits: test (-1)
- **DDXPlus** — total ?; splits: test (-1)
- **AgentClinicNEJM** — total ?; splits: test (-1)
- **AgentClinicMedQA** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Standard exact-match accuracy calculated as the number of correctly answered questions divided by the total number of questions. For open-ended AgentClinic tasks, accuracy is determined by an external LLM judge (Gemini-2.0-flash) comparing the model's generated response against the gold standard.

## Input / output format

**Input**: For multiple-choice: a clinical question with four or five options. For open-ended: a clinical scenario or question requiring a free-text response. During training/PRM scoring, retrieved medical documents (truncated to 3072 tokens) are prepended to the prompt with reasoning steps separated by a special token, but evaluation uses standard benchmark prompts.

**Output**: For multiple-choice: a single selected option (letter or text). For open-ended: a generated natural language response.

## Scoring recipe

```python
def compute_accuracy(predictions, golds, open_ended=False):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if open_ended:
            # Evaluated by Gemini-2.0-flash as per paper
            score = llm_judge_gemini_flash(pred, gold)
        else:
            score = (normalize(pred) == normalize(gold))
        correct += score
    return (correct / len(golds)) * 100
```

## Common pitfalls

- Open-ended AgentClinic results depend entirely on the Gemini-2.0-flash judge, which may introduce evaluation bias or inconsistency compared to exact-match MC scoring.
- Scores vary significantly based on the test-time computation strategy (Best-of-N vs. SC+RM); results must be reported with the specific strategy used.
- AgentClinic variants are simplified versions of the original benchmarks, so direct comparison with full AgentClinic scores is invalid.

## Evidence (verbatim from paper)

> Table 1: Accuracy of proprietary and open-source models across multiple-choice and open-ended medical QA benchmarks. We evaluate Med-PRM on MedQA (4 and 5 options), MedMCQA (validation set), six medical MMLU subsets, DDXPlus, and two open-ended AgentClinic variants based on NEJM and MedQA. AgentClinic adopts an open-ended format and is evaluated using Gemini-2.0-flash.

## Citation

```bibtex
@misc{yun2025medprm,
  title={Med-PRM: Medical Reasoning Models with Stepwise, Guideline-verified Process Rewards},
  author={Yun et al. (2025)},
  year={2025},
  note={arXiv:2506.11474}
}
```

- arXiv: 2506.11474