Medical Reasoning Eval

---
name: medical-reasoning-eval
description: Evaluates large language models on complex medical reasoning and knowledge retrieval across multiple-choice and open-ended clinical questions. It probes the model's ability to apply domain-specific knowledge, perform multi-step clinical reasoning, and handle challenging benchmarks that require more than simple fact recall. Use when the user wants to benchmark on MedQA (USMLE), MedMCQA, PubMedQA, MMLU-Pro, GPQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.18925
  bibtex_key: chen2024huatuogpt
  confidence: high
---

# medical-reasoning-eval

> HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs — Junying Chen et al. (2024) (arXiv:2412.18925, 2024)

## What this evaluates

Evaluates large language models on complex medical reasoning and knowledge retrieval across multiple-choice and open-ended clinical questions. It probes the model's ability to apply domain-specific knowledge, perform multi-step clinical reasoning, and handle challenging benchmarks that require more than simple fact recall.

## Datasets

- **MedQA (USMLE)** — total ?; splits: test (-1)
- **MedMCQA** — total ?; splits: val (-1)
- **PubMedQA** — total ?; splits: test (-1)
- **MMLU-Pro** — total ?; splits: test (-1)
- **GPQA** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions. Calculated as (number of correct predictions / total number of questions) * 100.

## Input / output format

**Input**: Medical multiple-choice or open-ended clinical questions, typically presented as vignettes or direct knowledge queries.

**Output**: Model generates a textual response containing reasoning and/or a final answer/option letter.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if normalize_answer(pred) == normalize_answer(gold):
        correct += 1
accuracy = (correct / len(gold_labels)) * 100
```

## Common pitfalls

- GPQA evaluation is run 5 times and averaged due to limited question count, introducing variance compared to deterministic benchmarks.
- MMLU-Pro and GPQA scores reported are only for specific medical-related tracks (health/biology, genetics/molecular biology), not the full benchmark.
- Exact-match parsing may fail on verbose CoT outputs; the paper notes LLM-based verifiers are more reliable than regex exact-match for training, though evaluation uses standard option matching.

## Evidence (verbatim from paper)

> GPT-4o achieved 96.5% accuracy in Stage 1 and 94.5% in Stage 2, demonstrating its reliability.

## Citation

```bibtex
@misc{chen2024huatuogpt,
  title={HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMs},
  author={Junying Chen et al. (2024)},
  year={2024},
  note={arXiv:2412.18925}
}
```

- arXiv: 2412.18925