Medxpertqa Eval

---
name: medxpertqa-eval
description: Evaluates expert-level medical reasoning and clinical understanding using real-world board exam questions, patient records, and multimodal clinical data. Use when the user wants to benchmark on MedXpertQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.18362
  bibtex_key: [REDACTED]
  confidence: high
---

# medxpertqa-eval

> MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding — Zuo et al. (2025) (arXiv:2501.18362, 2025)

## What this evaluates

Evaluates expert-level medical reasoning and clinical understanding using real-world board exam questions, patient records, and multimodal clinical data.

## Datasets

- **MedXpertQA** — total 4460; splits: Reasoning (-1), Understanding (-1); repo https://github.com/TsinghuaC3I/MedXpertQA

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions after answer extraction. Calculated as (number of correct predictions / total questions) × 100.

## Input / output format

**Input**: Multiple-choice clinical questions with patient records and clinical data; multimodal subset includes medical images alongside text.

**Output**: Model generates a response using zero-shot chain-of-thought prompting; final answer is extracted as a single letter corresponding to the correct option.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_ans in zip(predictions, gold):
        pred_letter = extract_answer_letter(pred) # via answer cleansing script
        if pred_letter == gold_ans:
            correct += 1
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Direct performance comparisons between MedXpertQA Text and MedXpertQA MM are not informative due to differing numbers of answer choices and filtering parameters.
- o1 and o3-mini were evaluated on only a 10% sampled subset due to API costs, which may affect comparability with full-set results.
- Answer cleansing is required to extract the final choice from chain-of-thought outputs; skipping it leads to incorrect accuracy scores.

## Evidence (verbatim from paper)

> Overall, the low accuracies of evaluated models demonstrate MedXpertQA's ability to pose challenges to state-of-the-art models.

## Citation

```bibtex
@misc{zuo2025medxpertqa,
  title={MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding},
  author={Zuo et al. (2025)},
  year={2025},
  note={arXiv:2501.18362}
}
```

- arXiv: 2501.18362