Medagents Medical Qa Eval

---
name: medagents-medical-qa-eval
description: Evaluates zero-shot medical reasoning and multiple-choice question answering capabilities of LLMs using a training-free multi-agent collaboration framework. It probes the model's ability to simulate domain expert role-playing and reach consensus without retrieval-augmented generation. Use when the user wants to benchmark on MedQA, MedMCQA, PubMedQA, MMLU Anatomy, MMLU Clinical Knowledge, MMLU College Medicine, MMLU Medical Genetics, MMLU Professional Medicine, MMLU College Biology, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2311.10537
  bibtex_key: tang2023medagents
  confidence: high
---

# medagents-medical-qa-eval

> MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning — Tang et al. (2023) (arXiv:2311.10537, 2023)

## What this evaluates

Evaluates zero-shot medical reasoning and multiple-choice question answering capabilities of LLMs using a training-free multi-agent collaboration framework. It probes the model's ability to simulate domain expert role-playing and reach consensus without retrieval-augmented generation.

## Datasets

- **MedQA** — total ?; splits: test (300)
- **MedMCQA** — total ?; splits: test (300)
- **PubMedQA** — total ?; splits: test (300)
- **MMLU Anatomy** — total ?; splits: test (300)
- **MMLU Clinical Knowledge** — total ?; splits: test (300)
- **MMLU College Medicine** — total ?; splits: test (300)
- **MMLU Medical Genetics** — total ?; splits: test (300)
- **MMLU Professional Medicine** — total ?; splits: test (300)
- **MMLU College Biology** — total ?; splits: test (300)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions out of the total evaluated instances.

## Input / output format

**Input**: Multiple-choice medical question with 3 or 4 options.

**Output**: Selected answer option (e.g., A, B, C, or D).

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().upper() == gold.strip().upper():
        correct += 1
return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Chain-of-thought prompting can cause hallucinations and degrade performance in medical domains.
- Cost and inference time vary significantly with the number of agents and iterations.
- Zero-shot settings may underperform compared to few-shot baselines without careful prompt engineering.

## Evidence (verbatim from paper)

> We evaluate our MEDAGENTS framework on three benchmark datasets MedQA (Jin et al., 2021), MedMCQA (Pal et al., 2022), and PubMedQA (Jin et al., 2019), as well as six subtasks most relevant to the medical domain from MMLU datasets (Hendrycks et al., 2020) including anatomy, clinical knowledge, college medicine, medical genetics, professional medicine, and college biology. We randomly sample 300 examples for each dataset and conduct experiments on them.

## Citation

```bibtex
@misc{tang2023medagents,
  title={MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning},
  author={Tang et al. (2023)},
  year={2023},
  note={arXiv:2311.10537}
}
```

- arXiv: 2311.10537