Medbench Eval

---
name: medbench-eval
description: This benchmark evaluates Chinese large language models on clinical knowledge, diagnostic reasoning, and conversational ability. It probes performance across three standardized medical licensing exams and real-world clinical case scenarios, highlighting gaps in multi-hop reasoning, diagnostic precision, and response fluency. Use when the user wants to benchmark on MedBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2312.12806
  bibtex_key: cai2023medbench
  confidence: high
---

# medbench-eval

> MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models — Cai et al. (2023) (arXiv:2312.12806, 2023)

## What this evaluates

This benchmark evaluates Chinese large language models on clinical knowledge, diagnostic reasoning, and conversational ability. It probes performance across three standardized medical licensing exams and real-world clinical case scenarios, highlighting gaps in multi-hop reasoning, diagnostic precision, and response fluency.

## Datasets

- **MedBench** — total 41066; splits: test (41066)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Calculated as the number of correct predictions divided by the total number of questions. For multiple-choice items, a prediction is correct only if it exactly matches the ground-truth option letter.
- `BLEU` — range: [0, 1]
  - Standard n-gram precision metric (BLEU-1 and BLEU-4 reported) measuring lexical overlap between generated responses and reference clinical answers.
- `ROUGE-L` — range: [0, 1]
  - Longest common subsequence metric measuring recall/precision of the longest matching word sequence between generated and reference responses.

## Input / output format

**Input**: Multiple-choice medical exam questions (categorized as A1/A2/B or A3/A4 types) and real-world clinical case prompts covering Examinations, Treatments, and Diagnoses.

**Output**: For exam questions: a single letter choice (A-E). For clinical cases: a free-text generated response.

## Scoring recipe

```python
def compute_metrics(predictions, golds, case_preds, case_golds):
    # Exam accuracy
    correct = sum(1 for p, g in zip(predictions, golds) if p == g)
    accuracy = correct / len(golds)
    
    # Clinical case BLEU & ROUGE-L
    bleu_scores = [compute_bleu(p, g) for p, g in zip(case_preds, case_golds)]
    rouge_scores = [compute_rouge_l(p, g) for p, g in zip(case_preds, case_golds)]
    return accuracy, bleu_scores, rouge_scores
```

## Common pitfalls

- BLEU and ROUGE scores are inherently limited for clinical text and often yield moderate values even for top models, failing to holistically capture medical correctness or reasoning quality.
- Models frequently guess invalid options (e.g., 'F') on multiple-choice questions without proper prompting, artificially deflating accuracy if not filtered.
- Chain-of-Thought prompting yields inconsistent improvements across base models (e.g., helps Baichuan-13B but not ChatGLM-6B), so evaluation should account for prompt sensitivity.

## Evidence (verbatim from paper)

> In Table 1, we present a comprehensive analysis of the accuracy metrics for various LLMs across the three exams. We evaluated the outcomes using the BLEU and ROUGE F1-score metrics.

## Citation

```bibtex
@misc{cai2023medbench,
  title={MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models},
  author={Cai et al. (2023)},
  year={2023},
  note={arXiv:2312.12806}
}
```

- arXiv: 2312.12806