Explaincpe Eval

---
name: explaincpe-eval
description: This benchmark evaluates large language models on Chinese medical multiple-choice questions, specifically probing their ability to select correct answers and generate faithful, logically consistent free-text explanations. It measures both factual accuracy and the quality of interpretability in high-stakes healthcare domains. Use when the user wants to benchmark on ExplainCPE, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.12945
  bibtex_key: li2023explaincpe
  confidence: high
---

# explaincpe-eval

> ExplainCPE: A Free-text Explanation Benchmark of Chinese Pharmacist Examination — Dongfang Li et al. (2023) (arXiv:2305.12945, 2023)

## What this evaluates

This benchmark evaluates large language models on Chinese medical multiple-choice questions, specifically probing their ability to select correct answers and generate faithful, logically consistent free-text explanations. It measures both factual accuracy and the quality of interpretability in high-stakes healthcare domains.

## Datasets

- **ExplainCPE** — total ?; splits: train (-1), test (-1); repo https://github.com/HITsz-TMG/ExplainCPE

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of test instances where the model's predicted answer option exactly matches the gold standard option.
- `ROUGE-1` — range: [0, 1]
  - Unigram-level recall/precision F1 score computed between the generated explanation and the gold explanation.
- `ROUGE-2` — range: [0, 1]
  - Bigram-level recall/precision F1 score computed between the generated explanation and the gold explanation.
- `ROUGE-L` — range: [0, 1]
  - Longest common subsequence-based recall/precision F1 score computed between the generated explanation and the gold explanation.

## Input / output format

**Input**: A multiple-choice medical question with options, optionally preceded by few-shot examples (zero, one, four, or eight). Prompts may include or exclude explicit instructions.

**Output**: The selected answer option followed by a free-text explanation justifying the choice.

## Scoring recipe

```python
def score(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p['option'] == g['option'])
    acc = (correct / len(predictions)) * 100
    r1, r2, rl = 0.0, 0.0, 0.0
    for p, g in zip(predictions, golds):
        r1 += rouge1_score(g['explanation'], p['explanation'])
        r2 += rouge2_score(g['explanation'], p['explanation'])
        rl += rougeL_score(g['explanation'], p['explanation'])
    n = len(predictions)
    return {'Accuracy': acc, 'ROUGE-1': r1/n, 'ROUGE-2': r2/n, 'ROUGE-L': rl/n}
```

## Common pitfalls

- ROUGE metrics only measure lexical overlap and do not capture factual correctness, logical consistency, or hallucination of contraindications in medical explanations.
- Few-shot examples for 4-shot and 8-shot settings were manually selected to balance question types, which may introduce selection bias and limit reproducibility compared to random sampling.
- Prompt variations (with/without instructions) interact differently with chat vs. non-chat models, making direct cross-model comparisons highly sensitive to prompt formatting.

## Evidence (verbatim from paper)

> we designed prompts to request the model to provide an answer option along with an explanation in the test set. Table 3: Performance comparison on ExplainCPE dataset. <table><tr><td>Model</td><td>Acc(%)</td><td>Rouge-1</td><td>Rouge-2</td><td>Rouge-L</td></tr>

## Citation

```bibtex
@misc{li2023explaincpe,
  title={ExplainCPE: A Free-text Explanation Benchmark of Chinese Pharmacist Examination},
  author={Dongfang Li et al. (2023)},
  year={2023},
  note={arXiv:2305.12945}
}
```

- arXiv: 2305.12945