Medical Qa Explanation Eval

---
name: medical-qa-explanation-eval
description: Evaluates large language models' ability to answer challenging medical multiple-choice questions and generate step-by-step clinical reasoning explanations. It probes both factual accuracy in clinical decision-making and the quality of model-generated rationales compared to expert-written references. Use when the user wants to benchmark on JAMA Clinical Challenge, Medbullets, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.18060
  bibtex_key: chen2024benchmarking
  confidence: high
---

# medical-qa-explanation-eval

> Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions — Chen et al. (2024) (arXiv:2402.18060, 2024)

## What this evaluates

Evaluates large language models' ability to answer challenging medical multiple-choice questions and generate step-by-step clinical reasoning explanations. It probes both factual accuracy in clinical decision-making and the quality of model-generated rationales compared to expert-written references.

## Datasets

- **JAMA Clinical Challenge** — total ?; splits: test (-1)
- **Medbullets** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Computed as the proportion of model predictions that exactly match the ground truth answer choice.
- `ROUGE-L` — range: [0, 1]
  - Computes the longest common subsequence overlap between model-generated explanations and reference (gold) explanations to measure surface-form similarity.
- `BERTScore` — range: [0, 1]
  - Measures semantic similarity between model-generated and gold explanations using contextual embeddings from BERT.
- `BARTScore+` — range: [0, 1]
  - Finetuned variant of BARTScore using CNN data to compute semantic similarity between generated and reference explanations.
- `BARTScore++` — range: [0, 1]
  - Finetuned variant of BARTScore using CNN and paraphrase data to compute semantic similarity between generated and reference explanations.

## Input / output format

**Input**: Clinical case description, question, and multiple-choice answer options (X).

**Output**: Model-generated answer choice (Y), optionally preceded by step-by-step reasoning (R), or an explanation (R) given the input and gold answer (Y*).

## Scoring recipe

```python
def compute_metrics(predictions, gold_answers, explanations, gold_explanations):
    acc = sum(1 for p, g in zip(predictions, gold_answers) if p == g) / len(predictions)
    rouge_l = rouge_l_score(explanations, gold_explanations)
    bert = bertscore_f1(explanations, gold_explanations)
    bart_plus = bartscore_score(explanations, gold_explanations, variant='CNN')
    bart_plus_plus = bartscore_score(explanations, gold_explanations, variant='CNN+Para')
    return {'accuracy': acc, 'ROUGE-L': rouge_l, 'BERTScore': bert, 'BARTScore+': bart_plus, 'BARTScore++': bart_plus_plus}
```

## Common pitfalls

- The evaluation uses three distinct prompting strategies (X→Y, X→RY, XY*→R) that drastically change the model's output format and reasoning process; results must be reported per strategy rather than aggregated.
- Explanation metrics (ROUGE-L, BERTScore, BARTScore variants) measure text similarity to gold references rather than clinical correctness or logical validity, which may not align with expert medical judgment.

## Evidence (verbatim from paper)

> We use accuracy (prediction compared to ground truth) to evaluate the predictions of each model. We explore several methods to evaluate model-generated explanations: ROUGE-L, BERTScore, and two finetuned variants of BARTScore—BARTScore+CNN (BARTScore+) and BARTScore+CNN+Para (BARTScore++). These metrics compute the surface-form or semantic similarities between model-generated explanations and reference (gold) explanations.

## Citation

```bibtex
@misc{chen2024benchmarking,
  title={Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2402.18060}
}
```

- arXiv: 2402.18060