Medical Mcqa Eval

---
name: medical-mcqa-eval
description: Evaluates a model's ability to answer medical multiple-choice questions by selecting the correct option from a set of distractors, including natural differential diagnoses. It tests both internal knowledge retrieval and the impact of synthetic pretraining with cue-masking strategies. Use when the user wants to benchmark on MedQA-USMLE, MedMCQA, DBPedia, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2303.07069
  bibtex_key: sileo2023generating
  confidence: high
---

# medical-mcqa-eval

> Generating multiple-choice questions for medical question answering with distractors and cue-masking — Sileo et al. (2023) (arXiv:2303.07069, 2023)

## What this evaluates

Evaluates a model's ability to answer medical multiple-choice questions by selecting the correct option from a set of distractors, including natural differential diagnoses. It tests both internal knowledge retrieval and the impact of synthetic pretraining with cue-masking strategies.

## Datasets

- **MedQA-USMLE** — total ?; splits: test (-1)
- **MedMCQA** — total ?; splits: train (-1)
- **DBPedia** — total 3446; splits: test (3446)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard multiple-choice accuracy: the proportion of instances where the model's predicted option (argmax of softmax probabilities) matches the gold correct option.
- `precision@3 / recall@3` — range: [0, 1]
  - For distractor prediction: precision@3 is the fraction of top-3 retrieved differential diagnoses that are correct; recall@3 is the fraction of all correct differential diagnoses retrieved in the top-3.

## Input / output format

**Input**: Question text concatenated with each answer option to compute logit scores. Optionally, the top 10 retrieved Wikipedia passages are concatenated to the question for knowledge-augmented evaluation.

**Output**: Logit scores for each option, passed through softmax to yield probabilities. The model selects the option with the highest probability as its prediction.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

def compute_pr_at_k(retrieved, relevant, k=3):
    top_k = retrieved[:k]
    precision = len(set(top_k) & set(relevant)) / k
    recall = len(set(top_k) & set(relevant)) / len(relevant) if relevant else 0
    return precision, recall
```

## Common pitfalls

- Extraneous cue leakage occurs if masking strategies inadvertently hide tokens that signal the correct answer, artificially lowering performance.
- Models may rely on external knowledge retrieval (BM25) rather than internalized medical knowledge, making results sensitive to the retrieval setup.
- Probability-matching masking requires careful handling of negative examples to avoid masking tokens that would otherwise indicate the correct answer.

## Evidence (verbatim from paper)

> We use a multiple-choice-question answering setup (we predict logit scores for each option by concatenating the question and the option, then use a softmax and optimize the likelihood of the correct option). Table 2 shows the test accuracy of BioLinkBERT fine-tuned on WikiMedQA then on various datasets, compared to the task-specific state-of-the-art.

## Citation

```bibtex
@misc{sileo2023generating,
  title={Generating multiple-choice questions for medical question answering with distractors and cue-masking},
  author={Sileo et al. (2023)},
  year={2023},
  note={arXiv:2303.07069}
}
```

- arXiv: 2303.07069