Medmcqa Eval

---
name: medmcqa-eval
description: Evaluates a model's ability to answer medical multiple-choice questions, testing both domain-specific knowledge retrieval and deep medical reasoning capabilities. Use when the user wants to benchmark on MedMCQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2203.14371
  bibtex_key: pal2022medmcqa
  confidence: high
---

# medmcqa-eval

> MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering — Pal et al. (2022) (arXiv:2203.14371, 2022)

## What this evaluates

Evaluates a model's ability to answer medical multiple-choice questions, testing both domain-specific knowledge retrieval and deep medical reasoning capabilities.

## Datasets

- **MedMCQA** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard multiple-choice accuracy: the proportion of questions where the model's predicted option exactly matches the ground-truth correct option.

## Input / output format

**Input**: Concatenated sequence: [CLS] Context [SEP] Question [SEP] Option [SEP] (with context) or [CLS] Question [SEP] Option [SEP] (without context). Passages are truncated to 250 tokens.

**Output**: A single predicted option index selected via softmax over the [CLS] token's final hidden state.

## Scoring recipe

```python
correct = 0
for q, options, gold_idx in dataset:
    logits = model(input_ids)
    pred_idx = argmax(logits[0])
    if pred_idx == gold_idx:
        correct += 1
accuracy = correct / len(dataset)
```

## Common pitfalls

- Retrieval errors from the dense passage retriever are often conflated with the reader model's reasoning failures.
- Truncating medical passages to 250 tokens may discard critical clinical details needed to answer the question.
- Evaluating only on the test set without reporting calibration or confidence scores limits understanding of model reliability in high-stakes medical domains.

## Evidence (verbatim from paper)

> A linear layer with softmax is applied over the output of the [CLS] token of the encoder. This is to select the most appropriate option for a question and context pair. ... The model checkpoint with the highest validation score in the 5 epochs was selected and used to evaluate the Test Set.

## Citation

```bibtex
@misc{pal2022medmcqa,
  title={MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering},
  author={Pal et al. (2022)},
  year={2022},
  note={arXiv:2203.14371}
}
```

- arXiv: 2203.14371