Kormedmcqa Eval

---
name: kormedmcqa-eval
description: Probes large language models' ability to answer multiple-choice questions derived from South Korean healthcare professional licensing exams. It evaluates domain-specific medical knowledge, regional clinical guideline adherence, and reasoning capabilities in Korean. Use when the user wants to benchmark on KorMedMCQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.01469
  bibtex_key: kweon2024kormedmcqa
  confidence: high
---

# kormedmcqa-eval

> KorMedMCQA: Multi-Choice Question Answering Benchmark for Korean Healthcare Professional Licensing Examinations — Kweon et al. (2024) (arXiv:2403.01469, 2024)

## What this evaluates

Probes large language models' ability to answer multiple-choice questions derived from South Korean healthcare professional licensing exams. It evaluates domain-specific medical knowledge, regional clinical guideline adherence, and reasoning capabilities in Korean.

## Datasets

- **KorMedMCQA** — total 7469; splits: test (-1), dev (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Calculated as the proportion of correctly predicted answer indices compared to the ground truth correct answer indices.

## Input / output format

**Input**: Multiple-choice question with options, formatted in a 5-shot prompt using examples from the development set.

**Output**: Model-generated text containing the predicted answer index, extracted via regular expressions.

## Scoring recipe

```python
correct = 0
for pred_idx, gold_idx in zip(predictions, gold_indices):
    if pred_idx == gold_idx:
        correct += 1
return correct / len(predictions)
```

## Common pitfalls

- Regex extraction of answer indices can fail if models output reasoning text or alternative formatting.
- 5-shot prompt selection from the development set may introduce bias or data leakage if not carefully randomized.
- Regional medical guidelines and terminology differ significantly from US benchmarks, so direct translation or English-centric training data yields poor performance.

## Evidence (verbatim from paper)

> Generated model outputs were processed using regular expressions to extract predicted answers, which were then compared to the correct answer indices.

## Citation

```bibtex
@misc{kweon2024kormedmcqa,
  title={KorMedMCQA: Multi-Choice Question Answering Benchmark for Korean Healthcare Professional Licensing Examinations},
  author={Kweon et al. (2024)},
  year={2024},
  note={arXiv:2403.01469}
}
```

- arXiv: 2403.01469