Biomed Enriched Eval

---
name: biomed-enriched-eval
description: Evaluates the biomedical knowledge, clinical reasoning, and domain-specific comprehension of language models using multiple-choice question-answering benchmarks across anatomy, clinical medicine, genetics, and multilingual medical QA. Use when the user wants to benchmark on MMLU Professional Medicine, MedQA, MedMCQA, PubMedQA, FrenchMedMCQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.20331
  bibtex_key: touchent2025biomedenriched
  confidence: high
---

# biomed-enriched-eval

> Biomed-Enriched: A Biomedical Dataset Enriched with LLMs for Pretraining and Extracting Rare and Hidden Content — Touchent et al. (2025) (arXiv:2506.20331, 2025)

## What this evaluates

Evaluates the biomedical knowledge, clinical reasoning, and domain-specific comprehension of language models using multiple-choice question-answering benchmarks across anatomy, clinical medicine, genetics, and multilingual medical QA.

## Datasets

- **MMLU Professional Medicine** — total ?; splits: test (-1)
- **MedQA** — total ?; splits: test (-1)
- **MedMCQA** — total ?; splits: test (-1)
- **PubMedQA** — total ?; splits: test (-1)
- **FrenchMedMCQA** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions. Calculated as (number of correct predictions / total number of questions) * 100.

## Input / output format

**Input**: Multiple-choice questions covering biomedical, clinical, and biological topics, typically presented with a question stem and four or five options.

**Output**: The model's predicted answer choice (letter or full text) for each question.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if normalize_answer(pred) == normalize_answer(gold):
        correct += 1
return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Assuming 'Clinical Knowledge' and 'Medical Genetics' are separate datasets rather than MMLU subsets or specialized benchmarks without checking the paper's appendix.
- Comparing data efficiency claims without normalizing for the exact number of training tokens, as the paper highlights performance gains using only 1/3 of the baseline tokens.
- Overlooking that FrenchMedMCQA results are reported in accuracy but may use a different option format or language-specific preprocessing than the English benchmarks.

## Evidence (verbatim from paper)

> BE-French showed clear improvements in French medical QA (FrenchMedMCQA), achieving 40.5% accuracy, significantly surpassing BE-Base and the OLMo2-7B-stage1 baseline (38.32%, Figure 1).

## Citation

```bibtex
@misc{touchent2025biomedenriched,
  title={Biomed-Enriched: A Biomedical Dataset Enriched with LLMs for Pretraining and Extracting Rare and Hidden Content},
  author={Touchent et al. (2025)},
  year={2025},
  note={arXiv:2506.20331}
}
```

- arXiv: 2506.20331