Medical Benchmarks Eval

---
name: medical-benchmarks-eval
description: Evaluates large language models on medical knowledge, clinical reasoning, and safety alignment using multiple-choice and open-ended healthcare QA tasks. It measures standard accuracy across major medical benchmarks and quantifies unsafe response rates via automated safety classifiers. Use when the user wants to benchmark on MultiMedQA, MedMCQA, MedQA, PubMedQA, MMLU Med., CareQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.01886
  bibtex_key: gururajan2024aloe
  confidence: high
---

# medical-benchmarks-eval

> Aloe: A Family of Fine-tuned Open Healthcare LLMs — Gururajan et al. (2024) (arXiv:2405.01886, 2024)

## What this evaluates

Evaluates large language models on medical knowledge, clinical reasoning, and safety alignment using multiple-choice and open-ended healthcare QA tasks. It measures standard accuracy across major medical benchmarks and quantifies unsafe response rates via automated safety classifiers.

## Datasets

- **MultiMedQA** — total ?; splits: test (-1)
- **MedMCQA** — total ?; splits: test (-1)
- **MedQA** — total ?; splits: test (-1)
- **PubMedQA** — total ?; splits: test (-1)
- **MMLU Med.** — total ?; splits: test (-1)
- **CareQA** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted options out of total questions. For MultiMedQA, a weighted average accuracy across subtasks is computed. An overall arithmetic mean is calculated across all medical datasets.
- `Attack Success Rate (ASR)` — range: [0, 1]
  - Ratio of unsafe answers over total evaluated responses, classified using Llama Guard 2.

## Input / output format

**Input**: Medical multiple-choice questions, optionally augmented with 5 nearest-neighbor few-shot examples (Medprompt) or evaluated via self-consistency Chain-of-Thought (5 or 20 ensembles with shuffled options).

**Output**: Step-by-step reasoning followed by a selected option (A, B, C, etc.), or direct option selection. Final answer determined via majority voting across ensemble runs.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

def compute_medical_avg(results):
    non_careqa = {k: v for k, v in results.items() if k != 'CareQA'}
    weighted_multi = sum(v * w for v, w in non_careqa.items())
    overall = sum(results.values()) / len(results)
    return weighted_multi, overall
```

## Common pitfalls

- Ensemble size (5 vs 20) in Medprompt/SC-CoT yields only ~1% accuracy gain but increases compute cost by 4x.
- Medprompt few-shot examples must be drawn from the benchmark's training set (or MedMCQA's if unavailable), limited to 20k random samples to save compute.
- ASR scores are highly sensitive to the underlying safety classifier (Llama Guard 2) and jailbreak attack styles.

## Evidence (verbatim from paper)

> We produce the standard MultiMedQA score for reference, by computing the weighted average accuracy on all scores except CareQA. Additionally, we calculate the arithmetic mean across all datasets.

## Citation

```bibtex
@misc{gururajan2024aloe,
  title={Aloe: A Family of Fine-tuned Open Healthcare LLMs},
  author={Gururajan et al. (2024)},
  year={2024},
  note={arXiv:2405.01886}
}
```

- arXiv: 2405.01886