Medhelm Eval

---
name: medhelm-eval
description: Evaluates large language models on a comprehensive taxonomy of real-world clinical workflows, covering tasks like clinical note generation, patient communication, medical research assistance, clinical decision support, and administration/workflow. It probes both closed-ended factual/reasoning tasks and open-ended free-text generation capabilities in medical domains. Use when the user wants to benchmark on MedHELM, or asks about evaluating this task. Reports Macro-average performance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.23802
  bibtex_key: bedi2025medhelm
  confidence: high
---

# medhelm-eval

> MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks — Suhana Bedi et al. (arXiv:2505.23802, 2025)

## What this evaluates

Evaluates large language models on a comprehensive taxonomy of real-world clinical workflows, covering tasks like clinical note generation, patient communication, medical research assistance, clinical decision support, and administration/workflow. It probes both closed-ended factual/reasoning tasks and open-ended free-text generation capabilities in medical domains.

## Datasets

- **MedHELM** — total ?; splits: test (-1)

## Metrics

- `Macro-average performance` **(primary)** — range: [0, 1]
  - Average of normalized performance scores across all 35 benchmarks.
- `Win-rate` — range: [0, 1]
  - Proportion of pairwise comparisons where a model outperforms another across all 35 benchmarks.
- `LLM-jury score` — range: [0, 1]
  - Average normalized score assigned by three frontier LLMs to open-ended model outputs.
- `exact-match accuracy` — range: [0, 1]
  - Binary indicator of whether the model's output exactly matches the ground truth answer.
- `F1 score` — range: [0, 1]
  - Harmonic mean of precision and recall for extracting ICD-10 codes from clinical notes.

## Input / output format

**Input**: Prompt containing a medical task description, patient notes, or clinical question, varying by benchmark (closed-ended multiple choice or open-ended generation).

**Output**: Model-generated response: either a selected answer choice for closed-ended tasks or free-text generation for open-ended tasks.

## Scoring recipe

```python
def score(predictions, golds, is_open_ended=False):
    if is_open_ended:
        jury_scores = [jury_llm.evaluate(pred, gold) for pred, gold in zip(predictions, golds)]
        return {'LLM-jury score': mean(jury_scores)}
    else:
        em = [1 if pred == gold else 0 for pred, gold in zip(predictions, golds)]
        return {'exact-match accuracy': mean(em)}
# Macro-average is computed by averaging normalized scores across all 35 benchmarks.
# Win-rate is computed via pairwise comparisons across all benchmarks.
```

## Common pitfalls

- Win-rate is calculated as pairwise comparisons across all 35 benchmarks, not a simple average of individual benchmark scores.
- LLM-jury scores are aggregated from three frontier LLMs and validated against clinician ratings using ICC(3,k), not standard lexical metrics like ROUGE-L.
- Cost estimates are upper-bound based on maximum output token usage, not actual inference costs.

## Evidence (verbatim from paper)

> Win-rate represents the proportion of pairwise comparisons where each model achieved superior performance across all 35 benchmarks (possible range: 0-1). Win standard deviation (SD) measures how consistently a model wins (lower values = more consistent). Macro-avg is the average performance score across all 35 benchmarks.

## Citation

```bibtex
@misc{bedi2025medhelm,
  title={MedHELM: Holistic Evaluation of Large Language Models for Medical Tasks},
  author={Suhana Bedi et al.},
  year={2025},
  note={arXiv:2505.23802}
}
```

- arXiv: 2505.23802