Medeval Eval

---
name: medeval-eval
description: Evaluates language models on multi-level (sentence/document) and multi-task (NLU/NLG) medical benchmarks across diverse clinical domains. It probes a model's ability to perform clinical text classification, report code prediction, and medical report summarization using both fine-tuned PLMs and prompted LLMs. Use when the user wants to benchmark on MedEval, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.14088
  bibtex_key: he2023medeval
  confidence: high
---

# medeval-eval

> MedEval: A Multi-Level, Multi-Task, and Multi-Domain Medical Benchmark for Language Model Evaluation — He et al. (2023) (arXiv:2310.14088, 2023)

## What this evaluates

Evaluates language models on multi-level (sentence/document) and multi-task (NLU/NLG) medical benchmarks across diverse clinical domains. It probes a model's ability to perform clinical text classification, report code prediction, and medical report summarization using both fine-tuned PLMs and prompted LLMs.

## Datasets

- **MedEval** — total ?; splits: (unstated); repo https://github.com/ZexueHe/MedEval

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted class labels or report codes out of the total number of test instances.
- `exact match rate (EMR)` — range: [0, 1]
  - Proportion of predictions that exactly match the ground-truth label or code string.
- `ROUGE-1/2/L` — range: [0, 1]
  - Standard ROUGE recall/precision/f1 scores measuring n-gram overlap between generated and reference clinical summaries.
- `BLEU4` — range: [0, 1]
  - Geometric mean of modified n-gram precisions up to 4-grams, penalizing brevity in generated text.

## Input / output format

**Input**: Clinical text instances (individual sentences or full EMR documents) provided as prompts. For LLM evaluation, prompts include zero-shot or few-shot examples with explicit instructions for classification or generation tasks.

**Output**: For NLU tasks: discrete class labels (e.g., abnormality/ambiguity flags or report codes). For NLG tasks: free-text generated sentences or summarized clinical reports.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    acc = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    emr = sum(1 for p, g in zip(predictions, golds) if p == g) / len(golds)
    rouge_scores = rouge_score(golds, predictions) # ROUGE-1, 2, L
    bleu4 = bleu_score(golds, predictions)
    return {'accuracy': acc, 'emr': emr, 'rouge': rouge_scores, 'bleu4': bleu4}
```

## Common pitfalls

- Models lacking instruction tuning (e.g., BioMed LM) fail to follow prompts or hit token limits, yielding artificially low scores.
- Few-shot prompting consistently improves NLU accuracy but shows no reliable trend for NLG generation tasks.
- Performance heavily depends on domain familiarity; models excel on common domains (e.g., chest) but struggle on rare/ambiguous ones (miscellaneous).

## Evidence (verbatim from paper)

> We report the average accuracy over all classes of diseases and the exact match rate (EMR) between predictions and labels. ... We report the Rouge scores and BLEU4 scores.

## Citation

```bibtex
@misc{he2023medeval,
  title={MedEval: A Multi-Level, Multi-Task, and Multi-Domain Medical Benchmark for Language Model Evaluation},
  author={He et al. (2023)},
  year={2023},
  note={arXiv:2310.14088}
}
```

- arXiv: 2310.14088