Clinical Modernbert Eval

---
name: clinical-modernbert-eval
description: Evaluates a biomedical language model on short- and long-context clinical NLP tasks including classification, named entity recognition, and retrieval. It also measures pre-training masked language modeling accuracy and measures inference efficiency under varying computational loads. Use when the user wants to benchmark on EHR-Prediction (MIMIC-IV ED), MedNER, Pubmed-NCT, PMC-Retrieval, i2b2 2006, i2b2 2010, i2b2 2012, i2b2 2014, or asks about evaluating this task. Reports top-k accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.03964
  bibtex_key: lee2025clinicalmodernbert
  confidence: high
---

# clinical-modernbert-eval

> Clinical ModernBERT: An efficient and long context encoder for biomedical text — Lee et al. (2025) (arXiv:2504.03964, 2025)

## What this evaluates

Evaluates a biomedical language model on short- and long-context clinical NLP tasks including classification, named entity recognition, and retrieval. It also measures pre-training masked language modeling accuracy and measures inference efficiency under varying computational loads.

## Datasets

- **EHR-Prediction (MIMIC-IV ED)** — total 400019; splits: unspecified (-1)
- **MedNER** — total 3655; splits: unspecified (-1)
- **Pubmed-NCT** — total 221186; splits: unspecified (-1)
- **PMC-Retrieval** — total 167034; splits: unspecified (-1)
- **i2b2 2006** — total 66034; splits: unspecified (-1)
- **i2b2 2010** — total 43947; splits: unspecified (-1)
- **i2b2 2012** — total 13108; splits: unspecified (-1)
- **i2b2 2014** — total 83466; splits: unspecified (-1)

## Metrics

- `top-k accuracy` **(primary)** — range: [0, 1]
  - Computed over masked token positions. A prediction is correct if the ground truth token appears in the top-k most probable tokens predicted by the model. Top-1 reflects precision, top-25 captures broader lexical recall.
- `forward pass latency` — range: milliseconds
  - Wall-clock time measured from input tokenization through to the final hidden state output, excluding disk/network I/O. Reported as mean across 3 runs.

## Input / output format

**Input**: Tokenized clinical or biomedical text sequences. For pre-training, sequences are tokenized and a subset of tokens is replaced with [MASK]. For downstream tasks, inputs are padded or truncated to model-specific context limits (e.g., 512 or 8192 tokens).

**Output**: For MLM: predicted probability distribution over the vocabulary for masked positions. For downstream tasks: predicted class labels, entity spans, or retrieved document IDs. For efficiency: wall-clock time in milliseconds.

## Scoring recipe

```python
def compute_topk_accuracy(gold_tokens, predicted_logits, k):
    correct = 0
    total = 0
    for gold, logits in zip(gold_tokens, predicted_logits):
        top_k_indices = logits.argsort(descending=True)[:k]
        if gold in top_k_indices:
            correct += 1
        total += 1
    return correct / total if total > 0 else 0.0
```

## Common pitfalls

- Pre-training MLM evaluation uses top-k accuracy, but downstream task metrics (e.g., F1 for NER, accuracy for classification) are not explicitly defined in this section and follow standard task conventions.
- Efficiency benchmarking uses synthetic clinical text fixed at 512 tokens and excludes I/O; real-world clinical inference latency will be higher and variable.
- Long-context tasks (i2b2 series) exceed standard 512-token limits, requiring models to handle sequences up to ~14k tokens without truncation.

## Evidence (verbatim from paper)

> To quantify performance, we compute top-k accuracy over masked positions, where a prediction is considered correct if the ground truth token appears in the top-k most probable tokens predicted by the model. Specifically, for each masked token $x_{i}$, we sort the predicted distribution and check whether $x_{i}$ lies in the top $k$ logits.

## Citation

```bibtex
@misc{lee2025clinicalmodernbert,
  title={Clinical ModernBERT: An efficient and long context encoder for biomedical text},
  author={Lee et al. (2025)},
  year={2025},
  note={arXiv:2504.03964}
}
```

- arXiv: 2504.03964