Bioclinical Modernbert Eval

---
name: bioclinical-modernbert-eval
description: Evaluates long-context clinical NLP encoders on biomedical entity recognition, clinical text classification, and demographic information extraction. Probes the model's ability to process full-length clinical notes (up to 8,192 tokens) and retain domain-specific knowledge without truncation. Use when the user wants to benchmark on ChemProt, Phenotype, Social History, DEID, COS, or asks about evaluating this task. Reports F1 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.10896
  bibtex_key: sounack2025bioclinicalmodernbert
  confidence: high
---

# bioclinical-modernbert-eval

> BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP — Sounack et al. (2025) (arXiv:2506.10896, 2025)

## What this evaluates

Evaluates long-context clinical NLP encoders on biomedical entity recognition, clinical text classification, and demographic information extraction. Probes the model's ability to process full-length clinical notes (up to 8,192 tokens) and retain domain-specific knowledge without truncation.

## Datasets

- **ChemProt** — total ?; splits: test (-1)
- **Phenotype** — total ?; splits: test (-1)
- **Social History** — total ?; splits: test (-1)
- **DEID** — total ?; splits: test (-1)
- **COS** — total ?; splits: test (-1)

## Metrics

- `F1 score` **(primary)** — range: percent
  - Harmonic mean of precision and recall. For classification tasks, macro-averaged across all classes. For NER tasks, entity-level exact match F1 is used.

## Input / output format

**Input**: Clinical text sequences (PubMed abstracts, clinical notes, social history records) tokenized with a 50,368-token vocabulary, processed at fixed lengths (512, 4096, or 8192 tokens) or as variable-length sequences.

**Output**: Class labels for classification tasks (e.g., chemical-protein relations, phenotype categories) or token-level entity tags for NER tasks (e.g., social history entities, demographic identifiers, clinical observations).

## Scoring recipe

```python
def compute_f1(predictions, gold_labels):
    # Classification: macro-averaged F1
    # NER: entity-level exact match F1
    tp = sum(1 for p, g in zip(predictions, gold_labels) if p == g and p != 'O')
    fp = sum(1 for p, g in zip(predictions, gold_labels) if p != g and p != 'O')
    fn = sum(1 for p, g in zip(predictions, gold_labels) if p != g and g != 'O')
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return f1 * 100  # Return as percentage
```

## Common pitfalls

- Confusing fixed-length vs variable-length inference speed benchmarks; some models degrade significantly on variable-length sequences due to lack of unpadding.
- Using short-context truncation (e.g., 512 tokens) for clinical notes, which loses critical information compared to the model's designed 8,192-token context window.
- Evaluating Phase 1 vs Phase 2 checkpoints without clarifying that Phase 2 includes clinical specialization data, which can artificially inflate clinical task scores.

## Evidence (verbatim from paper)

> For classification tasks, BioClinical ModernBERT large achieves state-of-the-art results with an F1 score of 90.8% on ChemProt and 60.8% on Phenotype. The base model also outperforms all other base models, achieving 89.9% on ChemProt and 58.1% on Phenotype. In named entity recognition, the base model achieves state-of-the-art performance on Social History and outperforms other base models on DEID.

## Citation

```bibtex
@misc{sounack2025bioclinicalmodernbert,
  title={BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP},
  author={Sounack et al. (2025)},
  year={2025},
  note={arXiv:2506.10896}
}
```

- arXiv: 2506.10896