Blurb Eval

---
name: blurb-eval
description: Evaluates biomedical language models on a comprehensive suite of downstream NLP tasks, including named entity recognition, relation extraction, sentence similarity, document classification, and question answering. It measures how well domain-specific pretraining transfers to specialized clinical and biomedical text understanding. Use when the user wants to benchmark on BLURB, or asks about evaluating this task. Reports BLURB score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2007.15779
  bibtex_key: gu2020domain
  confidence: high
---

# blurb-eval

> Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing — Gu et al. (2020) (arXiv:2007.15779, 2020)

## What this evaluates

Evaluates biomedical language models on a comprehensive suite of downstream NLP tasks, including named entity recognition, relation extraction, sentence similarity, document classification, and question answering. It measures how well domain-specific pretraining transfers to specialized clinical and biomedical text understanding.

## Datasets

- **BLURB** — total ?; splits: test (-1)

## Metrics

- `BLURB score` **(primary)** — range: percent
  - Macro average of the average test results across six downstream tasks: NER, PICO, relation extraction, sentence similarity, document classification, and question answering.
- `Task-specific F1 / Accuracy` — range: percent
  - Entity-level F1 for NER tasks; Micro F1 for relation extraction tasks; and standard accuracy or similarity scores for sentence similarity, document classification, and QA tasks.

## Input / output format

**Input**: Text snippets from biomedical literature (abstracts or full texts) formatted for specific downstream tasks such as entity recognition, relation extraction, sentence similarity, document classification, and question answering.

**Output**: Predicted labels or scores per instance (e.g., entity tags, relation types, similarity scores, class labels, or answer probabilities).

## Scoring recipe

```python
def compute_blurb_score(predictions, golds):
    task_scores = []
    for task in ['NER', 'PICO', 'RE', 'SS', 'DC', 'QA']:
        task_scores.append(average_test_results(task, predictions, golds))
    return mean(task_scores)

# Task-specific metrics use standard F1 or accuracy depending on the task type.
```

## Common pitfalls

- Using out-of-domain vocabulary or mixed-domain pretraining significantly degrades performance on biomedical tasks compared to domain-specific pretraining from scratch.
- Complex BIO tagging schemes for NER do not improve over simple IO tagging when using self-attention models like BERT, contrary to traditional sequential models.
- Adversarial pretraining and adding full-text PMC data can degrade performance unless training is extended significantly (e.g., 60% longer).

## Evidence (verbatim from paper)

> The BLURB score is the macro average of average test results for each of the six tasks (NER, PICO, relation extraction, sentence similarity, document classification, question answering). See [Table 3](#S2.T3 "Table 3 ‣ 2.3. BLURB: A Comprehensive Benchmark for Biomedical NLP ‣ 2. Methods ‣ Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing") for the evaluation metric used in each task.

## Citation

```bibtex
@misc{gu2020domain,
  title={Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing},
  author={Gu et al. (2020)},
  year={2020},
  note={arXiv:2007.15779}
}
```

- arXiv: 2007.15779