Biobert Ner Eval

---
name: biobert-ner-eval
description: Evaluates a model's ability to identify and classify biomedical entities (diseases, drugs/chemicals, genes/proteins, species) in text using transfer learning from domain-specific pre-training. It tests whether contextualized representations improve entity boundary detection and classification on small biomedical corpora. Use when the user wants to benchmark on NCBI disease, 2010 i2b2/VA, BC5CDR, BC4CHEMD, BC2GM, JNLPBA, LINNAEUS, Species-800, or asks about evaluating this task. Reports entity-level F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1901.08746
  bibtex_key: lee2019biobert
  confidence: high
---

# biobert-ner-eval

> BioBERT: a pre-trained biomedical language representation model for biomedical text mining — Lee et al. (2019) (arXiv:1901.08746, 2019)

## What this evaluates

Evaluates a model's ability to identify and classify biomedical entities (diseases, drugs/chemicals, genes/proteins, species) in text using transfer learning from domain-specific pre-training. It tests whether contextualized representations improve entity boundary detection and classification on small biomedical corpora.

## Datasets

- **NCBI disease** — total ?; splits: test (-1)
- **2010 i2b2/VA** — total ?; splits: test (-1)
- **BC5CDR** — total ?; splits: test (-1)
- **BC4CHEMD** — total ?; splits: test (-1)
- **BC2GM** — total ?; splits: test (-1)
- **JNLPBA** — total ?; splits: test (-1)
- **LINNAEUS** — total ?; splits: test (-1)
- **Species-800** — total ?; splits: test (-1)

## Metrics

- `entity-level F1` **(primary)** — range: percent
  - Harmonic mean of precision and recall calculated at the entity span level. P = correct entities / predicted entities, R = correct entities / gold entities, F1 = 2PR/(P+R).

## Input / output format

**Input**: Raw biomedical text or sentences.

**Output**: Predicted entity spans with corresponding labels (e.g., Disease, Gene, Chemical, Species).

## Scoring recipe

```python
def compute_entity_f1(preds, gold):
  pred_set = set(preds)
  gold_set = set(gold)
  tp = len(pred_set & gold_set)
  fp = len(pred_set - gold_set)
  fn = len(gold_set - pred_set)
  p = tp / (tp + fp) if (tp + fp) > 0 else 0
  r = tp / (tp + fn) if (tp + fn) > 0 else 0
  f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0
  return p, r, f1
```

## Common pitfalls

- Evaluating at token level instead of entity level as explicitly specified.
- Not averaging F1 scores across all 9 datasets for the overall comparison.
- Comparing against state-of-the-art models that use multi-task learning or different architectures, which BioBERT explicitly avoids.

## Evidence (verbatim from paper)

> For the evaluation metric, we used entity level precision, recall, and f1 score. First, we observe that BERT which was pre-trained on only the general domain corpus is quite effective. However, on average, performance of BERT was lower than that of state-of-the-art models by 2.28 in terms of F1 score. BioBERT achieves higher scores than BERT on all the datasets. On 6 out of 9 datasets, BioBERT even outperformed the current state-of-the-art models, and BioBERT (+ PubMed + PMC) outperformed the state-of-the-art models by 0.51 in terms of F1 score on average.

## Citation

```bibtex
@misc{lee2019biobert,
  title={BioBERT: a pre-trained biomedical language representation model for biomedical text mining},
  author={Lee et al. (2019)},
  year={2019},
  note={arXiv:1901.08746}
}
```

- arXiv: 1901.08746