Biobert Re Eval

---
name: biobert-re-eval
description: Tests a model's capability to extract biomedical relations (gene-disease, gene-chemical) from text using minimal task-specific modifications. It probes whether domain-specific pre-training improves relation classification on small-scale biomedical corpora. Use when the user wants to benchmark on GAD, EU-ADR, CHEMPROT, or asks about evaluating this task. Reports entity-level F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1901.08746
  bibtex_key: lee2019biobert
  confidence: high
---

# biobert-re-eval

> BioBERT: a pre-trained biomedical language representation model for biomedical text mining — Lee et al. (2019) (arXiv:1901.08746, 2019)

## What this evaluates

Tests a model's capability to extract biomedical relations (gene-disease, gene-chemical) from text using minimal task-specific modifications. It probes whether domain-specific pre-training improves relation classification on small-scale biomedical corpora.

## Datasets

- **GAD** — total ?; splits: test (-1)
- **EU-ADR** — total ?; splits: test (-1)
- **CHEMPROT** — total ?; splits: test (-1)

## Metrics

- `entity-level F1` **(primary)** — range: percent
  - Harmonic mean of precision and recall calculated at the entity pair level. P = correct relations / predicted relations, R = correct relations / gold relations, F1 = 2PR/(P+R).

## Input / output format

**Input**: Biomedical text containing entity pairs.

**Output**: Predicted relation labels between entity pairs.

## Scoring recipe

```python
def compute_entity_f1(preds, gold):
  pred_set = set(preds)
  gold_set = set(gold)
  tp = len(pred_set & gold_set)
  fp = len(pred_set - gold_set)
  fn = len(gold_set - pred_set)
  p = tp / (tp + fp) if (tp + fp) > 0 else 0
  r = tp / (tp + fn) if (tp + fn) > 0 else 0
  f1 = 2 * p * r / (p + r) if (p + r) > 0 else 0
  return p, r, f1
```

## Common pitfalls

- Using complex linguistic features or multi-task learning architectures that differ from the single-architecture setup used for BERT/BioBERT.
- Not reporting entity-level metrics as specified in the protocol.
- Averaging results across datasets without weighting or reporting per-dataset scores separately.

## Evidence (verbatim from paper)

> The RE results are shown in Table 5. Unlike the NER results, BERT achieves better performance than that of the state-of-the-art models, which demonstrates its effectiveness in RE. On average, BioBERT (+ PubMed + PMC) outperformed the state-of-the-art models by 3.49 in terms of F1 score. On 2 out of 3 biomedical RE datasets, BioBERT achieved state-of-the-art performance in terms of F1 score. Considering the complexity of the state-of-the-art models which use numerous linguistic features (Bhasuran and Natarajan, 2018), the results are very promising.

## Citation

```bibtex
@misc{lee2019biobert,
  title={BioBERT: a pre-trained biomedical language representation model for biomedical text mining},
  author={Lee et al. (2019)},
  year={2019},
  note={arXiv:1901.08746}
}
```

- arXiv: 1901.08746