Long Ner Eval

---
name: long-ner-eval
description: Evaluates named entity recognition capabilities, specifically probing a model's ability to handle class imbalance, out-of-vocabulary terms, and long or complex entity names across biomedical and general domain texts. Use when the user wants to benchmark on NCBI-disease, BC5CDR-disease, BC5CDR-chemical, BC4CHEMD, BC2GM, JNLPBA, LINNAEUS, Species-800, CoNLL-2003, WNUT-2017, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2104.07249
  bibtex_key: jeong2021regularization
  confidence: high
---

# long-ner-eval

> Regularization for Long Named Entity Recognition — Minbyul Jeong, Jaewoo Kang (2021) (arXiv:2104.07249, 2021)

## What this evaluates

Evaluates named entity recognition capabilities, specifically probing a model's ability to handle class imbalance, out-of-vocabulary terms, and long or complex entity names across biomedical and general domain texts.

## Datasets

- **NCBI-disease** — total ?; splits: train (-1), val (-1), test (-1)
- **BC5CDR-disease** — total ?; splits: train (-1), val (-1), test (-1)
- **BC5CDR-chemical** — total ?; splits: train (-1), val (-1), test (-1)
- **BC4CHEMD** — total ?; splits: train (-1), val (-1), test (-1)
- **BC2GM** — total ?; splits: train (-1), val (-1), test (-1)
- **JNLPBA** — total ?; splits: train (-1), val (-1), test (-1)
- **LINNAEUS** — total ?; splits: train (-1), val (-1), test (-1)
- **Species-800** — total ?; splits: train (-1), val (-1), test (-1)
- **CoNLL-2003** — total ?; splits: train (-1), val (-1), test (-1)
- **WNUT-2017** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall calculated over all predicted entity mentions. Precision is the fraction of predicted entities that are correct, and recall is the fraction of gold entities that are correctly predicted.
- `Recall` — range: [0, 1]
  - Fraction of gold standard entity mentions correctly identified by the model. The paper partitions recall into Memorization (Mem), Synonym (Syn), Concept (Con), and Unseen categories to evaluate generalization.

## Input / output format

**Input**: Tokenized text sentences with corresponding token-level entity labels.

**Output**: Token-level BIO/IOB entity labels predicting the entity type and boundary for each token in the input sequence.

## Scoring recipe

```python
def calculate_ner_f1(preds, gold):
    pred_set = set(extract_mentions(preds))
    gold_set = set(extract_mentions(gold))
    tp = len(pred_set & gold_set)
    fp = len(pred_set - gold_set)
    fn = len(gold_set - pred_set)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0

def extract_mentions(labels):
    mentions = set()
    curr_type, start = None, None
    for i, lbl in enumerate(labels):
        if lbl.startswith('B-'):
            if curr_type: mentions.add((curr_type, start, i-1))
            curr_type, start = lbl[2:], i
        elif lbl.startswith('I-') and curr_type:
            pass
        else:
            if curr_type: mentions.add((curr_type, start, i-1))
            curr_type, start = None, None
    if curr_type: mentions.add((curr_type, start, len(labels)-1))
    return mentions
```

## Common pitfalls

- The paper partitions datasets into Memorization (Mem), Synonym (Syn), Concept (Con), and Unseen categories; reporting only aggregate F1 masks performance differences across these generalization splits.
- Entity length heavily influences results; models often perform well on short entities but drop significantly on long entities (≥8 tokens), requiring length-stratified evaluation.
- Out-of-vocabulary (OOV) handling relies on subword tokenization rather than word-level frequency, so evaluating at the word level without considering subword debiasing will misrepresent OOV performance.

## Evidence (verbatim from paper)

> Using three components, namely Subword, Class, and Temp, showed significant improvements in recall on Syn and Con, as well as overall improvements to in-domain performance (F1).

## Citation

```bibtex
@misc{jeong2021regularization,
  title={Regularization for Long Named Entity Recognition},
  author={Minbyul Jeong, Jaewoo Kang (2021)},
  year={2021},
  note={arXiv:2104.07249}
}
```

- arXiv: 2104.07249