Nested Ner Eval

---
name: nested-ner-eval
description: Evaluates a model's ability to identify and classify named entities that can overlap or be contained within other entities (nested NER) across multiple domains. It probes the model's span-level understanding and label assignment capabilities in complex textual contexts. Use when the user wants to benchmark on ACE2004, ACE2005, GENIA, KBP2017, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2110.07480
  bibtex_key: yuan2021triaffine
  confidence: high
---

# nested-ner-eval

> Fusing Heterogeneous Factors with Triaffine Mechanism for Nested Named Entity Recognition — Zheng Yuan et al. (2021) (arXiv:2110.07480, 2021)

## What this evaluates

Evaluates a model's ability to identify and classify named entities that can overlap or be contained within other entities (nested NER) across multiple domains. It probes the model's span-level understanding and label assignment capabilities in complex textual contexts.

## Datasets

- **ACE2004** — total ?; splits: train (6200), dev (745), test (812)
- **ACE2005** — total ?; splits: train (7194), dev (969), test (1047)
- **GENIA** — total ?; splits: train (16692), test (1854)
- **KBP2017** — total ?; splits: train (10546), dev (545), test (4267)

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - Computed at the span level: Precision = |correctly predicted spans| / |predicted spans|, Recall = |correctly predicted spans| / |gold spans|, F1 = 2 * Precision * Recall / (Precision + Recall). Only exact matches of span boundaries and entity labels count as correct.

## Input / output format

**Input**: Tokenized sentence with contextual embeddings (BERT/BioBERT), supplemented by fastText word embeddings, POS embeddings, and character-level BiLSTM embeddings. Spans are implicitly formed by token indices.

**Output**: A set of predicted spans, each defined by a start token index, an end token index, and a predicted entity label.

## Scoring recipe

```python
def compute_span_f1(preds, golds):
    correct = sum(1 for p in preds if p in golds)
    precision = correct / len(preds) if preds else 0
    recall = correct / len(golds) if golds else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return f1
```

## Common pitfalls

- Using token-level or sequence-labeling metrics (e.g., BIO accuracy) instead of exact span-level matching for nested entities.
- Ignoring the specific dataset splits mandated by prior works (Lu & Roth 2015 for ACE, Lin et al. 2019 for GENIA/KBP), which differ from standard LDC splits.
- Failing to handle overlapping/nested spans correctly during evaluation, leading to inflated or deflated scores if not matched exactly by (start, end, label) tuples.

## Evidence (verbatim from paper)

> Following previous work, we measure the results using span-level precision, recall, and $F_{1}$ scores.

## Citation

```bibtex
@misc{yuan2021triaffine,
  title={Fusing Heterogeneous Factors with Triaffine Mechanism for Nested Named Entity Recognition},
  author={Zheng Yuan et al. (2021)},
  year={2021},
  note={arXiv:2110.07480}
}
```

- arXiv: 2110.07480