Entity Linking Eval

---
name: entity-linking-eval
description: Evaluates end-to-end entity linking systems on their ability to detect entity mentions and correctly disambiguate them to knowledge base entities. It specifically probes for systemic benchmark biases, such as overreliance on named entities, ambiguous disambiguation choices, and underrepresented entity types, by introducing fairer evaluation protocols. Use when the user wants to benchmark on Existing and new EL benchmarks, or asks about evaluating this task. Reports Micro F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.14937
  bibtex_key: bast2023fair
  confidence: high
---

# entity-linking-eval

> A Fair and In-Depth Evaluation of Existing End-to-End Entity Linking Systems — Bast et al. (2023) (arXiv:2305.14937, 2023)

## What this evaluates

Evaluates end-to-end entity linking systems on their ability to detect entity mentions and correctly disambiguate them to knowledge base entities. It specifically probes for systemic benchmark biases, such as overreliance on named entities, ambiguous disambiguation choices, and underrepresented entity types, by introducing fairer evaluation protocols.

## Datasets

- **Existing and new EL benchmarks** — total ?; splits: test (-1); repo https://github.com/ad-freiburg/fair-entity-linking-benchmarks

## Metrics

- `Micro F1` **(primary)** — range: [0, 1]
  - Micro-averaged F1 score computed over all mention-level predictions across the dataset. Global precision and recall are aggregated before computing F1.
- `Disambiguation Accuracy` — range: [0, 1]
  - The number of correctly linked entities divided by the number of correctly detected entity mentions. Ignores detection errors.
- `Disambiguation Error Rate` — range: [0, 1]
  - One minus the disambiguation accuracy.

## Input / output format

**Input**: Document text with annotated mention spans and candidate entity lists.

**Output**: Predicted entity ID for each mention span, or NIL if no entity matches.

## Scoring recipe

```python
tp = sum(1 for p, g in zip(preds, gold) if p == g and p != 'NIL')
fp = sum(1 for p, g in zip(preds, gold) if p != g and p != 'NIL')
fn = sum(1 for p, g in zip(preds, gold) if p == 'NIL' and g != 'NIL')
prec = tp / (tp + fp) if (tp + fp) > 0 else 0
rec = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
detected = tp + fn
linked = tp
disamb_acc = linked / detected if detected > 0 else 0
```

## Common pitfalls

- Micro-averaging can mask poor performance on rare or ambiguous entity types.
- Disambiguation accuracy is computed only on correctly detected mentions, ignoring detection errors.
- Fine-grained error categories (e.g., 'Lowercased', 'Ground truth NIL') require strict adherence to the ELEVANT tool's definitions to avoid misclassification.

## Evidence (verbatim from paper)

> We report micro precision, recall and F1 scores, both for the overall EL task and for the ER subtask. ... The disambiguation accuracy is defined as the correctly linked entities divided by the correctly detected entity mentions.

## Citation

```bibtex
@misc{bast2023fair,
  title={A Fair and In-Depth Evaluation of Existing End-to-End Entity Linking Systems},
  author={Bast et al. (2023)},
  year={2023},
  note={arXiv:2305.14937}
}
```

- arXiv: 2305.14937