Wikidata Ned Eval

---
name: wikidata-ned-eval
description: Evaluates a model's ability to disambiguate named entities in text by matching them to correct Wikidata entries using graph-based representations. It probes how well different neural architectures leverage graph triplet information versus full graph topology for entity resolution. Use when the user wants to benchmark on Wikidata-Disamb, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1810.09164
  bibtex_key: cetoli2018nedgraphs
  confidence: medium
---

# wikidata-ned-eval

> Named Entity Disambiguation using Deep Learning on Graphs — Alberto Cetoli et al. (arXiv:1810.09164, 2018)

## What this evaluates

Evaluates a model's ability to disambiguate named entities in text by matching them to correct Wikidata entries using graph-based representations. It probes how well different neural architectures leverage graph triplet information versus full graph topology for entity resolution.

## Datasets

- **Wikidata-Disamb** — total ?; splits: dev (-1), test (-1); repo https://github.com/contextscout/ned-graphs

## Metrics

- `F1` **(primary)** — range: percent
  - Harmonic mean of precision and recall: F1 = 2 * (precision * recall) / (precision + recall). Reported as a percentage.

## Input / output format

**Input**: Input text containing a named entity mention, accompanied by a set of graph triplets representing the candidate entity's Wikidata graph.

**Output**: Disambiguated entity identifier (Wikidata ID) or classification label indicating the correct entity from the candidate set.

## Scoring recipe

```python
def compute_f1(precision, recall):
    if precision + recall == 0:
        return 0.0
    return 2 * (precision * recall) / (precision + recall)
# F1 is computed per model on dev/test splits and reported as percentage
```

## Common pitfalls

- GCN-based models underperform because they overemphasize graph topology/shape rather than triplet content, which is the key signal in this dataset.
- Statistical significance is not rigorously tested; the authors estimate error at 0.6% based on only two runs per model.
- The dataset consists of simple tree-like graphs, so models relying on complex topological features may not generalize to richer graph structures.

## Evidence (verbatim from paper)

> The simple vector distance baseline is seen here performing narrowly better than random chance, with F1 = 55.4% on the test set. ... The second best results of the paper is given by the RNN of triplets model, with F1 = 91.1% on the test set. ... Table 3: Results of our architectures expressed as a percentage (best results in bold).

## Citation

```bibtex
@misc{cetoli2018nedgraphs,
  title={Named Entity Disambiguation using Deep Learning on Graphs},
  author={Alberto Cetoli et al.},
  year={2018},
  note={arXiv:1810.09164}
}
```

- arXiv: 1810.09164