Entity Canonicalization Eval

---
name: entity-canonicalization-eval
description: Evaluates a model's ability to cluster entity mentions (noun phrases) into canonical forms within open knowledge graphs. It probes unsupervised representation learning and clustering capabilities without relying on manually annotated ground truth for training. Use when the user wants to benchmark on Base, Ambiguous, ReVerb45K, CanonicNell, or asks about evaluating this task. Reports Macro F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2012.04780
  bibtex_key: dash2020openkgcanonicalization
  confidence: high
---

# entity-canonicalization-eval

> Open Knowledge Graphs Canonicalization using Variational Autoencoders — Dash et al. (2020) (arXiv:2012.04780, 2020)

## What this evaluates

Evaluates a model's ability to cluster entity mentions (noun phrases) into canonical forms within open knowledge graphs. It probes unsupervised representation learning and clustering capabilities without relying on manually annotated ground truth for training.

## Datasets

- **Base** — total 290; splits: test (-1), val (-1); repo https://github.com/mallabiisc/cesi/
- **Ambiguous** — total 717; splits: test (-1), val (-1); repo https://github.com/mallabiisc/cesi/
- **ReVerb45K** — total 15500; splits: test (-1), val (-1); repo https://github.com/mallabiisc/cesi/
- **CanonicNell** — total 8700; splits: test (-1), val (-1); repo https://github.com/IBM/Open-KG-canonicalization

## Metrics

- `Macro F1` **(primary)** — range: [0, 1]
  - Averages the F1 score computed independently for each cluster, giving equal weight to clusters regardless of their size.
- `Micro F1` — range: [0, 1]
  - Aggregates true positives, false positives, and false negatives across all clusters to compute a global F1 score.
- `Pair F1` — range: [0, 1]
  - Evaluates clustering by considering all pairs of items. Counts true positives if pairs are in the same cluster in both predictions and gold, and false positives/negatives otherwise.

## Input / output format

**Input**: Noun phrases (entity mentions) from KG triples, optionally augmented with side information (e.g., IDF token overlap, pretrained embeddings, structural graph features).

**Output**: A cluster identifier (canonical label) assigned to each noun phrase.

## Scoring recipe

```python
def compute_f1_metrics(predictions, gold):
    # predictions, gold: lists of cluster IDs per noun phrase
    macro_f1 = mean(f1_per_cluster(p, g) for p, g in zip(predictions, gold))
    micro_f1 = global_precision_recall_f1(predictions, gold)
    pair_f1 = pair_f1_score(predictions, gold)
    mean_f1 = (macro_f1 + micro_f1 + pair_f1) / 3
    return {'macro_f1': macro_f1, 'micro_f1': micro_f1, 'pair_f1': pair_f1, 'mean_f1': mean_f1}
```

## Common pitfalls

- The task is strictly unsupervised; no training data is used, so hyperparameters are tuned solely on the validation set.
- Initial evaluations only considered head entity mentions; tail mentions were added later, causing score discrepancies across benchmarks.
- Pair F1 is highly sensitive to cluster size imbalance and often reports significantly lower values than Macro/Micro F1 on the same dataset.

## Evidence (verbatim from paper)

> Following Galárraga et al. (2014a), we use the macro, micro, and pair F1 scores for evaluations. The task is unsupervised in nature, hence we do not possess any training data. For CanonicNell, we did a random 80:20 split of the triples into validation and test folds.

## Citation

```bibtex
@misc{dash2020openkgcanonicalization,
  title={Open Knowledge Graphs Canonicalization using Variational Autoencoders},
  author={Dash et al. (2020)},
  year={2020},
  note={arXiv:2012.04780}
}
```

- arXiv: 2012.04780