Conve Kg Link Prediction Eval

---
name: conve-kg-link-prediction-eval
description: Evaluates knowledge graph link prediction by measuring a model's ability to infer missing entities (head or tail) from subject-relation triples. It probes the expressiveness of 2D convolutional embeddings and tests robustness against test-set leakage via inverse relations. Use when the user wants to benchmark on WN18, FB15k, YAGO3-10, Countries, FB15k-237, WN18RR, or asks about evaluating this task. Reports MRR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1707.01476
  bibtex_key: dettmers2017conve
  confidence: high
---

# conve-kg-link-prediction-eval

> Convolutional 2D Knowledge Graph Embeddings — Dettmers et al. (2017) (arXiv:1707.01476, 2017)

## What this evaluates

Evaluates knowledge graph link prediction by measuring a model's ability to infer missing entities (head or tail) from subject-relation triples. It probes the expressiveness of 2D convolutional embeddings and tests robustness against test-set leakage via inverse relations.

## Datasets

- **WN18** — total 151442; splits: train (-1), val (-1), test (-1)
- **FB15k** — total ?; splits: train (-1), val (-1), test (-1)
- **YAGO3-10** — total ?; splits: train (-1), val (-1), test (-1)
- **Countries** — total ?; splits: train (-1), val (-1), test (-1)
- **FB15k-237** — total ?; splits: train (-1), val (-1), test (-1)
- **WN18RR** — total 93003; splits: train (-1), val (-1), test (-1)

## Metrics

- `MRR` **(primary)** — range: [0, 1]
  - Mean Reciprocal Rank: the average of the reciprocal ranks of the correct entity across all test triples. Calculated as (1/N) * Σ (1/rank_i).
- `Hits@1` — range: [0, 1]
  - Fraction of test triples where the correct entity appears at rank 1 in the predicted ranking.
- `Hits@3` — range: [0, 1]
  - Fraction of test triples where the correct entity appears in the top 3 predicted ranks.
- `Hits@10` — range: [0, 1]
  - Fraction of test triples where the correct entity appears in the top 10 predicted ranks.
- `AUC-PR` — range: [0, 1]
  - Area Under the Precision-Recall Curve, used specifically for the Countries dataset due to its high variance and multi-task nature.

## Input / output format

**Input**: Triple (subject, relation, object) with one entity masked (head or tail prediction), evaluated against a candidate set of all entities in the knowledge graph.

**Output**: Ranked list of candidate entities sorted by predicted score, or top-k predictions.

## Scoring recipe

```python
ranks = []
for triple in test_set:
    scores = model.predict(triple)
    rank = rank_of_correct_entity(scores)
    ranks.append(rank)
mrr = sum(1.0 / r for r in ranks) / len(ranks)
hits_at_k = sum(1 for r in ranks if r <= k) / len(ranks)
```

## Common pitfalls

- Test set leakage via inverse relations causes inflated scores on original WN18 and FB15k; must use filtered versions (WN18RR, FB15k-237).
- Countries dataset exhibits high variance; results must be averaged over 10 runs with 95% confidence intervals.
- Early stopping is based on validation MRR/AUC-PR, not test set performance.

## Evidence (verbatim from paper)

> We selected the hyperparameters of our ConvE model via grid search according to the mean reciprocal rank (MRR) on the validation set. ... We use early stopping according to the mean reciprocal rank (WN18, FB15k, YAGO3-10) and AUC-PR (Countries) statistics on the validation set, which we evaluate every three epochs. Unlike the other datasets, for Countries the results have a high variance, as such we average 10 runs and produce 95% confidence intervals.

## Citation

```bibtex
@misc{dettmers2017conve,
  title={Convolutional 2D Knowledge Graph Embeddings},
  author={Dettmers et al. (2017)},
  year={2017},
  note={arXiv:1707.01476}
}
```

- arXiv: 1707.01476