Irt2 Eval

---
name: irt2-eval
description: Evaluates neural and baseline models on inductive link prediction and ranking tasks across knowledge graphs of varying scales. It probes the models' ability to map textual entity mentions to graph vertices and rank candidate entities based on combined textual and structural signals, particularly under data scarcity conditions. Use when the user wants to benchmark on IRT2, or asks about evaluating this task. Reports MRR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2301.00716
  bibtex_key: hamann2023irt2
  confidence: high
---

# irt2-eval

> IRT2: Inductive Linking and Ranking in Knowledge Graphs of Varying Scale — Hamann et al. (2023) (arXiv:2301.00716, 2023)

## What this evaluates

Evaluates neural and baseline models on inductive link prediction and ranking tasks across knowledge graphs of varying scales. It probes the models' ability to map textual entity mentions to graph vertices and rank candidate entities based on combined textual and structural signals, particularly under data scarcity conditions.

## Datasets

- **IRT2** — total ?; splits: tiny (-1), small (-1), medium (-1), large (-1); repo https://github.com/lavis-nlp/irt2

## Metrics

- `hits@10` — range: [0, 1]
  - Fraction of test triples where the true entity is ranked within the top 10 predictions after target filtering.
- `hits@100` — range: [0, 1]
  - Fraction of test triples where the true entity is ranked within the top 100 predictions after target filtering.
- `MRR` **(primary)** — range: [0, 1]
  - Mean reciprocal rank of the true entity across all test triples. Calculated as the average of 1/rank for each query, where rank is the position of the true entity in the filtered prediction list.

## Input / output format

**Input**: Textual entity mentions (sentences) paired with a knowledge graph structure. For linking, a text context and a target graph; for ranking, a text context and a query relation.

**Output**: A ranked list of candidate entities (vertices) or a binary decision indicating the correct graph vertex for the mention.

## Scoring recipe

```python
def compute_metrics(predictions, gold, k=10):
    # Apply target filtering: keep only the single true positive of interest
    filtered = [x for x in predictions if x == gold] + \
               [x for x in predictions if x != gold and x not in other_true_positives]
    rank = filtered.index(gold) + 1 if gold in filtered else len(filtered) + 1
    hits_k = 1.0 if rank <= k else 0.0
    mrr = 1.0 / rank
    return hits_k, mrr
# Aggregate micro-averaged over all test triples
```

## Common pitfalls

- Target filtering must be applied to remove other true positives from the ranked list to avoid artificially worsening results for rankings with many true positives.
- The ranking task relies heavily on head prediction scores, which are often lower quality than tail predictions, leading to poor practical ranking performance despite neural models beating BM25.
- Models struggle to incorporate textual clues and rely heavily on structural embeddings, especially in larger datasets where the bag-of-words baseline can outperform them.

## Evidence (verbatim from paper)

> We measure hits@k and the mean reciprocal rank (MRR) for each test triple (i.e. micro-averaged). Following common practice, we apply target filtering [Sh19] to the scored predictions (mentions for ranking and vertices for ranking). Target filtering is a method where, for a given ranking, only a single true positive of interest is inspected, while all other true positives are removed from the list. This helps to not artificially worsen the result for rankings with many true positives.

## Citation

```bibtex
@misc{hamann2023irt2,
  title={IRT2: Inductive Linking and Ranking in Knowledge Graphs of Varying Scale},
  author={Hamann et al. (2023)},
  year={2023},
  note={arXiv:2301.00716}
}
```

- arXiv: 2301.00716