Kb Completion Eval

---
name: kb-completion-eval
description: Evaluates a model's ability to learn first-order logic rules for knowledge base completion and object classification. It probes rule generation efficiency, scalability to longer rules, and few-shot generalization on relational data. Use when the user wants to benchmark on Even-and-Successor (ES), FB15K-237, WN18, Visual Genome (via GQA), or asks about evaluating this task. Reports MRR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1910.02481
  bibtex_key: yang2019learn
  confidence: high
---

# kb-completion-eval

> Learn to Explain Efficiently via Neural Logic Inductive Learning — Yang et al. (2019) (arXiv:1910.02481, 2019)

## What this evaluates

Evaluates a model's ability to learn first-order logic rules for knowledge base completion and object classification. It probes rule generation efficiency, scalability to longer rules, and few-shot generalization on relational data.

## Datasets

- **Even-and-Successor (ES)** — total ?; splits: train (-1), valid (-1), test (-1)
- **FB15K-237** — total ?; splits: train (-1), valid (-1), test (-1)
- **WN18** — total ?; splits: train (-1), valid (-1), test (-1)
- **Visual Genome (via GQA)** — total ?; splits: train (-1), test (-1)

## Metrics

- `MRR` **(primary)** — range: [0, 1]
  - Mean Reciprocal Rank: the average of 1/rank for the correct answer across all queries.
- `Hits@10` — range: [0, 1]
  - Fraction of queries where the correct answer is ranked in the top 10 predictions.
- `R@1` — range: [0, 1]
  - Accuracy of predicting the correct object class label at rank 1.
- `R@5` — range: [0, 1]
  - Accuracy of predicting the correct object class label within the top 5 predictions.

## Input / output format

**Input**: Relational knowledge base triples (subject, relation, object) or scene-graphs; query triplets for KB completion; object-relation pairs for classification.

**Output**: Probability score for a fact triplet being present in the KB; predicted object class label.

## Scoring recipe

```python
def compute_metrics(predictions, gold_indices):
    ranks = []
    for pred, gold in zip(predictions, gold_indices):
        sorted_indices = np.argsort(-pred)
        rank = np.where(sorted_indices == gold)[0][0] + 1
        ranks.append(rank)
    mrr = np.mean(1.0 / ranks)
    hits10 = np.mean([r <= 10 for r in ranks])
    r1 = np.mean([r == 1 for r in ranks])
    r5 = np.mean([r <= 5 for r in ranks])
    return mrr, hits10, r1, r5
```

## Common pitfalls

- The Even-and-Successor benchmark is noise-free, so the paper only reports wall-clock time for it rather than accuracy scores.
- The Visual Genome dataset is highly noisy; the authors use a pre-processed GQA version and filter predicates with fewer than 1500 occurrences, which alters the original data distribution.

## Evidence (verbatim from paper)

> We use Mean Reciprocal Ranks (MRR) and Hits@10 for evaluation metrics (see Appendix[C](#A3 "Appendix C Experiments ‣ Learn to Explain Efficiently via Neural Logic Inductive Learning") for details). Quantitatively, we evaluate the learned rules on predicting the object class labels on a held-out set in terms of their R@1 and R@5.

## Citation

```bibtex
@misc{yang2019learn,
  title={Learn to Explain Efficiently via Neural Logic Inductive Learning},
  author={Yang et al. (2019)},
  year={2019},
  note={arXiv:1910.02481}
}
```

- arXiv: 1910.02481