Mokb6 Eval

---
name: mokb6-eval
description: This benchmark evaluates multilingual knowledge graph embedding models on the task of completing missing facts across six languages. It specifically probes the model's ability to leverage cross-lingual information flow, benefit from translated training triples, and retain facts when queried in different scripts. Use when the user wants to benchmark on mOKB6, or asks about evaluating this task. Reports H@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2211.06959
  bibtex_key: mittal2022mokb6
  confidence: high
---

# mokb6-eval

> mOKB6: A Multilingual Open Knowledge Base Completion Benchmark — Mittal et al. (2022) (arXiv:2211.06959, 2022)

## What this evaluates

This benchmark evaluates multilingual knowledge graph embedding models on the task of completing missing facts across six languages. It specifically probes the model's ability to leverage cross-lingual information flow, benefit from translated training triples, and retain facts when queried in different scripts.

## Datasets

- **mOKB6** — total ?; splits: test (-1); repo https://github.com/dair-iitd/mokb6.git

## Metrics

- `H@1` — range: percent
  - Hits@1 measures the percentage of queries where the correct entity is ranked exactly first among all candidates.
- `H@10` **(primary)** — range: percent
  - Hits@10 measures the percentage of queries where the correct entity is ranked within the top 10 predictions.
- `MRR` — range: percent
  - Mean Reciprocal Rank computes the average of the reciprocal ranks (1/rank) of the correct entity across all queries.

## Input / output format

**Input**: A query triple (head entity, relation) with a candidate tail entity to be ranked among a set of negative samples.

**Output**: A ranked list of candidate tail entities (or their scores) ordered by predicted likelihood.

## Scoring recipe

```python
def compute_metrics(ranks, n_queries):
    h10 = sum(1 for r in ranks if r <= 10) / n_queries * 100
    mrr = sum(1.0 / r for r in ranks) / n_queries * 100
    h1 = sum(1 for r in ranks if r == 1) / n_queries * 100
    return h1, h10, mrr
```

## Common pitfalls

- The test sets across the six languages contain identical underlying facts, so evaluating cross-lingual memorization requires training on one language's test set and testing on others, rather than using standard disjoint train/test splits.
- Performance heavily depends on script similarity (e.g., Latin vs. Devanagari/Han); models trained on Latin-script languages transfer poorly to non-Latin scripts, which can skew cross-lingual transfer conclusions if not accounted for.
- The 'Union' and 'Union+Trans' setups mix facts from all six languages during training, which can cause language interference or bias toward high-resource languages like English if not properly normalized.

## Evidence (verbatim from paper)

> For evaluation, we use three metrics — hits at rank 1 (H@1), hits at rank 10 (H@10), and mean reciprocal rank (MRR).

## Citation

```bibtex
@misc{mittal2022mokb6,
  title={mOKB6: A Multilingual Open Knowledge Base Completion Benchmark},
  author={Mittal et al. (2022)},
  year={2022},
  note={arXiv:2211.06959}
}
```

- arXiv: 2211.06959