Llmke Wikidata Eval

---
name: llmke-wikidata-eval
description: Evaluates LLMs' ability to predict object entities given subject-relation pairs in Wikidata, testing knowledge retrieval, entity disambiguation, and domain-specific reasoning across 21 relations spanning 7 domains. Use when the user wants to benchmark on ISWC 2023 LM-KBC Challenge dataset, or asks about evaluating this task. Reports F1-score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2309.08491
  bibtex_key: zhang2023llmke
  confidence: high
---

# llmke-wikidata-eval

> Using Large Language Models for Knowledge Engineering (LLMKE): A Case Study on Wikidata — Zhang et al. (2023) (arXiv:2309.08491, 2023)

## What this evaluates

Evaluates LLMs' ability to predict object entities given subject-relation pairs in Wikidata, testing knowledge retrieval, entity disambiguation, and domain-specific reasoning across 21 relations spanning 7 domains.

## Datasets

- **ISWC 2023 LM-KBC Challenge dataset** — total ?; splits: train (1940), val (1940), test (1940); repo https://github.com/bohuizhang/LLMKE

## Metrics

- `F1-score` **(primary)** — range: [0, 1]
  - Harmonic mean of precision and recall: F1 = 2 * (P * R) / (P + R). Computed per relation and averaged macroscopically across the 21 relations.

## Input / output format

**Input**: Subject entity, relation type, and optionally few-shot examples or retrieved context (e.g., Wikipedia introduction/infobox, IMDb data) formatted as prompts for question answering or triple completion.

**Output**: Predicted object entity string, which is then mapped to a Wikidata QID via disambiguation methods (baseline, improved keyword/LM/case-based).

## Scoring recipe

```python
def compute_macro_f1(predictions, golds):
    relation_f1s = []
    for rel in relations:
        preds_rel = [p for p, g in zip(predictions, golds) if g['rel'] == rel]
        golds_rel = [g for p, g in zip(predictions, golds) if g['rel'] == rel]
        tp = sum(1 for p, g in zip(preds_rel, golds_rel) if p in g)
        fp = sum(1 for p, g in zip(preds_rel, golds_rel) if p not in g)
        fn = sum(1 for p, g in zip(preds_rel, golds_rel) if not any(gi in p for gi in g))
        p = tp / (tp + fp) if (tp + fp) > 0 else 0
        r = tp / (tp + fn) if (tp + fn) > 0 else 0
        relation_f1s.append(2 * p * r / (p + r) if (p + r) > 0 else 0)
    return sum(relation_f1s) / len(relation_f1s)
```

## Common pitfalls

- Disambiguation errors: LLMs may predict correct entity strings but assign wrong QIDs, or use aliases absent from Wikidata's label/alias list.
- Context mismatch: Retrieval-augmented context from Wikipedia may not align with Wikidata's entity representations, sometimes hurting performance.
- Multi-answer handling: Relations can have 0 to 20 valid objects; F1 calculation must account for variable answer sets per instance.

## Evidence (verbatim from paper)

> It has 1,940 statements for each train, validation, and test sets. The results reported are based on the test set. In the dataset, the minimum and maximum number of object-entities for each relation is different, ranging from 0 to 20. The best F1-scores among the three settings and two disambiguation methods of the models are highlighted.

## Citation

```bibtex
@misc{zhang2023llmke,
  title={Using Large Language Models for Knowledge Engineering (LLMKE): A Case Study on Wikidata},
  author={Zhang et al. (2023)},
  year={2023},
  note={arXiv:2309.08491}
}
```

- arXiv: 2309.08491