Factcheck Kg Validation Eval

---
name: factcheck-kg-validation-eval
description: Evaluates large language models' ability to validate factual claims in knowledge graphs by classifying triples as true or false. It probes internal knowledge retrieval, retrieval-augmented generation (RAG) with external search results, and multi-model consensus strategies for fact-checking reliability. Use when the user wants to benchmark on FactBench, YAGO, DBpedia, or asks about evaluating this task. Reports Class-wise F1 Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.10748
  bibtex_key: shami2026benchmarking
  confidence: high
---

# factcheck-kg-validation-eval

> Benchmarking Large Language Models for Knowledge Graph Validation — Shami et al. (2026) (arXiv:2602.10748, 2026)

## What this evaluates

Evaluates large language models' ability to validate factual claims in knowledge graphs by classifying triples as true or false. It probes internal knowledge retrieval, retrieval-augmented generation (RAG) with external search results, and multi-model consensus strategies for fact-checking reliability.

## Datasets

- **FactBench** — total 2800; splits: test (-1); HF `FactCheck-AI/FactCheck`; repo https://github.com/FactCheck-AI/FactCheck
- **YAGO** — total 1386; splits: test (-1); HF `FactCheck-AI/FactCheck`; repo https://github.com/FactCheck-AI/FactCheck
- **DBpedia** — total 9344; splits: test (-1); HF `FactCheck-AI/FactCheck`; repo https://github.com/FactCheck-AI/FactCheck

## Metrics

- `Class-wise F1 Score` **(primary)** — range: [0, 1]
  - F1(c) = 2 * Precision(c) * Recall(c) / (Precision(c) + Recall(c)), calculated independently for True and False classes to handle class imbalance.
- `Consensus Alignment` — range: [0, 1]
  - CA_M = (1/|G|) * sum(I(response(M,t) == majorityVote(t))) over all facts G, measuring agreement between a model's predictions and the majority vote across all evaluated models.
- `Average Response Time` — range: other
  - IQR-filtered mean of per-fact response times in seconds, excluding outliers outside Q1-1.5*IQR and Q3+1.5*IQR.

## Input / output format

**Input**: A knowledge graph triple (subject, predicate, object) or a generated question derived from it, optionally accompanied by a fixed set of retrieved web documents (for RAG evaluation).

**Output**: Binary classification label: 'True' or 'False' indicating whether the fact/triple is valid.

## Scoring recipe

```python
def compute_metrics(predictions, gold, ensemble_preds=None):
    f1_scores = {}
    for c in [1, 0]:
        tp = sum(1 for p, g in zip(predictions, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1_scores[c] = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    ca = 0
    if ensemble_preds is not None:
        majority = [max(set(row), key=row.count) for row in zip(*ensemble_preds)]
        ca = sum(1 for p, m in zip(predictions, majority) if p == m) / len(predictions)
    return f1_scores, ca
```

## Common pitfalls

- YAGO has a 99% gold accuracy rate, so models may simply predict 'True' for all instances, inflating accuracy while failing to detect false facts. Class-wise F1 is required to expose this bias.
- Consensus Alignment measures agreement with the majority vote across all evaluated models, not agreement with ground truth. Misinterpreting it as a correctness metric leads to flawed conclusions.
- Live web search results change over time, breaking reproducibility. The benchmark provides a mock API with pre-fetched SERP results; using live queries instead violates the evaluation protocol.

## Evidence (verbatim from paper)

> To assess the effectiveness of the considered fact validation strategies, we focus on two key measures: Class-wise F1 Score and Consensus Alignment. These measures are chosen to account for class imbalance, capture per-class performance, and evaluate agreement for multi-model consensus approaches. The F1 score for a given class c∈{T,F} is defined as: F1(c)=2⋅Precision(c)⋅Recall(c)/(Precision(c)+Recall(c), where Precision(c) and Recall(c) denote the precision and recall calculated specifically for class c.

## Citation

```bibtex
@misc{shami2026benchmarking,
  title={Benchmarking Large Language Models for Knowledge Graph Validation},
  author={Shami et al. (2026)},
  year={2026},
  note={arXiv:2602.10748}
}
```

- arXiv: 2602.10748