Unifact Eval

---
name: unifact-eval
description: Evaluates large language models' factual correctness by dynamically generating responses to factual questions and measuring how well hallucination detection and fact verification methods can predict the ground-truth factuality label of those responses. It probes a model's susceptibility to hallucination and the effectiveness of external evidence retrieval in verifying generated claims. Use when the user wants to benchmark on TriviaQA, NQ-Open, PopQA, 2WikiMultihopQA, HotpotQA, or asks about evaluating this task. Reports factuality prediction accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.02772
  bibtex_key: su2025unifact
  confidence: high
---

# unifact-eval

> Towards Unification of Hallucination Detection and Fact Verification for Large Language Models — Su et al. (2025) (arXiv:2512.02772, 2025)

## What this evaluates

Evaluates large language models' factual correctness by dynamically generating responses to factual questions and measuring how well hallucination detection and fact verification methods can predict the ground-truth factuality label of those responses. It probes a model's susceptibility to hallucination and the effectiveness of external evidence retrieval in verifying generated claims.

## Datasets

- **TriviaQA** — total 500; splits: test (500)
- **NQ-Open** — total 500; splits: test (500)
- **PopQA** — total 500; splits: test (500)
- **2WikiMultihopQA** — total 500; splits: test (500)
- **HotpotQA** — total 500; splits: test (500)

## Metrics

- `factuality prediction accuracy` **(primary)** — range: [0, 1]
  - Binary accuracy calculated as the proportion of correctly predicted factuality labels (Consistent/Inconsistent) out of the total test instances.

## Input / output format

**Input**: For HD methods: question prompt q and real-time internal model states/features captured during generation. For FV methods: generated text y_gen and retrieved external documents from an external corpus (e.g., Wikipedia/DPR).

**Output**: Binary prediction of factuality: Consistent or Inconsistent.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

# Usage:
# predictions: list of model outputs ('Consistent' or 'Inconsistent')
# gold_labels: list of ground truth labels from automated annotation
# score = compute_accuracy(predictions, gold_labels)
```

## Common pitfalls

- Static benchmarks cannot evaluate HD on new or OOD models due to model obsolescence and lack of real-time internal states.
- FV methods in this framework retrieve evidence from an external corpus, unlike the automated judge which has privileged access to ground truth, creating a distribution shift in real-world FV evaluation.
- Automated labeling relies heavily on the judge model's access to ground-truth answers and evidence, which may not reflect zero-shot fact-checking scenarios.

## Evidence (verbatim from paper)

> Both paradigms aim to predict the same ground truth $l^*$, but HD is tested on its utilization of internal model uncertainty, while FV is tested on its ability to retrieve and reason over external knowledge.

## Citation

```bibtex
@misc{su2025unifact,
  title={Towards Unification of Hallucination Detection and Fact Verification for Large Language Models},
  author={Su et al. (2025)},
  year={2025},
  note={arXiv:2512.02772}
}
```

- arXiv: 2512.02772