Poly Fever Eval

---
name: poly-fever-eval
description: Evaluates large language models' ability to detect hallucinations by verifying factual claims across 11 languages. It probes cross-linguistic consistency, topic-aware fact-checking, and resistance to web-resource bias in multilingual settings. Use when the user wants to benchmark on Poly-FEVER, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.16541
  bibtex_key: zhang2025polyfever
  confidence: high
---

# poly-fever-eval

> Poly-FEVER: A Multilingual Fact Verification Benchmark for Hallucination Detection in Large Language Models — Zhang et al. (2025) (arXiv:2503.16541, 2025)

## What this evaluates

Evaluates large language models' ability to detect hallucinations by verifying factual claims across 11 languages. It probes cross-linguistic consistency, topic-aware fact-checking, and resistance to web-resource bias in multilingual settings.

## Datasets

- **Poly-FEVER** — total 77973; splits: full (77973)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Exact-match accuracy computed over binary veracity labels (true/false). Calculated as the proportion of claims where the model's predicted label matches the ground truth label.

## Input / output format

**Input**: A single textual claim provided in one of 11 languages (English, Mandarin Chinese, Hindi, Arabic, Bengali, Japanese, Korean, Tamil, Thai, Georgian, or Amharic).

**Output**: A binary veracity label: 'true' or 'false'.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_labels):
    if pred.strip().lower() == gold.strip().lower():
        correct += 1
return correct / len(gold_labels)
```

## Common pitfalls

- Original FEVER includes 'NotEnoughInfo' claims, but Poly-FEVER explicitly excludes them; evaluators must filter these out to avoid invalid comparisons.
- Non-English claims are machine-translated via Google Translate; poor translation quality in low-resource languages may be misattributed to model hallucination rather than linguistic artifacts.
- Topic distribution vectors are provided as metadata but are not part of the verification prompt; models may inadvertently learn topic priors that skew verification performance.

## Evidence (verbatim from paper)

> Poly-FEVER excludes claims labeled as NotEnoughInfo to focus on cases where factual accuracy can be measured directly. Each entry in Poly-FEVER contains four primary fields: ... Label: An annotated veracity label (true or false) indicating whether the claim aligns with established factual evidence.

## Citation

```bibtex
@misc{zhang2025polyfever,
  title={Poly-FEVER: A Multilingual Fact Verification Benchmark for Hallucination Detection in Large Language Models},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2503.16541}
}
```

- arXiv: 2503.16541