Livefact Eval

---
name: livefact-eval
description: Evaluates LLMs on fake news detection under dynamic, time-evolving evidence streams. It probes both binary classification capability (Real/Fake) and reasoning capability (handling ambiguity when evidence is incomplete), while explicitly measuring benchmark data contamination and epistemic humility. Use when the user wants to benchmark on LiveFact November 2025 dataset, or asks about evaluating this task. Reports average_score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.04815
  bibtex_key: xu2026livefact
  confidence: high
---

# livefact-eval

> LiveFact: A Dynamic, Time-Aware Benchmark for LLM-Driven Fake News Detection — Cheng Xu et al. (2026) (arXiv:2604.04815, 2026)

## What this evaluates

Evaluates LLMs on fake news detection under dynamic, time-evolving evidence streams. It probes both binary classification capability (Real/Fake) and reasoning capability (handling ambiguity when evidence is incomplete), while explicitly measuring benchmark data contamination and epistemic humility.

## Datasets

- **LiveFact November 2025 dataset** — total ?; splits: test (-1); repo https://github.com/bebxy/livefact

## Metrics

- `average_score` **(primary)** — range: percent
  - Average of Classification Accuracy and Inference Accuracy across the dataset.
- `reasoning_gap` — range: percent
  - Inference Accuracy minus Classification Accuracy at temporal offset δ=-3.

## Input / output format

**Input**: A news claim paired with a dynamic, time-stamped evidence stream, evaluated at specific temporal offsets (δ) relative to the claim's publication.

**Output**: A single label enclosed in double brackets, e.g., [[Real]], [[Fake]], or [[Ambiguous]].

## Scoring recipe

```python
def compute_accuracy(preds, gold):
    correct = sum(1 for p, g in zip(preds, gold) if p == g)
    return (correct / len(gold)) * 100

class_acc = compute_accuracy(class_preds, class_gold)
inf_acc = compute_accuracy(inf_preds, inf_gold)
average_score = (class_acc + inf_acc) / 2
reasoning_gap = inf_acc - class_acc
```

## Common pitfalls

- Base models fail due to format non-compliance (not outputting [[LABEL]]), resulting in near-random parsed predictions rather than true reasoning failures.
- Reasoning-tuned models require extended output limits (1024 tokens) to generate Chain-of-Thought; truncating at 128 tokens yields near-zero accuracy despite strong underlying capabilities.
- Classification mode forces definitive verdicts even when evidence is absent, artificially penalizing models that should express uncertainty.

## Evidence (verbatim from paper)

> The Qwen3-235B-A22B-Instruct model not only leads the open-source sector but also outperforms proprietary flagship models, including gpt-5.1, achieving the highest average score of 72.40%.

## Citation

```bibtex
@misc{xu2026livefact,
  title={LiveFact: A Dynamic, Time-Aware Benchmark for LLM-Driven Fake News Detection},
  author={Cheng Xu et al. (2026)},
  year={2026},
  note={arXiv:2604.04815}
}
```

- arXiv: 2604.04815