Llm Qe Fact Verification Eval

---
name: llm-qe-fact-verification-eval
description: Evaluates LLM-based query expansion methods for evidence retrieval and claim verification. It measures retrieval quality and final verdict accuracy, while also analyzing whether generated documents contain sentences entailed by ground-truth evidence to detect knowledge leakage. Use when the user wants to benchmark on FEVER, SciFact, AVeriTeC, or asks about evaluating this task. Reports Recall@5, F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.14175
  bibtex_key: yoon2025hypothetical
  confidence: high
---

# llm-qe-fact-verification-eval

> Hypothetical Documents or Knowledge Leakage? Rethinking LLM-based Query Expansion — Yejun Yoon et al. (2025) (arXiv:2504.14175, 2025)

## What this evaluates

Evaluates LLM-based query expansion methods for evidence retrieval and claim verification. It measures retrieval quality and final verdict accuracy, while also analyzing whether generated documents contain sentences entailed by ground-truth evidence to detect knowledge leakage.

## Datasets

- **FEVER** — total ?; splits: test (-1)
- **SciFact** — total ?; splits: test (-1)
- **AVeriTeC** — total ?; splits: test (-1)

## Metrics

- `Recall@5` **(primary)** — range: [0, 1]
  - Fraction of gold evidence sentences correctly retrieved in the top-5 results.
- `NDCG@5` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 5, measuring ranked retrieval quality.
- `F1` **(primary)** — range: [0, 1]
  - Macro F1 score for the final verdict prediction (True/False/NotEnoughInfo) using retrieved evidence.
- `METEOR` — range: [0, 100]
  - Token-level similarity between retrieved and reference evidence sentences, used for AVeriTeC.
- `BERTScore` — range: [0, 1]
  - Embedding-level similarity between retrieved and reference evidence sentences, used for AVeriTeC.

## Input / output format

**Input**: Claim query (c) and a knowledge store (K) containing candidate evidence sentences.

**Output**: Top-5 retrieved evidence sentences; a predicted verdict (True, False, or NotEnoughInfo).

## Scoring recipe

```python
def evaluate(query, retrieved_top5, gold_evidence, gold_verdict):
    recall = len(set(retrieved_top5) & set(gold_evidence)) / len(gold_evidence)
    ndcg = compute_ndcg(retrieved_top5, gold_evidence)
    f1 = macro_f1(predict_verdict(query, retrieved_top5), gold_verdict)
    meteor = compute_meteor(retrieved_top5, gold_evidence)
    bertscore = compute_bertscore(retrieved_top5, gold_evidence)
    return recall, ndcg, f1, meteor, bertscore
```

## Common pitfalls

- AVeriTeC uses human-written evidence rather than extracts from a shared knowledge store, requiring sentence-level similarity metrics (METEOR, BERTScore) instead of exact-match retrieval metrics.
- Performance gains from LLM-based query expansion are heavily driven by knowledge leakage; scores on claims where generated documents contain entailed sentences are significantly inflated compared to unmatched claims.
- Retrieval metrics are computed on the top-5 retrieved sentences, but the final verdict prediction uses all 5 retrieved pieces of evidence as context for a separate LLM.

## Evidence (verbatim from paper)

> For evidence retrieval, we used Recall@k and NDCG@k (k=5) as evaluation metrics on the FEVER and SciFact datasets, where both the ground-truth evidence E and retrieved evidence ě come from the knowledge store K. In contrast, ě in AveriTeC consists of human-written evidence rather than extracts from K. Therefore, following previous studies, we applied the Hungarian algorithm with METEOR and BERTScore on the top five retrieved sentences, computing token-level and embedding-level similarity, respectively, based on a binary assignment between generated and reference sequences. For verdict prediction, we used GPT-4o-mini with the five retrieved evidence and evaluated performance using macro F1.

## Citation

```bibtex
@misc{yoon2025hypothetical,
  title={Hypothetical Documents or Knowledge Leakage? Rethinking LLM-based Query Expansion},
  author={Yejun Yoon et al. (2025)},
  year={2025},
  note={arXiv:2504.14175}
}
```

- arXiv: 2504.14175