Factir Eval

---
name: factir-eval
description: Evaluates open-domain retrieval and re-ranking systems on real-world fact-checking claims. It probes the ability to retrieve indirect, multifaceted evidence from unstructured web sources to support or refute complex queries involving health, politics, and economics. Use when the user wants to benchmark on FactIR, or asks about evaluating this task. Reports nDCG@k.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.06006
  bibtex_key: venktesh2025factir
  confidence: high
---

# factir-eval

> FactIR: A Real-World Zero-shot Open-Domain Retrieval Benchmark for Fact-Checking — Venktesh et al. (2025) (arXiv:2502.06006, 2025)

## What this evaluates

Evaluates open-domain retrieval and re-ranking systems on real-world fact-checking claims. It probes the ability to retrieve indirect, multifaceted evidence from unstructured web sources to support or refute complex queries involving health, politics, and economics.

## Datasets

- **FactIR** — total ?; splits: test (-1); repo https://github.com/factiverse/factIR

## Metrics

- `nDCG@k` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at cutoff k. Computed as DCG@k divided by the ideal DCG@k, where DCG discounts relevance by the logarithm of the document's rank position.
- `Recall@k` — range: [0, 1]
  - The proportion of relevant ground-truth documents retrieved within the top-k ranked results.

## Input / output format

**Input**: A fact-checking claim (served as the query) and a large corpus of unstructured web documents.

**Output**: A ranked list of retrieved documents (or top-k document IDs) for each claim.

## Scoring recipe

```python
def compute_metrics(predictions, gold, k):
    recall_k = len(set(predictions[:k]) & gold) / len(gold)
    dcg = sum(1 / math.log2(i + 2) for i, doc in enumerate(predictions[:k]) if doc in gold)
    idcg = sum(1 / math.log2(i + 2) for i in range(min(len(gold), k)))
    ndcg_k = dcg / idcg if idcg > 0 else 0.0
    return ndcg_k, recall_k
```

## Common pitfalls

- Queries are real-world production claims requiring indirect reasoning, not simple keyword searches.
- Relevance is multi-faceted (partial relevance, stance), so binary relevance assumptions can mislead evaluation.
- The cutoff k is not fixed in the protocol; results must explicitly state the k value used.

## Evidence (verbatim from paper)

> Metrics: We choose Normalized Discounted Cumulative Gain (nDCG@k) and Recall@k as the primary metrics for our results.

## Citation

```bibtex
@misc{venktesh2025factir,
  title={FactIR: A Real-World Zero-shot Open-Domain Retrieval Benchmark for Fact-Checking},
  author={Venktesh et al. (2025)},
  year={2025},
  note={arXiv:2502.06006}
}
```

- arXiv: 2502.06006