Webfaq Retrieval Eval

---
name: webfaq-retrieval-eval
description: Evaluates multilingual dense retrieval models on natural Q&A pairs by measuring ranking quality against gold answers. It tests the model's ability to retrieve relevant FAQ documents across multiple languages and assesses zero-shot generalization to other Wikipedia-based benchmarks. Use when the user wants to benchmark on WebFAQ, Mr. TyDi, MIRACL (Hard Negatives), or asks about evaluating this task. Reports NDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.20936
  bibtex_key: dinzinger2025webfaq
  confidence: high
---

# webfaq-retrieval-eval

> WebFAQ: A Multilingual Collection of Natural Q&A Datasets for Dense Retrieval — Dinzinger et al. (2025) (arXiv:2502.20936, 2025)

## What this evaluates

Evaluates multilingual dense retrieval models on natural Q&A pairs by measuring ranking quality against gold answers. It tests the model's ability to retrieve relevant FAQ documents across multiple languages and assesses zero-shot generalization to other Wikipedia-based benchmarks.

## Datasets

- **WebFAQ** — total ?; splits: test (-1), train (-1)
- **Mr. TyDi** — total ?; splits: test (-1)
- **MIRACL (Hard Negatives)** — total ?; splits: test (-1)

## Metrics

- `NDCG@10` **(primary)** — range: percent
  - Normalized Discounted Cumulative Gain at rank 10, calculated as the sum of graded relevance scores discounted by log2(rank+1), normalized by the ideal DCG. Reported as a percentage.

## Input / output format

**Input**: Query string and a candidate document corpus (or document text) per language.

**Output**: Ranked list of top-10 documents or similarity scores for each query.

## Scoring recipe

```python
import math
def compute_ndcg_at_10(retrieved, relevant, k=10):
    dcg = sum(1.0 / math.log2(i + 2) for i, d in enumerate(retrieved[:k]) if d in relevant)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(relevant), k)))
    return (dcg / idcg) * 100.0 if idcg > 0 else 0.0
```

## Common pitfalls

- Evaluating on Mr. TyDi and MIRACL is strictly zero-shot; models fine-tuned on WebFAQ may show different relative gains compared to in-domain WebFAQ test splits.
- BM25 baselines use fixed Pyserini defaults (k1=0.9, b=0.4); changing these will break reproducibility with the reported table.
- Hybrid retrieval combines dense cosine similarity and BM25 scores using lambda=1.1; missing scores for documents in only one set are set to zero.

## Evidence (verbatim from paper)

> Table[3] outlines retrieval performances on six languages – the intersection set of languages covered by WebFAQ, Mr. TyDi and MIRACL. Table[3]. Comparing retrieval performance on 3 multilingual datasets using NDCG@10 in %, including SotA embedding models and BM25 as baselines.

## Citation

```bibtex
@misc{dinzinger2025webfaq,
  title={WebFAQ: A Multilingual Collection of Natural Q&A Datasets for Dense Retrieval},
  author={Dinzinger et al. (2025)},
  year={2025},
  note={arXiv:2502.20936}
}
```

- arXiv: 2502.20936