Retrieval Rerank Rag Eval

---
name: retrieval-rerank-rag-eval
description: Evaluates the effectiveness of sparse and dense retrieval models, re-rankers, and retrieval-augmented generation (RAG) pipelines on open-domain QA, multi-hop QA, and fact verification tasks. Use when the user wants to benchmark on Natural Questions, TriviaQA, HotpotQA, 2WikiMultiHopQA, ArchivalQA, MSMARCO, WebQuestions, PopQA, ChroniclingAmericaQA, or asks about evaluating this task. Reports Top-k accuracy, Exact match (EM).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.02464
  bibtex_key: [REDACTED]
  confidence: high
---

# retrieval-rerank-rag-eval

> Rankify: A Comprehensive Python Toolkit for Retrieval, Re-Ranking, and Retrieval-Augmented Generation — Abdelrahman Abdallah et al. (2025) (arXiv:2502.02464, 2025)

## What this evaluates

Evaluates the effectiveness of sparse and dense retrieval models, re-rankers, and retrieval-augmented generation (RAG) pipelines on open-domain QA, multi-hop QA, and fact verification tasks.

## Datasets

- **Natural Questions** — total ?; splits: test (-1)
- **TriviaQA** — total ?; splits: test (-1)
- **HotpotQA** — total ?; splits: test (-1)
- **2WikiMultiHopQA** — total ?; splits: test (-1)
- **ArchivalQA** — total ?; splits: test (-1)
- **MSMARCO** — total ?; splits: test (-1)
- **WebQuestions** — total ?; splits: test (-1)
- **PopQA** — total ?; splits: test (-1)
- **ChroniclingAmericaQA** — total ?; splits: test (-1)

## Metrics

- `Top-k accuracy` **(primary)** — range: percent
  - Measures the proportion of queries where the ground-truth document appears in the top-k retrieved results. Evaluated at k=1, 5, 10, 20, 50, 100.
- `Exact match (EM)` **(primary)** — range: percent
  - Measures the proportion of generated answers that exactly match the ground-truth reference string.
- `Precision` — range: percent
  - Standard retrieval precision metric.
- `Recall` — range: percent
  - Standard retrieval recall metric.
- `Contains` — range: percent
  - Binary metric indicating whether the ground-truth answer string is contained within the generated answer.
- `F1` — range: percent
  - Harmonic mean of Precision and Recall.

## Input / output format

**Input**: A query and a corpus of candidate passages (e.g., 100-word non-overlapping Wikipedia passages).

**Output**: A ranked list of retrieved documents or a generated answer string for RAG.

## Scoring recipe

```python
def score_topk_accuracy(predictions, gold, k):
    correct = 0
    for pred, gold_set in zip(predictions, gold):
        if any(g in pred[:k] for g in gold_set):
            correct += 1
    return correct / len(predictions)

def score_exact_match(predictions, gold):
    correct = 0
    for p, g in zip(predictions, gold):
        if p.strip() == g.strip():
            correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- Datasets use different underlying corpora or splits, making direct comparison across studies difficult.
- Re-ranking experiments used top 100 retrieved documents, whereas original studies used 1,000, causing result differences.
- Some datasets (e.g., ChroniclingAmericaQA) do not report retrieval accuracy as a standard Top-k metric.

## Evidence (verbatim from paper)

> We evaluated retrieval performance on datasets such as Natural Questions (NQ), TriviaQA, HotpotQA, 2WikiMultiHopQA, and ArchivalQA, while re-ranking performance was analyzed on MSMARCO, WebQuestions, and PopQA. For retrieval evaluation, we measured Top-k accuracy at k=1, 5, 10, 20, 50, 100. Exact match (EM), Precision, Recall, Contains, and F1 were used as the primary evaluation metrics for QA tasks.

## Citation

```bibtex
@misc{abdallah2025rankify,
  title={Rankify: A Comprehensive Python Toolkit for Retrieval, Re-Ranking, and Retrieval-Augmented Generation},
  author={Abdelrahman Abdallah et al. (2025)},
  year={2025},
  note={arXiv:2502.02464}
}
```

- arXiv: 2502.02464