Dpr Retrieval Eval

---
name: dpr-retrieval-eval
description: Evaluates a model's ability to retrieve relevant passages from a large unstructured corpus for open-domain question answering. It probes semantic matching and dense retrieval capabilities by measuring how often the correct answer span appears in the top-k retrieved passages. Use when the user wants to benchmark on Natural Questions, TriviaQA, WebQuestions, CuratedTREC, SQuAD v1.1, or asks about evaluating this task. Reports top-k retrieval accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2004.04906
  bibtex_key: karpukhin2020dpr
  confidence: high
---

# dpr-retrieval-eval

> Dense Passage Retrieval for Open-Domain Question Answering — Karpukhin et al. (2020) (arXiv:2004.04906, 2020)

## What this evaluates

Evaluates a model's ability to retrieve relevant passages from a large unstructured corpus for open-domain question answering. It probes semantic matching and dense retrieval capabilities by measuring how often the correct answer span appears in the top-k retrieved passages.

## Datasets

- **Natural Questions** — total ?; splits: train (58880), dev (8757), test (3610)
- **TriviaQA** — total ?; splits: train (60413), dev (8837), test (11313)
- **WebQuestions** — total ?; splits: train (2474), dev (361), test (2032)
- **CuratedTREC** — total ?; splits: train (1125), dev (133), test (694)
- **SQuAD v1.1** — total ?; splits: train (70096), dev (8886), test (10570)

## Metrics

- `top-k retrieval accuracy` **(primary)** — range: percent
  - Percentage of top-k retrieved passages that contain the gold answer span. Calculated as the number of questions where the answer is found in the top-k results divided by the total number of questions.

## Input / output format

**Input**: A natural language question and a candidate pool of Wikipedia passages (100-word blocks, each prepended with the article title and a [SEP] token).

**Output**: A ranked list of candidate passages for the given question.

## Scoring recipe

```python
def compute_topk_accuracy(predictions, gold_answers, k=20):
    correct = 0
    for pred_passages, gold_span in zip(predictions, gold_answers):
        top_k = pred_passages[:k]
        if any(gold_span in passage for passage in top_k):
            correct += 1
    return correct / len(gold_answers)
```

## Common pitfalls

- Wikipedia version and preprocessing differences require filtering out questions where gold passages cannot be matched to the candidate pool.
- For TREC, WebQuestions, and TriviaQA, positive passages are selected using BM25 top-100 results containing the answer, introducing a retrieval bias.
- SQuAD is explicitly noted as suboptimal for open-domain QA because many questions lack sufficient context without the original provided paragraph.

## Evidence (verbatim from paper)

> Table 2: Top-20 \& Top-100 retrieval accuracy on test sets, measured as the percentage of top 20/100 retrieved passages that contain the answer.

## Citation

```bibtex
@misc{karpukhin2020dpr,
  title={Dense Passage Retrieval for Open-Domain Question Answering},
  author={Karpukhin et al. (2020)},
  year={2020},
  note={arXiv:2004.04906}
}
```

- arXiv: 2004.04906