Dureader Retrieval Eval

---
name: dureader-retrieval-eval
description: Passage retrieval for web search queries, evaluating a model's ability to rank relevant documents from a large collection. It probes in-domain retrieval accuracy as well as out-of-domain and cross-lingual generalization, highlighting challenges like salient phrase mismatch, syntactic mismatch, and false negatives. Use when the user wants to benchmark on DuReader_retrieval, or asks about evaluating this task. Reports MRR@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2203.10232
  bibtex_key: qiu2022dureader_retrieval
  confidence: high
---

# dureader-retrieval-eval

> DuReader_retrieval: A Large-scale Chinese Benchmark for Passage Retrieval from Web Search Engine — Qiu et al. (2022) (arXiv:2203.10232, 2022)

## What this evaluates

Passage retrieval for web search queries, evaluating a model's ability to rank relevant documents from a large collection. It probes in-domain retrieval accuracy as well as out-of-domain and cross-lingual generalization, highlighting challenges like salient phrase mismatch, syntactic mismatch, and false negatives.

## Datasets

- **DuReader_retrieval** — total ?; splits: train (-1), dev (-1), test (-1); repo https://github.com/baidu/DuReader

## Metrics

- `MRR@10` **(primary)** — range: [0, 1]
  - Mean Reciprocal Rank for the top 10 retrieved documents. Calculated as the average of 1/rank for the first relevant passage in the top-10 results across all queries.
- `Recall@1` — range: [0, 1]
  - Recall for the top-1 retrieved item. The fraction of queries where the single most relevant passage is ranked first.
- `Recall@50` — range: [0, 1]
  - Recall for the top-50 retrieved items. The fraction of queries where at least one relevant passage appears in the top-50 results.

## Input / output format

**Input**: A query (Chinese or English) and a candidate passage (for re-ranking) or a passage collection (for retrieval).

**Output**: A ranked list of passages corresponding to the query.

## Scoring recipe

```python
def compute_metrics(predictions, gold_indices):
    mrr_scores = []
    recall_1_scores = []
    recall_50_scores = []
    for pred, gold in zip(predictions, gold_indices):
        if gold in pred[:10]:
            rank = pred.index(gold) + 1
            mrr_scores.append(1.0 / rank)
        else:
            mrr_scores.append(0.0)
        recall_1_scores.append(1.0 if pred[0] == gold else 0.0)
        recall_50_scores.append(1.0 if gold in pred[:50] else 0.0)
    return {
        'MRR@10': sum(mrr_scores) / len(mrr_scores),
        'Recall@1': sum(recall_1_scores) / len(recall_1_scores),
        'Recall@50': sum(recall_50_scores) / len(recall_50_scores)
    }
```

## Common pitfalls

- The dataset contains false negatives (relevant passages incorrectly labeled as irrelevant), which can artificially suppress model scores if not accounted for.
- Dense retrievers trained on this dataset show poor zero-shot cross-domain generalization, often performing worse than BM25 without fine-tuning on the target domain.
- Cross-lingual retrieval (English query to Chinese passage) is significantly harder than monolingual retrieval due to lack of shared lexical terms, requiring semantic matching rather than simple term overlap.

## Evidence (verbatim from paper)

> We use the following evaluation metrics in our experiments: (1) Mean Reciprocal Rank for the top 10 retrieved documents (MRR@10), (2) Recall for the top-1 retrieved items (Recall@1) and (3) Recall for the top-50 retrieved items (Recall@50). Recall@50 is more suitable for evaluating the first-stage retrievers, while MRR@10 and Recall@1 are more suitable for assessing the second-stage re-rankers.

## Citation

```bibtex
@misc{qiu2022dureader_retrieval,
  title={DuReader_retrieval: A Large-scale Chinese Benchmark for Passage Retrieval from Web Search Engine},
  author={Qiu et al. (2022)},
  year={2022},
  note={arXiv:2203.10232}
}
```

- arXiv: 2203.10232