Nci Document Retrieval Eval

---
name: nci-document-retrieval-eval
description: Evaluates a model's ability to retrieve relevant documents from a large corpus given a natural language query. It measures ranking quality and recall at various cutoffs to assess end-to-end document retrieval performance. Use when the user wants to benchmark on NQ320k, TriviaQA, or asks about evaluating this task. Reports Recall@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2206.02743
  bibtex_key: wang2022neuralcorpusindexer
  confidence: high
---

# nci-document-retrieval-eval

> A Neural Corpus Indexer for Document Retrieval — Wang et al. (2022) (arXiv:2206.02743, 2022)

## What this evaluates

Evaluates a model's ability to retrieve relevant documents from a large corpus given a natural language query. It measures ranking quality and recall at various cutoffs to assess end-to-end document retrieval performance.

## Datasets

- **NQ320k** — total 320000; splits: train (-1), val (-1)
- **TriviaQA** — total 78000; splits: train (-1), val (-1)

## Metrics

- `Recall@1` **(primary)** — range: [0, 1]
  - Measures how often the desired document is hit by the top-1 retrieved candidate.
- `Recall@10` — range: [0, 1]
  - Measures how often the desired document is hit by the top-10 retrieved candidates.
- `Recall@100` — range: [0, 1]
  - Measures how often the desired document is hit by the top-100 retrieved candidates.
- `MRR@100` — range: [0, 1]
  - Calculates the reciprocal of the rank at which the first relevant document is retrieved, computed over the top-100 results.
- `R-Precision` — range: [0, 1]
  - Precision after R documents have been retrieved, where R is the number of relevant documents for the query.

## Input / output format

**Input**: Natural language query.

**Output**: A ranked list of document identifiers (or documents) retrieved for the query.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    recalls, mrrs, r_precs = [], [], []
    for pred, rels in zip(predictions, gold):
        top100 = pred[:100]
        for N in [1, 10, 100]:
            recalls.append(len(set(pred[:N]) & set(rels)) / len(rels))
        mrr = 0.0
        for i, doc in enumerate(top100):
            if doc in rels:
                mrr = 1.0 / (i + 1)
                break
        mrrs.append(mrr)
        R = len(rels)
        topR = pred[:R]
        r_precs.append(len(set(topR) & set(rels)) / R)
    return {
        'Recall@1': sum(recalls[::3]) / len(recalls[::3]),
        'Recall@10': sum(recalls[1::3]) / len(recalls[1::3]),
        'Recall@100': sum(recalls[2::3]) / len(recalls[2::3]),
        'MRR@100': sum(mrrs) / len(mrrs),
        'R-Precision': sum(r_precs) / len(r_precs)
    }
```

## Common pitfalls

- Recall is evaluated at different cutoffs (@1, @10, @100) depending on the dataset; ensure the correct cutoff is used for each benchmark.
- R-Precision uses R = number of relevant documents per query, which can be >1 in TriviaQA, unlike standard single-relevance benchmarks.
- MRR is computed over the top-100 results, not the full generated list.

## Evidence (verbatim from paper)

> Recall@N measures how often the desired document is hit by the top-N retrieved candidates. MRR calculates the reciprocal of the rank at which the first relevant document is retrieved. R-Precision is the precision after R documents have been retrieved, where R is the number of relevant documents for the query.

## Citation

```bibtex
@misc{wang2022neuralcorpusindexer,
  title={A Neural Corpus Indexer for Document Retrieval},
  author={Wang et al. (2022)},
  year={2022},
  note={arXiv:2206.02743}
}
```

- arXiv: 2206.02743