Trec Dl Eval

---
name: trec-dl-eval
description: Evaluates document retrieval models on a large-scale corpus by ranking millions of documents against a set of queries. It probes the model's ability to perform full-document retrieval and produce accurate ranked lists using explicit and latent matching signals. Use when the user wants to benchmark on TREC Deep Learning track (MS MARCO), or asks about evaluating this task. Reports MRR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2007.10434
  bibtex_key: mitra2020conformerkernel
  confidence: high
---

# trec-dl-eval

> Conformer-Kernel with Query Term Independence for Document Retrieval — Mitra et al. (2020) (arXiv:2007.10434, 2020)

## What this evaluates

Evaluates document retrieval models on a large-scale corpus by ranking millions of documents against a set of queries. It probes the model's ability to perform full-document retrieval and produce accurate ranked lists using explicit and latent matching signals.

## Datasets

- **TREC Deep Learning track (MS MARCO)** — total 3213835; splits: train (384597), test (-1)

## Metrics

- `MRR` **(primary)** — range: [0, 1]
  - Mean Reciprocal Rank: the average of the reciprocal ranks of the first relevant document across all queries. Calculated as 1/rank_of_first_relevant.
- `NDCG@10` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10: measures ranking quality by summing graded relevance scores discounted logarithmically by position, normalized by the ideal DCG@10.
- `NCG@100` — range: [0, 1]
  - Normalized Cumulative Gain at rank 100: measures ranking quality by summing graded relevance scores up to rank 100, normalized by the ideal cumulative gain.

## Input / output format

**Input**: Query (first 20 terms) and document text (first 4000 terms). Optional ORCAS click log field (max 2000 terms).

**Output**: Ranked list of documents from the full collection (3,213,835 documents) for each query.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    mrr, ndcg10, ncg100 = [], [], []
    for pred, rels in zip(predictions, gold):
        rr = 1.0 / (next(i for i, d in enumerate(pred) if d in rels) + 1)
        mrr.append(rr)
        dcg10 = sum(1.0 / (i + 1) for i, d in enumerate(pred[:10]) if d in rels)
        idcg10 = min(10, len(rels))
        dcg100 = sum(1.0 / (i + 1) for i, d in enumerate(pred[:100]) if d in rels)
        idcg100 = min(100, len(rels))
        ndcg10.append(dcg10 / idcg10 if idcg10 > 0 else 0)
        ncg100.append(dcg100 / idcg100 if idcg100 > 0 else 0)
    return {'MRR': sum(mrr)/len(mrr), 'NDCG@10': sum(ndcg10)/len(ndcg10), 'NCG@100': sum(ncg100)/len(ncg100)}
```

## Common pitfalls

- Evaluating on a small candidate set instead of the full 3.2M document collection as specified.
- Using truncated document lengths (4000 terms) which may miss relevant content in long documents.
- Confusing NCG with NDCG; NCG does not apply position discounting.

## Evidence (verbatim from paper)

> We compare different runs based on following three metrics: mean reciprocal rank (MRR) [Craswell, 2009], normalized discounted cumulative gain (NDCG) [Järvelin and Kekäläinen, 2002], and normalized cumulative gain (NCG) [Rosset et al., 2018].

## Citation

```bibtex
@misc{mitra2020conformerkernel,
  title={Conformer-Kernel with Query Term Independence for Document Retrieval},
  author={Mitra et al. (2020)},
  year={2020},
  note={arXiv:2007.10434}
}
```

- arXiv: 2007.10434