Trec Dl Track Eval

---
name: trec-dl-track-eval
description: Evaluates the reliability and best practices for using TREC Deep Learning test collections for ranking model evaluation. It probes whether researchers properly separate model selection from final evaluation and accounts for training variance. Use when the user wants to benchmark on TREC Deep Learning Track, or asks about evaluating this task. Reports NDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2104.09399
  bibtex_key: craswell2021trecdl
  confidence: high
---

# trec-dl-track-eval

> TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime — Craswell et al. (2021) (arXiv:2104.09399, 2021)

## What this evaluates

Evaluates the reliability and best practices for using TREC Deep Learning test collections for ranking model evaluation. It probes whether researchers properly separate model selection from final evaluation and accounts for training variance.

## Datasets

- **TREC Deep Learning Track** — total ?; splits: dev (-1), test (-1); repo https://github.com/bmitra-msft/TREC-Deep-Learning-Quick-Start

## Metrics

- `NDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. It computes the weighted sum of graded relevance scores up to position 10, normalized by the ideal DCG@10 for the same query to yield a score between 0 and 1.

## Input / output format

**Input**: Query, document collection, and pre-computed relevance judgments for each query-document pair.

**Output**: A ranked list of documents for each query.

## Scoring recipe

```python
def compute_ndcg_at_10(relevance_scores, k=10):
    dcg = sum((2**rel - 1) / log2(i + 2) for i, rel in enumerate(relevance_scores[:k]))
    ideal = sorted(relevance_scores, reverse=True)
    idcg = sum((2**rel - 1) / log2(i + 2) for i, rel in enumerate(ideal[:k]))
    return dcg / idcg if idcg > 0 else 0.0
```

## Common pitfalls

- Selecting the best model checkpoint or architecture based on test set performance instead of a held-out dev set.
- Reporting dev set metrics as final results without a separate held-out test set.
- Relying on a single training run with one random seed, ignoring variance that can lead to false positive 'significant' differences.

## Evidence (verbatim from paper)

> The valid numbers to report are the TREC 2019 and TREC 2020 NDCG@10 results for ranker A, which can be read by following the dotted line from A to the two other curves.

## Citation

```bibtex
@misc{craswell2021trecdl,
  title={TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime},
  author={Craswell et al. (2021)},
  year={2021},
  note={arXiv:2104.09399}
}
```

- arXiv: 2104.09399