Trec 2019 Dl Track Eval

---
name: trec-2019-dl-track-eval
description: Evaluates ad-hoc information retrieval systems on document and passage ranking tasks using large-scale human-labeled judgments. It probes the ability of neural and traditional models to rank relevant items highly for a set of test queries. Use when the user wants to benchmark on TREC 2019 Deep Learning Track, or asks about evaluating this task. Reports NDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2003.07820
  bibtex_key: craswell2020overview
  confidence: high
---

# trec-2019-dl-track-eval

> Overview of the TREC 2019 deep learning track — Craswell et al. (2020) (arXiv:2003.07820, 2020)

## What this evaluates

Evaluates ad-hoc information retrieval systems on document and passage ranking tasks using large-scale human-labeled judgments. It probes the ability of neural and traditional models to rank relevant items highly for a set of test queries.

## Datasets

- **TREC 2019 Deep Learning Track** — total ?; splits: test (43)

## Metrics

- `NDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. Computed over 4-level graded relevance judgments, discounting gains by log2(rank+1) and normalizing by the ideal DCG.
- `NCG@100` — range: [0, 1]
  - Normalized Cumulative Gain at rank 100. Measures the cumulative gain of retrieved items discounted by rank, normalized by the ideal gain.
- `AP` — range: [0, 1]
  - Average Precision. Computes the mean of precision values at ranks where relevant documents appear.
- `RR` — range: [0, 1]
  - Reciprocal Rank. The inverse of the rank of the first relevant document in the list.

## Input / output format

**Input**: Query string and a list of candidate documents or passages (top-k for reranking, or full candidate set for full-ranking).

**Output**: A ranked list of document or passage IDs corresponding to the input candidates.

## Scoring recipe

```python
def compute_ndcg_at_k(gold_relevance, pred_rank_order, k=10):
    ranked_golds = [gold_relevance[i] for i in pred_rank_order[:k]]
    dcg = sum(r / log2(i + 2) for i, r in enumerate(ranked_golds))
    ideal_golds = sorted(gold_relevance, reverse=True)[:k]
    idcg = sum(r / log2(i + 2) for i, r in enumerate(ideal_golds))
    return dcg / idcg if idcg > 0 else 0.0
```

## Common pitfalls

- Confusing NIST labels (used for official track metrics) with MS MARCO labels (used for the RR (MS) column in results).
- Failing to distinguish between the 'rerank' subtask (re-ranking provided top-k candidates) and 'fullrank' subtask (generating candidates from scratch).
- Using binary relevance instead of the track's 4-level graded relevance judgments for NDCG computation.

## Evidence (verbatim from paper)

> Our main metric in both tasks is Normalized Discounted Cumulative Gain (NDCG)—specifically, NDCG@10, since it makes use of our 4-level judgments and focuses on the first results that users will see.

## Citation

```bibtex
@misc{craswell2020overview,
  title={Overview of the TREC 2019 deep learning track},
  author={Craswell et al. (2020)},
  year={2020},
  note={arXiv:2003.07820}
}
```

- arXiv: 2003.07820