Trec 2021 Dl Eval

---
name: trec-2021-dl-eval
description: Evaluates the ability of ranking models to retrieve and order relevant passages or documents for given queries. It probes multi-stage retrieval pipelines, including dense/sparse retrieval, query expansion, and re-ranking capabilities. Use when the user wants to benchmark on TREC 2021 Deep Learning Track, or asks about evaluating this task. Reports NDCG@5.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2205.11245
  bibtex_key: qiao2022pash
  confidence: high
---

# trec-2021-dl-eval

> PASH at TREC 2021 Deep Learning Track: Generative Enhanced Model for Multi-stage Ranking — Qiao et al. (2022) (arXiv:2205.11245, 2022)

## What this evaluates

Evaluates the ability of ranking models to retrieve and order relevant passages or documents for given queries. It probes multi-stage retrieval pipelines, including dense/sparse retrieval, query expansion, and re-ranking capabilities.

## Datasets

- **TREC 2021 Deep Learning Track** — total ?; splits: test (-1)

## Metrics

- `NDCG@5` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at cutoff 5. Calculated as DCG@5 / IDCG@5, where DCG@5 = sum_{i=1}^{5} (2^{rel_i} - 1) / log2(i+1) and IDCG@5 is the DCG of the ideal ranking.

## Input / output format

**Input**: Query text and a candidate set of passages or documents to be ranked.

**Output**: A ranked list of candidate passages or documents ordered by relevance score for each query.

## Scoring recipe

```python
def ndcg_at_k(ranked_rels, k=5):
    dcg = sum((2**r - 1) / math.log2(i + 2) for i, r in enumerate(ranked_rels[:k]))
    ideal_rels = sorted(ranked_rels, reverse=True)
    idcg = sum((2**r - 1) / math.log2(i + 2) for i, r in enumerate(ideal_rels[:k]))
    return dcg / idcg if idcg > 0 else 0.0
```

## Common pitfalls

- Cutoff depth must be strictly enforced; scores beyond position k are ignored.
- Ties in retrieval scores must be broken consistently (e.g., randomly or by ID) as they directly impact ranking metrics like NDCG.
- Passage vs. document tasks require different aggregation levels; mixing them invalidates the evaluation.

## Evidence (verbatim from paper)

> achieving zero-shot NDCG@5 of 0.3651 for T5-11B. Ensemble learning across random seeds improves stability, with full-ranking runs achieving NDCG@5 of 0.7596 in passage and 0.7516 in document tasks.

## Citation

```bibtex
@misc{qiao2022pash,
  title={PASH at TREC 2021 Deep Learning Track: Generative Enhanced Model for Multi-stage Ranking},
  author={Qiao et al. (2022)},
  year={2022},
  note={arXiv:2205.11245}
}
```

- arXiv: 2205.11245