Ms Marco V2 Ranking Eval

---
name: ms-marco-v2-ranking-eval
description: Evaluates passage and document ranking systems on a large-scale, document-native corpus. It probes a model's ability to retrieve relevant content from millions of documents using sparse, crowd-sourced relevance judgments, while handling realistic corpus drift and query-independent passage extraction. Use when the user wants to benchmark on MS MARCO v2, or asks about evaluating this task. Reports NDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.08890
  bibtex_key: craswell2025trec2023dl
  confidence: medium
---

# ms-marco-v2-ranking-eval

> Overview of the TREC 2023 deep learning track — Craswell et al. (2025) (arXiv:2507.08890, 2025)

## What this evaluates

Evaluates passage and document ranking systems on a large-scale, document-native corpus. It probes a model's ability to retrieve relevant content from millions of documents using sparse, crowd-sourced relevance judgments, while handling realistic corpus drift and query-independent passage extraction.

## Datasets

- **MS MARCO v2** — total 11900000; splits: train (-1), dev (-1), eval (-1)

## Metrics

- `NDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. It measures the quality of a ranked list by comparing the discounted cumulative gain of the predicted ranking against the ideal ranking derived from relevance judgments.

## Input / output format

**Input**: A natural language query and a candidate set of passages or documents (up to 1000 for reranking, or full corpus for full ranking).

**Output**: A ranked list of passages or documents ordered by predicted relevance to the query.

## Scoring recipe

```python
def compute_ndcg_at_10(pred_ranking, gold_relevance, k=10):
    dcg = sum((2**rel - 1) / math.log2(i + 2) for i, rel in enumerate(pred_ranking[:k]))
    ideal = sorted(gold_relevance, reverse=True)[:k]
    idcg = sum((2**rel - 1) / math.log2(i + 2) for i, rel in enumerate(ideal))
    return dcg / idcg if idcg > 0 else 0.0
```

## Common pitfalls

- Relevance judgments (qrels) are highly sparse and derived from crowd workers selecting from only 10 initial results, not exhaustive ground truth.
- Passage-to-document mapping assumes relevance transfers if passages share a URL and sufficient text similarity, which may introduce noise.
- Corpus content drift: documents extracted later may have changed or disappeared, requiring systems to handle stale or missing content.

## Evidence (verbatim from paper)

> We can consider how big this problem is by analyzing the disagreement rate between MS MARCO qrels and NIST qrels (in v1 and v2), and seeing whether training on MS MARCO qrels yields improved NIST NDCG on the test set.

## Citation

```bibtex
@misc{craswell2025trec2023dl,
  title={Overview of the TREC 2023 deep learning track},
  author={Craswell et al. (2025)},
  year={2025},
  note={arXiv:2507.08890}
}
```

- arXiv: 2507.08890