Trec Cast 2019 Eval

---
name: trec-cast-2019-eval
description: Evaluates a system's ability to perform multi-turn conversational information retrieval by selecting relevant passages for user utterances while leveraging prior dialogue history, including handling coreference, omissions, and topic shifts. Use when the user wants to benchmark on TREC CAsT 2019, or asks about evaluating this task. Reports NDCG@3.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2003.13624
  bibtex_key: dalton2020trecast
  confidence: high
---

# trec-cast-2019-eval

> TREC CAsT 2019: The Conversational Assistance Track Overview — Dalton et al. (2020) (TREC 2019, 2020)

## What this evaluates

Evaluates a system's ability to perform multi-turn conversational information retrieval by selecting relevant passages for user utterances while leveraging prior dialogue history, including handling coreference, omissions, and topic shifts.

## Datasets

- **TREC CAsT 2019** — total ?; splits: test (-1); repo https://github.com/microsoft/MSMARCO-Conversational-Search

## Metrics

- `NDCG@3` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 3. Computes the discounted gain of graded relevance scores at the top 3 positions, normalized by the ideal DCG@3 for the given relevance judgments.
- `MAP` — range: [0, 1]
  - Mean Average Precision. Averages the precision at each relevant document across all queries.
- `MRR` — range: [0, 1]
  - Mean Reciprocal Rank. Averages the reciprocal of the rank of the first relevant document across all queries.

## Input / output format

**Input**: Multi-turn dialogue history (previous user utterances and system responses) plus the current user utterance (query).

**Output**: Ranked list of candidate text passages retrieved for the current utterance.

## Scoring recipe

```python
def compute_ndcg_at_3(predictions, gold):
    dcg = 0.0
    for i, doc_id in enumerate(predictions[:3]):
        rel = gold.get(doc_id, 0)
        dcg += (2**rel - 1) / math.log2(i + 2)
    ideal_rels = sorted(gold.values(), reverse=True)[:3]
    idcg = sum((2**r - 1) / math.log2(i + 2) for i, r in enumerate(ideal_rels))
    return dcg / idcg if idcg > 0 else 0.0
```

## Common pitfalls

- Automatic runs use raw, unresolved queries with coreference/omissions, while manual runs use rewritten resolved queries; results differ significantly (~26% median gap).
- Pool incompleteness: only top 2 runs per group are pooled for assessment, leaving ~0.6-1.67 unjudged documents in top 10 per turn.
- Washington Post (WaPo) passages make up ~10% of pool candidates; filtering them affects some systems differently.

## Evidence (verbatim from paper)

> We use three standard TREC evaluation measures, Mean-average Precision (MAP), and Normalized Discounted Cumulative Gain (NDCG), and Mean Reciprocal Rank (MRR). In particular, we use NDCG@3 as the primary measure because we focus on graded relevance of results at the top ranks.

## Citation

```bibtex
@misc{dalton2020trecast,
  title={TREC CAsT 2019: The Conversational Assistance Track Overview},
  author={Dalton et al. (2020)},
  year={2020},
  note={TREC 2019}
}
```

- arXiv: 2003.13624