Trec Cast21 Eval

---
name: trec-cast21-eval
description: Evaluates conversational search retrieval pipelines by reproducing baseline and top-performing systems from the TREC CAsT 2021 track. It probes the effectiveness of query rewriting, sparse/dense retrieval fusion, and re-ranking strategies in multi-turn conversational settings. Use when the user wants to benchmark on TREC CAsT 2021, or asks about evaluating this task. Reports NDCG@3.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2301.10493
  bibtex_key: lajewska2023reproducibility
  confidence: high
---

# trec-cast21-eval

> From Baseline to Top Performer: A Reproducibility Study of Approaches at the TREC 2021 Conversational Assistance Track — Lajewska et al. (2023) (arXiv:2301.10493, 2023)

## What this evaluates

Evaluates conversational search retrieval pipelines by reproducing baseline and top-performing systems from the TREC CAsT 2021 track. It probes the effectiveness of query rewriting, sparse/dense retrieval fusion, and re-ranking strategies in multi-turn conversational settings.

## Datasets

- **TREC CAsT 2021** — total ?; splits: test (-1)

## Metrics

- `NDCG@3` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 3. Measures the quality of the top-3 ranked documents, discounting relevance by position and normalizing by the ideal ranking.
- `Recall@500` — range: [0, 1]
  - Fraction of relevant documents (binary relevance threshold ≥ 2) retrieved within the top 500 ranked passages.
- `MAP` — range: [0, 1]
  - Mean Average Precision across conversational turns, averaging precision at each relevant document's rank.
- `MRR` — range: [0, 1]
  - Mean Reciprocal Rank, averaging the inverse rank of the first relevant document per turn.
- `NDCG@500` — range: [0, 1]
  - Normalized Discounted Cumulative Gain computed with a rank cutoff of 500.

## Input / output format

**Input**: Multi-turn conversational queries with history (previous queries and canonical responses) for query rewriting; document passages from the TREC CAsT index for retrieval and re-ranking.

**Output**: Ranked list of document passages (initially top 1000 candidates, then final ranked list).

## Scoring recipe

```python
def compute_metrics(predictions, gold, cutoff=3, recall_cutoff=500, rel_thresh=2):
    import math
    rel_scores = [gold.get(d, 0) for d in predictions[:cutoff]]
    dcg = sum(s / math.log2(i + 2) for i, s in enumerate(rel_scores))
    ideal = sorted(rel_scores, reverse=True)
    idcg = sum(s / math.log2(i + 2) for i, s in enumerate(ideal))
    ndcg = dcg / idcg if idcg > 0 else 0.0
    rel_in_top = sum(1 for d in predictions[:recall_cutoff] if gold.get(d, 0) >= rel_thresh)
    total_rel = sum(1 for s in gold.values() if s >= rel_thresh)
    recall = rel_in_top / total_rel if total_rel > 0 else 0.0
    return {'NDCG@3': ndcg, 'Recall@500': recall}
```

## Common pitfalls

- Missing documentation on query rewriting input formatting and model length handling, leading to trimmed inputs or removed raw queries.
- Undisclosed QA system and PRF algorithm details for the WaterlooClarke baseline, causing significant performance drops during reproduction.
- BM25 parameter sensitivity to document preprocessing and index construction, making reported parameters suboptimal without exact preprocessing details.
- Discrepancies between officially reported track results and direct runfile evaluation, questioning the reliability of published baselines.

## Evidence (verbatim from paper)

> Following the official setup, we consider measures with both binary and graded relevance. The main measure is NDCG@3; other measures are computed with a rank cutoff of 500. For binary measures, we apply a relevance threshold of 2.

## Citation

```bibtex
@misc{lajewska2023reproducibility,
  title={From Baseline to Top Performer: A Reproducibility Study of Approaches at the TREC 2021 Conversational Assistance Track},
  author={Lajewska et al. (2023)},
  year={2023},
  note={arXiv:2301.10493}
}
```

- arXiv: 2301.10493