Convdr Eval

---
name: convdr-eval
description: Evaluates conversational dense retrieval models on their ability to rank relevant documents given multi-turn conversational queries. It probes context capture, few-shot learning effectiveness, and robustness to noisy conversation history compared to query rewriting baselines. Use when the user wants to benchmark on TREC CAsT, OR-QuAC, or asks about evaluating this task. Reports NDCG@3, MRR@5.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2105.04166
  bibtex_key: shi2021fewshotconversational
  confidence: high
---

# convdr-eval

> Few-Shot Conversational Dense Retrieval — Shi Yu et al. (2021) (arXiv:2105.04166, 2021)

## What this evaluates

Evaluates conversational dense retrieval models on their ability to rank relevant documents given multi-turn conversational queries. It probes context capture, few-shot learning effectiveness, and robustness to noisy conversation history compared to query rewriting baselines.

## Datasets

- **TREC CAsT** — total ?; splits: test (-1); repo https://github.com/thunlp/ConvDR
- **OR-QuAC** — total ?; splits: test (-1)

## Metrics

- `NDCG@3` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 3. Computes the weighted sum of relevance scores for the top 3 retrieved documents, normalized by the ideal DCG@3.
- `MRR@5` **(primary)** — range: [0, 1]
  - Mean Reciprocal Rank at rank 5. Takes the reciprocal of the rank of the first relevant document in the top 5 results, averaged over queries.
- `Win/Tie/Loss (%)` — range: percent
  - Pairwise comparison metric reporting the percentage of queries where the model wins, ties, or loses against a baseline based on ranking accuracy.

## Input / output format

**Input**: Multi-turn conversational query history concatenated with the current user query, paired with a candidate document.

**Output**: A ranked list of documents based on dot-product similarity scores, optionally reranked by a cross-encoder (e.g., BERT).

## Scoring recipe

```python
def compute_ndcg_at_k(relevance_scores, k=3):
    dcg = sum(r / (2**i - 1) for i, r in enumerate(relevance_scores[:k], 1))
    ideal = sorted(relevance_scores, reverse=True)[:k]
    idcg = sum(r / (2**i - 1) for i, r in enumerate(ideal, 1))
    return dcg / idcg if idcg > 0 else 0.0

def compute_mrr_at_k(relevance_scores, k=5):
    for i, r in enumerate(relevance_scores[:k], 1):
        if r > 0: return 1.0 / i
    return 0.0
```

## Common pitfalls

- Confusing first-stage dense retrieval scores with reranking scores; the paper evaluates both separately and notes performance drops in reranking.
- Overlooking the 'hole rate' on CAsT, which indicates missing relevant documents and suggests reported metrics may underestimate true performance.
- Assuming synthetic OR-QuAC labels reflect real user feedback; the paper warns that artifacts in synthetic data over-simplify retrieval.

## Evidence (verbatim from paper)

> Table 4: Results of different training paradigms. Candidates from ConvDR (KD) and ConvDR (Multi-Task) are reranked with BERT on CAsT-19 and OR-QuAC. Win/Tie/Loss (%) are compared with ANCE-Manual or ANCE-Manual→BERT-Manual. | Method | CAsT 2019 | | OR-QuAC | | | NDCG@3 | W/T/L | MRR@5 | W/T/L |

## Citation

```bibtex
@misc{shi2021fewshotconversational,
  title={Few-Shot Conversational Dense Retrieval},
  author={Shi Yu et al. (2021)},
  year={2021},
  note={arXiv:2105.04166}
}
```

- arXiv: 2105.04166