Convsearch R1 Eval

---
name: convsearch-r1-eval
description: Evaluates conversational query reformulation (CQR) by measuring how effectively a model rewrites multi-turn queries into standalone search queries that retrieve relevant passages. It probes the model's ability to optimize rewrites using only retrieval signals, without human annotations or LLM distillation. Use when the user wants to benchmark on TopiOCQA, QReCC, or asks about evaluating this task. Reports MRR@3.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.15776
  bibtex_key: [REDACTED]
  confidence: high
---

# convsearch-r1-eval

> ConvSearch-R1: Enhancing Query Reformulation for Conversational Search with Reasoning via Reinforcement Learning — Zhu et al. (2025) (arXiv:2505.15776, 2025)

## What this evaluates

Evaluates conversational query reformulation (CQR) by measuring how effectively a model rewrites multi-turn queries into standalone search queries that retrieve relevant passages. It probes the model's ability to optimize rewrites using only retrieval signals, without human annotations or LLM distillation.

## Datasets

- **TopiOCQA** — total ?; splits: test (-1)
- **QReCC** — total ?; splits: test (-1)

## Metrics

- `MRR@3` **(primary)** — range: [0, 1]
  - Mean Reciprocal Rank at cutoff 3. For each query, compute 1/rank of the first relevant document in the top 3 results, then average across all queries.
- `NDCG@3` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at cutoff 3. Measures the quality of the ranked list by discounting relevance scores logarithmically by position and normalizing by the ideal DCG.
- `Recall@K` — range: [0, 1]
  - Fraction of relevant documents retrieved within the top K results. Computed as |relevant_retrieved| / |total_relevant|.

## Input / output format

**Input**: Conversational dialogue history (previous turns) and the current user query.

**Output**: A single rewritten query string optimized for retrieval.

## Scoring recipe

```python
def compute_metrics(ranked_docs, relevant_docs, k=3):
    # ranked_docs: list of doc IDs in retrieval order
    # relevant_docs: set of ground truth relevant doc IDs
    
    # MRR@3
    rr = 0.0
    for i, doc in enumerate(ranked_docs[:k]):
        if doc in relevant_docs:
            rr = 1.0 / (i + 1)
            break
    mrr = rr
    
    # NDCG@3
    dcg = sum(rel / math.log2(i + 2) for i, rel in enumerate([1 if d in relevant_docs else 0 for d in ranked_docs[:k]]))
    idcg = sum(1 / math.log2(i + 2) for i in range(min(len(relevant_docs), k)))
    ndcg = dcg / idcg if idcg > 0 else 0.0
    
    # Recall@K
    recall = len(set(ranked_docs[:k]) & relevant_docs) / len(relevant_docs) if relevant_docs else 0.0
    
    return mrr, ndcg, recall
```

## Common pitfalls

- The paper evaluates using both BM25 and ANCE retrievers without retraining them, so results are highly sensitive to the pre-existing retrieval index quality and passage granularity.
- Recall@K requires a fixed K value (commonly 20 or 100 in conversational search benchmarks), which is not explicitly specified in the setup section and must be chosen consistently.
- Rewritten queries must be evaluated as standalone queries; failing to strip conversational context or pronouns before retrieval will artificially lower scores.

## Evidence (verbatim from paper)

> For metrics, we adopt MRR@3, NDCG@3, and Recall@K (referred to as R@K in this paper) for evaluation.

## Citation

```bibtex
@misc{zhu2025convsearchr1,
  title={ConvSearch-R1: Enhancing Query Reformulation for Conversational Search with Reasoning via Reinforcement Learning},
  author={Zhu et al. (2025)},
  year={2025},
  note={arXiv:2505.15776}
}
```

- arXiv: 2505.15776