Qder Re Ranking Eval

---
name: qder-re-ranking-eval
description: Evaluates the ability of neural re-ranking models to effectively re-order a candidate set of documents based on complex query semantics and entity relationships. It probes fine-grained semantic matching, entity-aware attention, and late aggregation capabilities in information retrieval tasks across news and complex answer domains. Use when the user wants to benchmark on CODEC, TREC Complex Answer Retrieval (CAR) 2017, TREC Robust 2004, TREC News 2021, TREC Core 2018, or asks about evaluating this task. Reports nDCG@20.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.11589
  bibtex_key: chatterjee2025qder
  confidence: high
---

# qder-re-ranking-eval

> QDER: Query-Specific Document and Entity Representations for Multi-Vector Document Re-Ranking — Chatterjee et al. (2025) (arXiv:2510.11589, 2025)

## What this evaluates

Evaluates the ability of neural re-ranking models to effectively re-order a candidate set of documents based on complex query semantics and entity relationships. It probes fine-grained semantic matching, entity-aware attention, and late aggregation capabilities in information retrieval tasks across news and complex answer domains.

## Datasets

- **CODEC** — total 729824; splits: test (-1)
- **TREC Complex Answer Retrieval (CAR) 2017** — total 4862; splits: train (-1)
- **TREC Robust 2004** — total 528024; splits: test (-1)
- **TREC News 2021** — total 728626; splits: test (-1)
- **TREC Core 2018** — total 595037; splits: test (-1)

## Metrics

- `nDCG@20` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 20. Calculated as DCG@20 divided by the ideal DCG@20, where DCG sums relevance scores discounted logarithmically by position: sum((2^rel - 1) / log2(pos + 2)).
- `Precision@20` — range: [0, 1]
  - Fraction of relevant documents in the top 20 ranked results.
- `MAP` — range: [0, 1]
  - Mean Average Precision across all queries, averaging the precision at each relevant document's rank.
- `MRR` — range: [0, 1]
  - Mean Reciprocal Rank, averaging the inverse rank of the first relevant document across queries.

## Input / output format

**Input**: Query (title, description, and/or narrative fields) paired with a candidate set of 1000 documents retrieved via BM25+RM3.

**Output**: A ranked list or relevance score for each of the 1000 candidate documents.

## Scoring recipe

```python
def compute_ndcg_at_20(gold_rels, pred_scores, k=20):
    ranked = sorted(range(len(pred_scores)), key=lambda i: pred_scores[i], reverse=True)[:k]
    dcg = sum((2**gold_rels[i] - 1) / (2**(j+1) - 1) for j, i in enumerate(ranked))
    ideal = sorted(gold_rels, reverse=True)[:k]
    idcg = sum((2**ideal[j] - 1) / (2**(j+1) - 1) for j in range(k))
    return dcg / idcg if idcg > 0 else 0.0
# Note: Paper uses official trec_eval tool with -c flag for all metrics.
```

## Common pitfalls

- The paper mixes re-ranking baselines (e.g., ColBERT, RoBERTa) with full-retrieval baselines (e.g., SPLADE, ANCE-MaxP) only on TREC Robust 2004, which can skew cross-dataset comparisons if not accounted for.
- Training negatives are sampled exclusively from the BM25+RM3 candidate set and balanced 1:1 with positives, differing from standard in-batch or hard-negative mining strategies.
- Significance testing relies on paired t-tests over queries, which may not fully capture variance in graded relevance judgments used by trec_eval.

## Evidence (verbatim from paper)

> Evaluation Metrics. (1) Precision at $k\=20$, (2) Normalized Discounted Cumulative Gain (nDCG) at $k\=20$, (3) Mean Average Precision (MAP), and (4) Mean Reciprocal Rank (MRR). We use the official trec_eval tool from NIST (with the -c flag) to evaluate each system.

## Citation

```bibtex
@misc{chatterjee2025qder,
  title={QDER: Query-Specific Document and Entity Representations for Multi-Vector Document Re-Ranking},
  author={Chatterjee et al. (2025)},
  year={2025},
  note={arXiv:2510.11589}
}
```

- arXiv: 2510.11589