Rank Distillm Eval

---
name: rank-distillm-eval
description: Probes the re-ranking capability of distilled cross-encoder models on passage retrieval tasks. It evaluates ranking quality on in-domain benchmarks (TREC Deep Learning tracks) and out-of-domain generalization across diverse corpora (TIREx framework), while also measuring computational efficiency. Use when the user wants to benchmark on Rank-DistiLLM, TREC DL 2019, TREC DL 2020, TIREx, or asks about evaluating this task. Reports nDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.07920
  bibtex_key: schlatt2024rankdistillm
  confidence: high
---

# rank-distillm-eval

> Rank-DistiLLM: Closing the Effectiveness Gap Between Cross-Encoders and LLMs for Passage Re-Ranking — Schlatt et al. (2024) (arXiv:2405.07920, 2024)

## What this evaluates

Probes the re-ranking capability of distilled cross-encoder models on passage retrieval tasks. It evaluates ranking quality on in-domain benchmarks (TREC Deep Learning tracks) and out-of-domain generalization across diverse corpora (TIREx framework), while also measuring computational efficiency.

## Datasets

- **Rank-DistiLLM** — total ?; splits: train (-1), val (-1); repo https://github.com/webis-de/ECIR-25
- **TREC DL 2019** — total ?; splits: test (-1)
- **TREC DL 2020** — total ?; splits: test (-1)
- **TIREx** — total ?; splits: test (-1)

## Metrics

- `nDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. It measures the quality of the top-10 ranked results against ground-truth relevance judgments by discounting gains logarithmically based on position.

## Input / output format

**Input**: Query (truncated to ≤32 tokens) paired with a candidate list of up to 100 retrieved passages (each truncated to ≤256 tokens).

**Output**: A ranked list or relevance scores for the candidate passages, ordered by predicted relevance to the query.

## Scoring recipe

```python
def compute_ndcg_at_10(pred_scores, gold_labels, k=10):
    import math
    ranked_indices = sorted(range(len(pred_scores)), key=lambda i: pred_scores[i], reverse=True)
    dcg = sum((2**gold_labels[ranked_indices[i]] - 1) / math.log2(i + 2) for i in range(k))
    ideal_labels = sorted(gold_labels, reverse=True)
    idcg = sum((2**ideal_labels[i] - 1) / math.log2(i + 2) for i in range(k))
    return dcg / idcg if idcg > 0 else 0.0
```

## Common pitfalls

- Models are evaluated on top-100 passages retrieved by specific baselines (BM25 or ColBERTv2), so results are not retrieval-agnostic and depend heavily on the initial retriever.
- Statistical significance is reported using Holm-Bonferroni-corrected t-tests against the best monoELECTRA baseline, not pairwise across all models.
- Efficiency metrics (latency/memory) are hardware-specific (NVIDIA A100 40GB) and measured on a fixed batch/window size, limiting direct cross-system comparisons.

## Evidence (verbatim from paper)

> Table[1] lists nDCG@10 scores of monoELECTRA, a cross-encoder using ELECTRA*[[10]]* as the backbone encoder, fine-tuned on the data mentioned in Section[3.2], and evaluated on the TREC DL 2019 and 2020 tasks when re-ranking the top 100 passages retrieved by BM25 and ColBERTv2.

## Citation

```bibtex
@misc{schlatt2024rankdistillm,
  title={Rank-DistiLLM: Closing the Effectiveness Gap Between Cross-Encoders and LLMs for Passage Re-Ranking},
  author={Schlatt et al. (2024)},
  year={2024},
  note={arXiv:2405.07920}
}
```

- arXiv: 2405.07920