Zero Shot Retrieval Leakage Eval

---
name: zero-shot-retrieval-leakage-eval
description: This evaluation probes the robustness of neural retrieval models to train-test data leakage by measuring how much performance on standard benchmarks artificially improves when training data contains near-duplicates or exact matches of test queries. It specifically assesses zero-shot transfer effectiveness under varying leakage conditions and training set sizes. Use when the user wants to benchmark on Robust04, TREC 2017 Common Core, TREC 2018 Common Core, or asks about evaluating this task. Reports nDCG@10.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2206.14759
  bibtex_key: frobe2022leakage
  confidence: high
---

# zero-shot-retrieval-leakage-eval

> How Train-Test Leakage Affects Zero-shot Retrieval — Fröbe et al. (2022) (arXiv:2206.14759, 2022)

## What this evaluates

This evaluation probes the robustness of neural retrieval models to train-test data leakage by measuring how much performance on standard benchmarks artificially improves when training data contains near-duplicates or exact matches of test queries. It specifically assesses zero-shot transfer effectiveness under varying leakage conditions and training set sizes.

## Datasets

- **Robust04** — total ?; splits: test (172)
- **TREC 2017 Common Core** — total ?; splits: test (37)
- **TREC 2018 Common Core** — total ?; splits: test (38)

## Metrics

- `nDCG@10` **(primary)** — range: [0, 1]
  - Normalized Discounted Cumulative Gain at rank 10. Computed over the top-100 re-ranked documents for each query, comparing the model's predicted relevance scores against binary ground truth judgments.
- `Precision@1` — range: [0, 1]
  - Binary metric that equals 1.0 if the top-ranked document is relevant, and 0.0 otherwise.
- `MFR` — range: other
  - Mean First Rank of a relevant document. Calculated as the average position of the first relevant document in the ranked list across all queries. Lower values indicate better performance, with 1.0 being optimal.

## Input / output format

**Input**: Training: query paired with one relevant and one non-relevant document. Inference: query with a candidate list of top-100 BM25-ranked documents.

**Output**: A re-ranked list of the top-100 documents for each query, ordered by the model's predicted relevance score. Ties are broken via alphanumeric ordering by document ID.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    # predictions: list of doc_ids ranked by model
    # gold: set of relevant doc_ids
    ndcg = calculate_ndcg_at_k(predictions, gold, k=10)
    prec1 = 1.0 if predictions[0] in gold else 0.0
    mfr = next(i+1 for i, doc in enumerate(predictions) if doc in gold)
    return ndcg, prec1, mfr
# Aggregate scores across queries, then compute mean across 5-fold CV splits.
# Significance tested via Student's t-test (p=0.05) with Bonferroni correction.
```

## Common pitfalls

- MFR is inverted (lower is better), which can lead to misinterpretation if treated like standard accuracy metrics.
- Tie-breaking relies on alphanumeric document ID ordering rather than random or semantic similarity, which can artificially skew rankings for documents with similar scores.
- High nDCG@10 scores may reflect memorization of leaked queries rather than genuine zero-shot generalization capability.

## Evidence (verbatim from paper)

> We report the effectiveness of the models as nDCG@10, Precision@1, and the mean first rank of a relevant document (MFR) [18]. While effectiveness scores measured via nDCG@10 and Precision@1 have the property that higher values are better (a score of 1 indicates “best” effectiveness), for MFR, lower scores are better—but still a score of 1 is the best case indicating that the document on rank 1 always is relevant.

## Citation

```bibtex
@misc{frobe2022leakage,
  title={How Train-Test Leakage Affects Zero-shot Retrieval},
  author={Fröbe et al. (2022)},
  year={2022},
  note={arXiv:2206.14759}
}
```

- arXiv: 2206.14759