Ir Metric Correlation Eval

---
name: ir-metric-correlation-eval
description: Evaluates the correlation and predictability of standard information retrieval metrics across multiple TREC test collections. It probes how well low-cost metrics can predict high-cost ones and how metric values vary between topic-wise and system-wise aggregations. Use when the user wants to benchmark on TREC Web & Robust Tracks (2000-2014), or asks about evaluating this task. Reports MAP.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1802.00323
  bibtex_key: kutlu2018correlation
  confidence: high
---

# ir-metric-correlation-eval

> Correlation and Prediction of Evaluation Metrics in Information Retrieval — Kutlu et al. (2018) (arXiv:1802.00323, 2018)

## What this evaluates

Evaluates the correlation and predictability of standard information retrieval metrics across multiple TREC test collections. It probes how well low-cost metrics can predict high-cost ones and how metric values vary between topic-wise and system-wise aggregations.

## Datasets

- **TREC Web & Robust Tracks (2000-2014)** — total ?; splits: test (-1); repo https://github.com/trec-web/trec-web-2014

## Metrics

- `MAP` **(primary)** — range: [0, 1]
  - Mean Average Precision: average of precision values at ranks where relevant documents occur, averaged over topics.
- `nDCG` — range: [0, 1]
  - Normalized Discounted Cumulative Gain: compares the actual ranked list to an ideal one using logarithmic discounting at each rank.
- `RBP@0.95` — range: [0, 1]
  - Rank-Biased Precision: models user browsing behavior with a persistence parameter p=0.95, weighting earlier ranks higher.
- `ERR@20` — range: [0, 1]
  - Expected Reciprocal Rank at cutoff 20: estimates the probability that a user finds a relevant document at each position, discounted by position.
- `P@10` — range: [0, 1]
  - Precision at rank 10: fraction of retrieved documents in the top 10 that are relevant.
- `Recall@1000` — range: [0, 1]
  - Recall at cutoff 1000: fraction of all relevant documents retrieved within the top 1000 results.
- `bpref` — range: [0, 1]
  - Binary preference: proportion of pairs of relevant and non-relevant documents where the relevant one is ranked higher.
- `RR` — range: [0, 1]
  - Reciprocal Rank: inverse of the rank of the first relevant document.
- `R-Prec` — range: [0, 1]
  - Precision at the rank equal to the total number of relevant documents for the query.

## Input / output format

**Input**: Ranked list of retrieved document IDs per query/topic, paired with ground-truth relevance judgments.

**Output**: Numerical score for each evaluation metric (e.g., MAP, nDCG, RBP) computed per system per topic or averaged across topics.

## Scoring recipe

```python
def compute_map(predictions, gold):
    hits = 0
    sum_prec = 0.0
    for i, doc in enumerate(predictions, 1):
        if doc in gold:
            hits += 1
            sum_prec += hits / i
    return sum_prec / len(gold) if gold else 0.0
# For other metrics (nDCG, RBP, ERR, etc.), use official trec_eval 9.0 or author-provided implementations with specified cutoffs and p values.
```

## Common pitfalls

- Cut-off thresholds are metric-specific (e.g., ERR uses @20, while others default to @1000).
- RBP requires selecting a specific user behavior parameter p (0.5, 0.8, or 0.95).
- Runs with missing relevance judgments or identical ranked lists across collections must be filtered out to avoid bias.

## Evidence (verbatim from paper)

> Using the system runs submitted to these selected TREC tracks and their respective relevance judgments, we calculated 9 different evaluation metrics, including AP, bpref [24], ERR [25], nDCG, P@K, RBP [2], recall (R), RR [26], and R-Prec. We used various cut-off thresholds for the metrics... In our calculations, we test 0.5, 0.8 and 0.95 for the p parameter... We also detected runs that have identical ranked lists in WT2013 and WT2014 test collections and filtered out identical submissions.

## Citation

```bibtex
@misc{kutlu2018correlation,
  title={Correlation and Prediction of Evaluation Metrics in Information Retrieval},
  author={Kutlu et al. (2018)},
  year={2018},
  note={arXiv:1802.00323}
}
```

- arXiv: 1802.00323