Coverage At Top K

---
name: coverage_at_top_k
description: Measures the alignment between a proxy benchmark's ranking and a target reward modeling benchmark's ranking at the top-k positions. It quantifies how many of the highest-performing models on a reward benchmark are also identified as top performers on a given proxy benchmark. Use when the user has predictions and gold and needs to compute coverage_at_top_k.
metadata:
  skill_kind: metric
  source_arxiv: 2505.10775
  bibtex_key: [REDACTED]
  confidence: high
---

# coverage_at_top_k

> A Systematic Analysis of Base Model Choice for Reward Modeling — Ahrabian et al. (2025) (arXiv:2505.10775, 2025)

## What this evaluates

Measures the alignment between a proxy benchmark's ranking and a target reward modeling benchmark's ranking at the top-k positions. It quantifies how many of the highest-performing models on a reward benchmark are also identified as top performers on a given proxy benchmark.

## Datasets

- **Curated Benchmark Suite (33 benchmarks)** — total ?; splits: test (-1)

## Metrics

- `coverage_at_top_k` **(primary)** — range: [0, 1]
  - C(β,ρ,L,k) = |T_β(L,k) ∩ T_ρ(L,k)| / k, where T_x(y,z) returns the top z models in y on benchmark x. It calculates the fraction of overlapping top-k models between a proxy benchmark β and a reward benchmark ρ.
- `pearson_correlation` — range: [-1, 1]
  - Standard Pearson correlation coefficient measuring linear relationship between benchmark scores and reward modeling performances.
- `spearman_correlation` — range: [-1, 1]
  - Standard Spearman rank correlation coefficient measuring monotonic relationship between benchmark scores and reward modeling performances.

## Input / output format

**Input**: A set of LLMs L, their scores on a proxy benchmark β, and their scores on a target reward benchmark ρ.

**Output**: A coverage value between 0 and 1 for a given k, or correlation coefficients.

## Scoring recipe

```python
def compute_coverage(proxy_scores, reward_scores, k):
    top_k_proxy = set(m for m, _ in sorted(proxy_scores, key=lambda x: x[1], reverse=True)[:k])
    top_k_reward = set(m for m, _ in sorted(reward_scores, key=lambda x: x[1], reverse=True)[:k])
    return len(top_k_proxy & top_k_reward) / k
```

## Common pitfalls

- Coverage is highly sensitive to k; the paper notes low coverage at higher ranks, making it less effective for broad model selection.
- High correlation does not guarantee high coverage; benchmarks can be strongly correlated with reward scores yet fail to identify top-performing models.

## Evidence (verbatim from paper)

> Given a benchmark β and reward bench ρ, we formally define the coverage at top-k as C(β,ρ,L,k)=|T_β(L,k)∩T_ρ(L,k)|/k where T_x(y,z) is the top z LLMs in y on benchmark x. To simulate a real-world search where we need high coverage at higher ranks, we filter out any benchmark with less than 0.4 and 0.7 coverage at k=5 and k=10, respectively.

## Citation

```bibtex
@misc{ahrabian2025systematic,
  title={A Systematic Analysis of Base Model Choice for Reward Modeling},
  author={Ahrabian et al. (2025)},
  year={2025},
  note={arXiv:2505.10775}
}
```

- arXiv: 2505.10775