Master Set Eval

---
name: master-set-eval
description: Evaluates a model's ability to recommend functionally indispensable (must-cite) papers for a given query paper based only on its title and abstract. It probes scientific retrieval capability by measuring how well systems rank baseline, core-relevant, or frequently mentioned papers from a large candidate pool. Use when the user wants to benchmark on MasterSet-CoreML-v1, or asks about evaluating this task. Reports Recall@K.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.17680
  bibtex_key: ratul2026masterset
  confidence: high
---

# master-set-eval

> MasterSet: A Large-Scale Benchmark for Must-Cite Citation Recommendation in the AI/ML Literature — Ratul et al. (2026) (arXiv:2604.17680, 2026)

## What this evaluates

Evaluates a model's ability to recommend functionally indispensable (must-cite) papers for a given query paper based only on its title and abstract. It probes scientific retrieval capability by measuring how well systems rank baseline, core-relevant, or frequently mentioned papers from a large candidate pool.

## Datasets

- **MasterSet-CoreML-v1** — total 74789; splits: train (67761), test (7028)

## Metrics

- `Recall@K` **(primary)** — range: [0, 1]
  - Fraction of relevant (must-cite) papers retrieved within the top K results. Calculated as |R ∩ R̂_K| / |R|, where R is the set of ground-truth must-cite papers and R̂_K is the top-K predicted list.
- `MAP` — range: [0, 1]
  - Mean Average Precision across queries. Averages the precision at each relevant position in the ranked list.
- `MRR` — range: [0, 1]
  - Mean Reciprocal Rank. Averages the inverse of the rank of the first relevant paper in the list.
- `nDCG@K` — range: [0, 1]
  - Normalized Discounted Cumulative Gain at K. Measures ranking quality by assigning higher scores to relevant papers appearing earlier, normalized by the ideal ranking.
- `HR@K` — range: [0, 1]
  - Hit Rate at K. Binary metric indicating whether at least one relevant paper appears in the top K results.

## Input / output format

**Input**: Title and abstract of the query paper.

**Output**: A ranked list of candidate papers retrieved from the MasterSet-CoreML-v1 pool.

## Scoring recipe

```python
def compute_metrics(pred_list, gold_set, K):
    top_k = set(pred_list[:K])
    gold = set(gold_set)
    recall_k = len(top_k & gold) / len(gold) if gold else 0.0
    hr_k = 1.0 if top_k & gold else 0.0
    # MAP, MRR, nDCG@K follow standard retrieval formulas
    return {'Recall@K': recall_k, 'HR@K': hr_k}
```

## Common pitfalls

- Using full manuscript text instead of the restricted title/abstract input violates the protocol.
- Including candidate papers published after the query paper violates the temporal split constraint.
- Treating all cited papers as positives instead of applying the three-tier must-cite criteria (baseline, core-relevance ≥4, or mention frequency ≥N).

## Evidence (verbatim from paper)

> Evaluation uses Recall@$K$ as the primary metric, reflecting the asymmetric cost of missing a must-cite paper. We also report MAP, MRR, nDCG@$K$, and HR@$K$ to characterise ranking quality at different depths.

## Citation

```bibtex
@misc{ratul2026masterset,
  title={MasterSet: A Large-Scale Benchmark for Must-Cite Citation Recommendation in the AI/ML Literature},
  author={Ratul et al. (2026)},
  year={2026},
  note={arXiv:2604.17680}
}
```

- arXiv: 2604.17680