Citation Rec Eval

---
name: citation-rec-eval
description: Evaluates the ability of citation recommendation systems to retrieve and rank relevant academic papers given a query context. It probes ranking quality, recall of relevant candidates, and normalized discounted cumulative gain across multiple academic datasets. Use when the user wants to benchmark on ACL-200, FullTextPeerRead, Refseer, arXiv, ArSyTa, or asks about evaluating this task. Reports MRR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.17361
  bibtex_key: goyal2026publicprofile
  confidence: high
---

# citation-rec-eval

> Public Profile Matters: A Scalable Integrated Approach to Recommend Citations in the Wild — Goyal et al. (2026) (arXiv:2603.17361, 2026)

## What this evaluates

Evaluates the ability of citation recommendation systems to retrieve and rank relevant academic papers given a query context. It probes ranking quality, recall of relevant candidates, and normalized discounted cumulative gain across multiple academic datasets.

## Datasets

- **ACL-200** — total ?; splits: test (-1)
- **FullTextPeerRead** — total ?; splits: test (-1)
- **Refseer** — total ?; splits: test (-1)
- **arXiv** — total ?; splits: test (-1)
- **ArSyTa** — total ?; splits: test (-1)

## Metrics

- `MRR` **(primary)** — range: percent
  - Mean Reciprocal Rank: the average of the reciprocal of the rank of the first relevant document in the ranked list for each query.
- `Recall@K` — range: percent
  - Recall@K: the fraction of relevant documents found within the top K retrieved results, averaged over queries.
- `NDCG@K` — range: percent
  - Normalized Discounted Cumulative Gain@K: measures ranking quality by assigning higher scores to relevant documents appearing higher in the list, normalized by the ideal DCG.

## Input / output format

**Input**: Citation context or paper text used as a query to retrieve candidate references from a corpus.

**Output**: A ranked list of candidate papers.

## Scoring recipe

```python
def compute_metrics(ranked_list, relevant_docs, K):
    # MRR
    rr = 1.0 / (ranked_list.index(relevant_docs[0]) + 1) if relevant_docs[0] in ranked_list else 0.0
    # Recall@K
    recall_k = len(set(ranked_list[:K]) & set(relevant_docs)) / len(relevant_docs)
    # NDCG@K
    dcg = sum(1.0 / math.log2(i + 2) for i, doc in enumerate(ranked_list[:K]) if doc in relevant_docs)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(relevant_docs), K)))
    ndcg_k = dcg / idcg if idcg > 0 else 0.0
    return rr, recall_k, ndcg_k
```

## Common pitfalls

- The evaluation enforces a strict inductive setting with temporal constraints, preventing models from using future papers or undefined task-specific parameters.
- Metrics are reported as percentages but represent standard IR scores; Recall@K and NDCG@K must be computed at multiple cutoffs (5, 10, 20, 50, 300) as specified in the tables.

## Evidence (verbatim from paper)

> To provide a multi-faceted assessment of ranking performance, we employ a suite of standard information retrieval metrics (%), namely, Mean Reciprocal Rank (MRR), Recall@K, and NDCG@K.

## Citation

```bibtex
@misc{goyal2026publicprofile,
  title={Public Profile Matters: A Scalable Integrated Approach to Recommend Citations in the Wild},
  author={Goyal et al. (2026)},
  year={2026},
  note={arXiv:2603.17361}
}
```

- arXiv: 2603.17361