Citesum Eval

---
name: citesum-eval
description: This benchmark evaluates a model's ability to generate concise, single-sentence summaries of scientific papers using citation sentences as ground truth. It probes extreme summarization capabilities and domain adaptation across academic disciplines. Use when the user wants to benchmark on CiteSum, or asks about evaluating this task. Reports ROUGE-1, ROUGE-2, ROUGE-L.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2205.06207
  bibtex_key: mao2022citesum
  confidence: high
---

# citesum-eval

> CiteSum: Citation Text-guided Scientific Extreme Summarization and Domain Adaptation with Limited Supervision — Mao et al. (2022) (arXiv:2205.06207, 2022)

## What this evaluates

This benchmark evaluates a model's ability to generate concise, single-sentence summaries of scientific papers using citation sentences as ground truth. It probes extreme summarization capabilities and domain adaptation across academic disciplines.

## Datasets

- **CiteSum** — total 92946; splits: train (83304), val (4721), test (4921); repo https://github.com/morningmoni/CiteSum

## Metrics

- `ROUGE-1, ROUGE-2, ROUGE-L` **(primary)** — range: [0, 1]
  - Standard ROUGE recall metrics measuring n-gram (ROUGE-1/2) and longest common subsequence (ROUGE-L) overlap between the generated summary and the reference citation sentence.

## Input / output format

**Input**: Abstract or full text of a scientific paper (average ~255 words)

**Output**: A single-sentence summary (citation sentence) of the cited paper (average ~23 words)

## Scoring recipe

```python
def compute_metrics(predictions, references):
    scores = {}
    for metric in ['ROUGE-1', 'ROUGE-2', 'ROUGE-L']:
        overlaps = [calculate_overlap(ref, pred) for ref, pred in zip(references, predictions)]
        scores[metric] = sum(overlaps) / len(overlaps)
    return scores
```

## Common pitfalls

- Citation texts are not always accurate summaries; ~10% are misleading or partially accurate, so models may inherit noise.
- The dataset is heavily skewed toward computer science, mathematics, and engineering due to the Related Work section filtering, limiting generalizability to other disciplines.
- Splits must be paper-level to prevent label leakage, as a single paper can be cited multiple times with different citation sentences.

## Evidence (verbatim from paper)

> We set T to 50/20/40 for ROUGE-1/2/L recall through manual examination, resulting in a ROUGE-1/2/L recall of 73.1/39.4/58.5 after filtering. After data filtering and preprocessing, there are 92,946 examples in the final citation text-guided summarization dataset, which we name as CiteSum. We take about 5% of the data as the validation and test sets respectively, and the remaining 90% as the training set.

## Citation

```bibtex
@misc{mao2022citesum,
  title={CiteSum: Citation Text-guided Scientific Extreme Summarization and Domain Adaptation with Limited Supervision},
  author={Mao et al. (2022)},
  year={2022},
  note={arXiv:2205.06207}
}
```

- arXiv: 2205.06207