Scitldr Eval

---
name: scitldr-eval
description: This benchmark evaluates the ability of models to generate extreme, single-sentence summaries (TLDRs) of scientific papers, capturing key contributions while bypassing background details. It tests both automated overlap metrics and human-judged informativeness and correctness under multi-target and multi-input settings. Use when the user wants to benchmark on SCITLDR, or asks about evaluating this task. Reports Rouge-1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2004.15011
  bibtex_key: cachola2020tldr
  confidence: high
---

# scitldr-eval

> TLDR: Extreme Summarization of Scientific Documents — Cachola et al. (2020) (arXiv:2004.15011, 2020)

## What this evaluates

This benchmark evaluates the ability of models to generate extreme, single-sentence summaries (TLDRs) of scientific papers, capturing key contributions while bypassing background details. It tests both automated overlap metrics and human-judged informativeness and correctness under multi-target and multi-input settings.

## Datasets

- **SCITLDR** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/allenai/scitldr

## Metrics

- `Rouge-1` **(primary)** — range: percent
  - Unigram overlap between generated TLDR and gold TLDRs. Score per paper is the maximum Rouge-1 score across all available gold TLDRs (author-written and reviewer-derived).
- `Rouge-2` — range: percent
  - Bigram overlap metric computed identically to Rouge-1, taking the maximum over all gold TLDRs per paper.
- `Rouge-L` — range: percent
  - Longest common subsequence overlap metric computed identically to Rouge-1, taking the maximum over all gold TLDRs per paper.
- `MRR` — range: [0, 1]
  - Mean Reciprocal Rank of length-normalized nugget counts for predicted vs gold TLDRs, used for human informativeness evaluation.
- `Correctness` — range: [1, 3]
  - Mean author-assigned score on a 1-3 scale (1=false/misleading, 2=partially accurate, 3=mostly correct).

## Input / output format

**Input**: Text of the paper's abstract, or the combined Abstract, Introduction, and Conclusion (AIC) sections.

**Output**: A single sentence TLDR summarizing the paper's key contributions.

## Scoring recipe

```python
def compute_rouge_max(predictions, gold_lists):
    paper_scores = []
    for pred, golds in zip(predictions, gold_lists):
        scores = [rouge_score(pred, g) for g in golds]
        paper_scores.append(max(scores))
    return sum(paper_scores) / len(paper_scores)
```

## Common pitfalls

- Averaging scores across multiple gold TLDRs instead of taking the maximum, which penalizes models for matching any valid summary.
- Failing to length-normalize nugget counts when computing human informativeness, which biases longer outputs.
- Evaluating on full-text input when the benchmark explicitly defines abstract-only and AIC (Abstract+Intro+Conclusion) settings.

## Evidence (verbatim from paper)

> Following recent work on extreme summarization (Narayan et al., 2018; Lewis et al., 2020), we use Rouge-1, Rouge-2, and Rouge-L (Lin, 2004) as our automated metrics. As discussed in Section 2, we have multiple target summaries available per paper. To exploit this during evaluation, we calculate the Rouge score of the system-generated TLDR with respect to each of the gold TLDRs for the corresponding paper (including its TLDR-Auth and all of its TLDRs-PR) individually. We take the maximum Rouge score over these gold TLDRs as the final Rouge score for that paper.

## Citation

```bibtex
@misc{cachola2020tldr,
  title={TLDR: Extreme Summarization of Scientific Documents},
  author={Cachola et al. (2020)},
  year={2020},
  note={arXiv:2004.15011}
}
```

- arXiv: 2004.15011