Sentence Summarization Eval

---
name: sentence-summarization-eval
description: Evaluates abstractive summarization models on condensing source sentences into title-like summaries. It measures summary quality via lexical/semantic overlap and human judgments, while explicitly quantifying the degree of verbatim copying from the source text. Use when the user wants to benchmark on Gigaword, Newsroom, or asks about evaluating this task. Reports ROUGE-2.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1911.10390
  bibtex_key: song2019controlling
  confidence: high
---

# sentence-summarization-eval

> Controlling the Amount of Verbatim Copying in Abstractive Summarization — Song et al. (2019) (arXiv:1911.10390, 2019)

## What this evaluates

Evaluates abstractive summarization models on condensing source sentences into title-like summaries. It measures summary quality via lexical/semantic overlap and human judgments, while explicitly quantifying the degree of verbatim copying from the source text.

## Datasets

- **Gigaword** — total 4011951; splits: train (4000000), val (10000), test (1951)
- **Newsroom** — total 241000; splits: train (199000), val (21000), test (21000)

## Metrics

- `ROUGE-2` **(primary)** — range: percent
  - Recall of overlapping 2-grams between the generated summary and the reference summary, expressed as a percentage.
- `BERTScore` — range: [0, 1]
  - Cosine similarity between BERT embeddings of generated and reference tokens, averaged across all tokens to measure semantic similarity.
- `Copy Rate` — range: percent
  - Percentage of summary n-grams (n=1,2,3,4) that appear verbatim in the source text, averaged across all n-gram orders.

## Input / output format

**Input**: Source sentence (first sentence of a news article)

**Output**: Title-like summary (single sentence)

## Scoring recipe

```python
def compute_metrics(predictions, references, sources):
    rouge2_recall = rouge(predictions, references, rouge_types=['rouge2']).recall * 100
    bert_score_val = bert_score(predictions, references)
    copy_rates = []
    for pred, src in zip(predictions, sources):
        ngrams = [pred[i:i+n] for n in range(1,5) for i in range(len(pred)-n+1)]
        matches = sum(1 for ng in ngrams if ng in src)
        copy_rates.append(matches / len(ngrams) if ngrams else 0)
    avg_copy_rate = sum(copy_rates) / len(copy_rates) * 100
    return rouge2_recall, bert_score_val, avg_copy_rate
```

## Common pitfalls

- Copy rate is averaged across 1-gram to 4-gram orders, not just unigrams.
- Human evaluation specifically checks grammaticality, informativeness, and factual consistency (true-to-original), not just overall fluency.
- Learning rate is halved when validation loss plateaus for 40,000 steps, which is critical for reproducing the reported results.

## Evidence (verbatim from paper)

> We evaluate our proposed method on the sentence summarization task. The goal is to condense a lengthy source sentence to a title-like summary. Comparing to single-document summarization, sentence summarization deals less with content selection; its ground-truth summaries also contain more paraphrasing and abstraction. We conduct experiments on the Gigaword (Parker 2011) and Newsroom (Grusky, Naaman, and Artzi 2018) datasets. Gigaword articles were collected during 1995-2010 and Newsroom spans the range of 1998-2017. We pair the first sentence of each article with its title to form an instance. The train-valid/test splits contain 4 million/10k/1951 instances for Gigaword and 199k/21k/21k instances for Newsroom. We experiment with both datasets to understand not only the copying behavior, but also domain adaptation effects for various models. Despite that only single reference summaries are available in benchmark evaluations, we are able to evaluate summary quality along multiple dimensions, using automatic metrics based on lexical similarity (ROUGE; Lin, 2004) and semantic similarity (BERTScore; Zhang et al., 2019), and through human assessment of grammaticality, informativeness, an

## Citation

```bibtex
@misc{song2019controlling,
  title={Controlling the Amount of Verbatim Copying in Abstractive Summarization},
  author={Song et al. (2019)},
  year={2019},
  note={arXiv:1911.10390}
}
```

- arXiv: 1911.10390