Summeval Eval

---
name: summeval-eval
description: This benchmark evaluates how well automatic summarization metrics align with human judgments across multiple quality dimensions. It probes whether standard n-gram, embedding-based, and reference-less metrics reliably predict human-perceived coherence, consistency, fluency, and relevance of generated summaries. Use when the user wants to benchmark on SummEval, or asks about evaluating this task. Reports Kendall’s tau.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2007.12626
  bibtex_key: [REDACTED]
  confidence: high
---

# summeval-eval

> SummEval: Re-evaluating Summarization Evaluation — Fabbri et al. (2020) (arXiv:2007.12626, 2020)

## What this evaluates

This benchmark evaluates how well automatic summarization metrics align with human judgments across multiple quality dimensions. It probes whether standard n-gram, embedding-based, and reference-less metrics reliably predict human-perceived coherence, consistency, fluency, and relevance of generated summaries.

## Datasets

- **SummEval** — total ?; splits: test (-1); repo https://github.com/Yale-LILY/SummEval

## Metrics

- `Kendall’s tau` **(primary)** — range: [-1, 1]
  - A rank correlation coefficient measuring the correspondence between the rankings of automatic metric scores and human judgment scores across models or examples. Values range from -1 to 1, where 1 indicates perfect agreement.
- `ROUGE-1` — range: [0, 1]
  - Measures unigram overlap between generated and reference summaries.
- `BERTScore` — range: [0, 1]
  - Computes token-level cosine similarity between contextualized BERT embeddings of candidate and reference texts.
- `MoverScore` — range: [0, 1]
  - Measures semantic distance using Word Mover’s Distance over n-gram embeddings pooled from BERT representations.

## Input / output format

**Input**: Source document, reference summaries (for most metrics), and model-generated summary.

**Output**: Generated summary text.

## Scoring recipe

```python
def evaluate_summaries(generated_summaries, references, source_docs, human_judgments):
    auto_scores = {}
    for metric in ['ROUGE-1', 'BERTScore', 'MoverScore', 'BLEU', 'METEOR']:
        auto_scores[metric] = [compute_metric(metric, gen, refs) for gen, refs in zip(generated_summaries, references)]
    
    # System-level correlation with human judgments
    correlations = {}
    for metric, scores in auto_scores.items():
        tau = kendalltau(scores, human_judgments)
        correlations[metric] = tau
    return correlations
```

## Common pitfalls

- ROUGE scores frequently correlate poorly with human judgments on fluency and coherence, leading to overestimation of model quality.
- Reference-less metrics like BLANC and SUPERT require the source document and cannot be computed without it.
- The benchmark emphasizes system-level correlation (Table 2) rather than instance-level accuracy, so per-example scores should not be treated as definitive quality indicators.

## Evidence (verbatim from paper)

> Table 2: Kendall’s tau correlation coefficients of expert annotations computed on a system-level along four quality dimensions with automatic metrics using 11 reference summaries per example.

## Citation

```bibtex
@misc{fabbri2020summeval,
  title={SummEval: Re-evaluating Summarization Evaluation},
  author={Fabbri et al. (2020)},
  year={2020},
  note={arXiv:2007.12626}
}
```

- arXiv: 2007.12626