Cnndm Eval

---
name: cnndm-eval
description: Evaluates abstractive summarization quality by scoring generated summaries against human-written references and expert rubric-based scores. It measures how well automatic metrics correlate with human judgments across different summarization systems. Use when the user wants to benchmark on CNNDM, or asks about evaluating this task. Reports COMET.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2112.04139
  bibtex_key: first2021billboard
  confidence: high
---

# cnndm-eval

> Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand — First Author et al. (2021) (arXiv:2112.04139, 2021)

## What this evaluates

Evaluates abstractive summarization quality by scoring generated summaries against human-written references and expert rubric-based scores. It measures how well automatic metrics correlate with human judgments across different summarization systems.

## Datasets

- **CNNDM** — total ?; splits: test (-1)

## Metrics

- `COMET` **(primary)** — range: [0, 1]
  - A neural reference-based metric trained to predict human judgments. Scores are typically normalized to [0, 1].
- `COMET-QE` — range: [0, 1]
  - A referenceless variant of COMET that uses only the source sentence and hypothesis to predict quality.
- `BERTScore` — range: [0, 1]
  - A reference-based metric that computes semantic similarity using contextual embeddings from BERT.

## Input / output format

**Input**: News article text.

**Output**: Generated summary.

## Scoring recipe

```python
scores = []
for article, refs, summary in test_subset:
    metric_score = compute_metric(summary, refs)
    human_score = average(expert_scores[summary]) # avg over 4 dims & 3 experts
    scores.append((metric_score, human_score))
return pearson_correlation([s[0] for s in scores], [s[1] for s in scores])
```

## Common pitfalls

- Including author-written highlights in the reference set, which are often noisy or lack coherence.
- Not applying NLTK detokenization or Stanford CoreNLP truecasing to model outputs before evaluation.

## Evidence (verbatim from paper)

> Table 1: Summary of Billboards as of Jan. 10, 2022. ... Top Metric: COMET ... Ensemble of Metrics: 2.85·COMET+0.26·COMET-QE+0.01·BERTScore ... 100 test articles are annotated with 10 summaries written by humans... Each output summary is evaluated by three experts along four dimensions: coherence, consistency, fluency, and relevance. An instance-level score is computed by averaging scores over all these categories and the three experts.

## Citation

```bibtex
@misc{first2021billboard,
  title={Bidimensional Leaderboards: Generate and Evaluate Language Hand in Hand},
  author={First Author et al. (2021)},
  year={2021},
  note={arXiv:2112.04139}
}
```

- arXiv: 2112.04139