Summarization Fact Consistency Eval

---
name: summarization-fact-consistency-eval
description: Evaluates the factual consistency of human reference summaries across popular abstractive summarization benchmark datasets. It probes whether widely used datasets contain systematic factual errors or low-abstraction artifacts that compromise their validity as training and evaluation standards. Use when the user wants to benchmark on CNN/DM, XSUM, XL-Sum (English), or asks about evaluating this task. Reports Factuality Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2210.17378
  bibtex_key: guo2022questioning
  confidence: medium
---

# summarization-fact-consistency-eval

> Questioning the Validity of Summarization Datasets and Improving Their Factual Consistency — Guo et al. (2022) (arXiv:2210.17378, 2022)

## What this evaluates

Evaluates the factual consistency of human reference summaries across popular abstractive summarization benchmark datasets. It probes whether widely used datasets contain systematic factual errors or low-abstraction artifacts that compromise their validity as training and evaluation standards.

## Datasets

- **CNN/DM** — total 311971; splits: full (311971)
- **XSUM** — total 226711; splits: full (226711)
- **XL-Sum (English)** — total 329592; splits: full (329592)

## Metrics

- `Factuality Score` **(primary)** — range: [0, 1]
  - Computed using three off-the-shelf models (BERTScore_Art, BARTScore, DAE) that score the entailment or factual alignment of a reference summary given its source document. Higher scores indicate greater factual consistency.

## Input / output format

**Input**: Source document and its corresponding human reference summary from benchmark datasets.

**Output**: A factual consistency score for each document-summary pair.

## Scoring recipe

```python
scores = []
for doc, summary in dataset:
    s_bert = BERTScore_Art(doc, summary)
    s_bart = BARTScore(doc, summary)
    s_dae = DAE(doc, summary)
    scores.append([s_bert, s_bart, s_dae])
# Aggregate per dataset to compare factuality levels
dataset_score = mean(scores, axis=0)
```

## Common pitfalls

- Assuming human reference summaries are inherently factually correct 'gold standards'.
- Interpreting high factuality scores on CNN/DM as evidence of high abstraction quality, when they actually reflect low abstraction and bullet-point concatenation.
- Comparing datasets without accounting for differences in abstraction level or language coverage (e.g., XL-Sum covers 44 languages).

## Evidence (verbatim from paper)

> Here, we perform an analysis on the human references using the three factuality models introduced in Section 3.1: BERTScore_Art, BARTScore and DAE. The results are shown in Figure 3. We remark that factuality scores produced by all three models rank CNN/DM as the most factual dataset by a large margin.

## Citation

```bibtex
@misc{guo2022questioning,
  title={Questioning the Validity of Summarization Datasets and Improving Their Factual Consistency},
  author={Guo et al. (2022)},
  year={2022},
  note={arXiv:2210.17378}
}
```

- arXiv: 2210.17378