Narrasum Eval

---
name: narrasum-eval
description: This benchmark evaluates a model's ability to perform abstractive and extractive summarization on long-form narrative texts (movie/TV plot descriptions). It probes the model's capacity to capture event causality, character motivations, temporal dynamics, and overall narrative coherence while maintaining faithfulness to the source document. Use when the user wants to benchmark on NarraSum, or asks about evaluating this task. Reports ROUGE F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2212.01476
  bibtex_key: zhao2022narrasum
  confidence: high
---

# narrasum-eval

> NarraSum: A Large-Scale Dataset for Abstractive Narrative Summarization — Zhao et al. (2022) (arXiv:2212.01476, 2022)

## What this evaluates

This benchmark evaluates a model's ability to perform abstractive and extractive summarization on long-form narrative texts (movie/TV plot descriptions). It probes the model's capacity to capture event causality, character motivations, temporal dynamics, and overall narrative coherence while maintaining faithfulness to the source document.

## Datasets

- **NarraSum** — total 122000; splits: train (-1), val (-1), test (-1); repo https://github.com/zhaochaocs/narrasum

## Metrics

- `ROUGE F1` **(primary)** — range: percent
  - F1 score computed at unigram (R-1), bigram (R-2), and longest common subsequence (R-L) overlap levels between generated and reference summaries.
- `SummaC` — range: [0, 1]
  - Automatic measure for summary faithfulness that detects inconsistencies between the summary and the source document, specifically designed for long inputs and outputs.
- `Human Evaluation Scores` — range: [1, 5]
  - Average score on a 5-point Likert scale rated by human annotators across four dimensions: Fluency, Faithfulness, Coherence, and Informativeness.

## Input / output format

**Input**: Long-form narrative document (movie or TV episode plot description)

**Output**: Generated abstractive or extractive summary

## Scoring recipe

```python
# Compute ROUGE F1 scores
rouge1 = rouge(predictions, references, n=1).f1
rouge2 = rouge(predictions, references, n=2).f1
rougeL = rouge(predictions, references, lcs=True).f1

# Compute SummaC for abstractive models only
sc_scores = [summac_faithfulness(pred, gold) for pred, gold in zip(predictions, references)]

# Human evaluation averages (5-point Likert)
human_avg = {dim: np.mean(ratings[dim]) for dim in ["Fluency", "Faithfulness", "Coherence", "Informativeness"]}
```

## Common pitfalls

- SummaC is explicitly noted as only applicable to abstractive summaries, as extractive ones are considered faithful by design.
- Human evaluation was restricted to the top 2 performing models (T5-Large, LED-Large) on a random sample of 100 test instances, not the full test set.
- Compression degree analysis requires splitting the test set by summary-to-document length ratio, which may skew results if not stratified properly.

## Evidence (verbatim from paper)

> We evaluate the generated summaries using ROUGE $F_{1}$ score. $^{8}$ We further include SummaC (Laban et al., 2022), an automatic measure for summary faithfulness. It achieves state-of-the-art on the benchmark of summary inconsistency detection, and is feasible to be applied to long input and output.

## Citation

```bibtex
@misc{zhao2022narrasum,
  title={NarraSum: A Large-Scale Dataset for Abstractive Narrative Summarization},
  author={Zhao et al. (2022)},
  year={2022},
  note={arXiv:2212.01476}
}
```

- arXiv: 2212.01476