Moviesum Eval

---
name: moviesum-eval
description: Evaluates the ability of abstractive summarization models to generate concise, coherent summaries of long, dispersed movie screenplay narratives. It probes long-document understanding, narrative coherence, and the model's capacity to synthesize information across thousands of tokens. Use when the user wants to benchmark on MovieSum, or asks about evaluating this task. Reports ROUGE F1 (1/2/L).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.06281
  bibtex_key: saxena2024moviesum
  confidence: high
---

# moviesum-eval

> MovieSum: An Abstractive Summarization Dataset for Movie Screenplays — Saxena et al. (2024) (arXiv:2408.06281, 2024)

## What this evaluates

Evaluates the ability of abstractive summarization models to generate concise, coherent summaries of long, dispersed movie screenplay narratives. It probes long-document understanding, narrative coherence, and the model's capacity to synthesize information across thousands of tokens.

## Datasets

- **MovieSum** — total ?; splits: test (-1); repo https://github.com/saxenarohit/MovieSum

## Metrics

- `ROUGE F1 (1/2/L)` **(primary)** — range: [0, 1]
  - F1 score computed over unigram (ROUGE-1), bigram (ROUGE-2), and longest common subsequence (ROUGE-L) overlaps between the generated summary and the reference Wikipedia plot summary.
- `BERTScore` — range: [0, 1]
  - F1 score computed using contextual embeddings from BERT to measure semantic similarity between generated and reference texts, independent of exact lexical overlap.

## Input / output format

**Input**: Full movie screenplay text (long document, varying token lengths depending on model context window)

**Output**: Generated abstractive summary text

## Scoring recipe

```python
def compute_metrics(predictions, references):
    # Compute ROUGE F1 for unigrams, bigrams, and LCS
    rouge_results = rouge_score(references, predictions, use_stemmer=True, use_aggregator=True)
    rouge_f1 = {k: v['fmeasure'] for k, v in rouge_results.items()}
    
    # Compute BERTScore F1
    p, r, f1 = bert_score.score(predictions, references, lang='en')
    bert_f1 = f1.mean().item()
    
    return {
        'rouge_1_f1': rouge_f1['rouge1'],
        'rouge_2_f1': rouge_f1['rouge2'],
        'rouge_l_f1': rouge_f1['rougeL'],
        'bertscore_f1': bert_f1
    }
```

## Common pitfalls

- ROUGE F1 inherently favors longer summaries, so extractive baselines like Lead-1024 often outperform abstractive models simply due to length bias.
- Zero-shot models with large context windows (e.g., 16K) do not automatically attend to the full input, requiring chunking strategies or fine-tuning to achieve competitive performance.
- Evaluating only on the test set without a validation split may lead to overfitting when fine-tuning long-context models like LongT5 or LED.

## Evidence (verbatim from paper)

> Table[3] shows the summarization evaluation results using ROUGE F1 (1/2/L) scores (Lin, [2004]) and BERTScore (Zhang et al., [2019]) on MovieSum.

## Citation

```bibtex
@misc{saxena2024moviesum,
  title={MovieSum: An Abstractive Summarization Dataset for Movie Screenplays},
  author={Saxena et al. (2024)},
  year={2024},
  note={arXiv:2408.06281}
}
```

- arXiv: 2408.06281