Debatsum Summarization Eval

---
name: debatsum-summarization-eval
description: Evaluates transformer-based models on word-level extractive summarization for policy debate evidence. It measures how well models can identify and extract relevant tokens to form summaries of debate arguments. Use when the user wants to benchmark on DebateSum, or asks about evaluating this task. Reports ROUGE F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2011.07251
  bibtex_key: roush2020debatesum
  confidence: high
---

# debatsum-summarization-eval

> DebateSum: A large-scale argument mining and summarization dataset — Roush et al. (2020) (arXiv:2011.07251, 2020)

## What this evaluates

Evaluates transformer-based models on word-level extractive summarization for policy debate evidence. It measures how well models can identify and extract relevant tokens to form summaries of debate arguments.

## Datasets

- **DebateSum** — total 187386; splits: test (18738); repo https://github.com/Hellisotherpeople/debate2vec

## Metrics

- `ROUGE F1` **(primary)** — range: percent
  - Computes the F1 score for unigram (ROUGE-1), bigram (ROUGE-2), and longest common subsequence (ROUGE-L) overlap between the predicted extractive summary and the gold summary. Evaluated using default settings of py-rouge.

## Input / output format

**Input**: A policy debate document represented as a sequence of tokens.

**Output**: A sequence of token-level labels indicating whether each token should be included in the extractive summary ('underlined' or 'not-underlined').

## Scoring recipe

```python
def score(predictions, golds):
    f1_scores = []
    for pred_tokens, gold_tokens in zip(predictions, golds):
        pred_summary = ' '.join([t for t, l in zip(pred_tokens, labels) if l == 'underlined'])
        gold_summary = ' '.join(gold_tokens)
        rouge = ROUGE()
        f1 = rouge.get_scores(pred_summary, gold_summary)[0]['rouge-1']['f']
        f1_scores.append(f1 * 100)
    return sum(f1_scores) / len(f1_scores)
```

## Common pitfalls

- ROUGE scores are computed on reconstructed summaries from token labels, so tokenization mismatches or boundary errors can disproportionately impact scores.
- The paper uses default py-rouge settings, which may differ from standard NLTK or official ROUGE-1.5.5 implementations, making cross-study comparison difficult.

## Evidence (verbatim from paper)

> We evaluate our models on a test split of 18,738 documents. The ROUGE metric is used for measuring summarization quality. We evaluate using the default settings of py-rogue on our models. We report the ROUGE F1 scores of these transformer models.

## Citation

```bibtex
@misc{roush2020debatesum,
  title={DebateSum: A large-scale argument mining and summarization dataset},
  author={Roush et al. (2020)},
  year={2020},
  note={arXiv:2011.07251}
}
```

- arXiv: 2011.07251