Reddit Tifu Summarization Eval

---
name: reddit-tifu-summarization-eval
description: This evaluation probes a model's ability to generate abstractive summaries from informal, user-generated text and formal documents. It measures how well the model captures long-range dependencies and abstracts key information without relying on extractive heuristics. Use when the user wants to benchmark on Reddit TIFU, Newsroom-Abs, XSum, or asks about evaluating this task. Reports ROUGE-1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1811.00783
  bibtex_key: kim2018abstractive
  confidence: high
---

# reddit-tifu-summarization-eval

> Abstractive Summarization of Reddit Posts with Multi-level Memory Networks — Kim et al. (2018) (arXiv:1811.00783, 2018)

## What this evaluates

This evaluation probes a model's ability to generate abstractive summaries from informal, user-generated text and formal documents. It measures how well the model captures long-range dependencies and abstracts key information without relying on extractive heuristics.

## Datasets

- **Reddit TIFU** — total ?; splits: test (-1)
- **Newsroom-Abs** — total ?; splits: test (-1)
- **XSum** — total ?; splits: test (-1)

## Metrics

- `ROUGE-1` **(primary)** — range: [0, 1]
  - F1 score based on overlapping unigrams between generated and reference summaries. Higher values indicate better performance.
- `ROUGE-2` — range: [0, 1]
  - F1 score based on overlapping bigrams between generated and reference summaries. Higher values indicate better performance.
- `ROUGE-L` — range: [0, 1]
  - F1 score based on the longest common subsequence (LCS) between generated and reference summaries. Higher values indicate better performance.
- `Perplexity` — range: other
  - Exponential of the average negative log-likelihood of the reference tokens under the model distribution. Lower values indicate better performance.

## Input / output format

**Input**: Source text (Reddit post or article) provided as a sequence of words/tokens.

**Output**: Abstractive summary generated as a sequence of words/tokens.

## Scoring recipe

```python
def evaluate(predictions, gold):
    r1 = rouge_f1(gold, predictions, ngram=1)
    r2 = rouge_f1(gold, predictions, ngram=2)
    rl = rouge_f1(gold, predictions, lcs=True)
    ppl = exp(-mean(log_prob(w, context) for w in gold))
    return {'ROUGE-1': r1, 'ROUGE-2': r2, 'ROUGE-L': rl, 'PPL': ppl}
```

## Common pitfalls

- ROUGE scores may underestimate performance on this dataset because it is highly abstractive and often paraphrases source text rather than copying exact phrases.
- Perplexity is evaluated on the model's internal probability distribution over the reference text, not on the generated output, and lower values indicate better performance (inverse of typical accuracy metrics).
- Human preference evaluation (AMT) uses pairwise comparisons with random ordering; results can be sensitive to prompt phrasing and annotator fatigue.

## Evidence (verbatim from paper)

> We evaluate the summarization performance with two language metrics: perplexity and standard F1 ROUGE scores Lin ([2004](#bib.bib20 "")). We remind that lower perplexity and higher ROUGE scores indicate better performance.

## Citation

```bibtex
@misc{kim2018abstractive,
  title={Abstractive Summarization of Reddit Posts with Multi-level Memory Networks},
  author={Kim et al. (2018)},
  year={2018},
  note={arXiv:1811.00783}
}
```

- arXiv: 1811.00783