Wikihow Eval

---
name: wikihow-eval
description: Evaluates text summarization systems on procedural, step-by-step articles written by non-journalists. It probes the model's ability to handle long sequences, non-inverted-pyramid structures, and high-abstraction content compared to standard news datasets. Use when the user wants to benchmark on WikiHow, or asks about evaluating this task. Reports ROUGE-L.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1810.09305
  bibtex_key: koupae2018wikihow
  confidence: high
---

# wikihow-eval

> WikiHow: A Large Scale Text Summarization Dataset — Koupae et al. (2018) (arXiv:1810.09305, 2018)

## What this evaluates

Evaluates text summarization systems on procedural, step-by-step articles written by non-journalists. It probes the model's ability to handle long sequences, non-inverted-pyramid structures, and high-abstraction content compared to standard news datasets.

## Datasets

- **WikiHow** — total 230000; splits: (unstated)

## Metrics

- `ROUGE-L` **(primary)** — range: [0, 1]
  - F1 score computed over the longest common subsequence of n-grams between the generated summary and the reference summary. Evaluated using exact matches, and optionally stem, paraphrase, and synonym matching (s/p/s).
- `METEOR` — range: [0, 1]
  - F1 score based on exact matches, stem matches, paraphrase matches, and synonym matches between the prediction and reference summary.

## Input / output format

**Input**: Procedural article text containing step-by-step instructions.

**Output**: Generated summary text.

## Scoring recipe

```python
def evaluate(predictions, gold):
    # Uses Pyrouge package as specified in the paper
    rouge_scores = pyrouge.compute_rouge_scores(predictions, gold)
    meteor_scores = pyrouge.compute_meteor_scores(predictions, gold)
    # Returns F1 scores for ROUGE-1, ROUGE-2, ROUGE-L and METEOR
    return rouge_scores, meteor_scores
```

## Common pitfalls

- Lead-3 baseline is adapted for WikiHow by concatenating the first sentence of each paragraph, rather than using the literal first three sentences of the article.
- Standard ROUGE/METEOR metrics may not fully capture the high abstraction level or compression ratio required for procedural text, often favoring extractive baselines.
- Sequence-to-sequence baselines without pointer-generator mechanisms fail on Out-Of-Vocabulary (OOV) words common in procedural articles.

## Evidence (verbatim from paper)

> To study the performance of the evaluated systems, we used the Pyrouge package to report the F1 score for ROUGE-1, ROUGE-2 and ROUGE-L (Lin, 2004) and the METEOR (Banerjee and Lavie, 2005) both based on the exact matches and on inclusion of stem, paraphrasing and synonyms $(s/p/s)$ to evaluate the methods.

## Citation

```bibtex
@misc{koupae2018wikihow,
  title={WikiHow: A Large Scale Text Summarization Dataset},
  author={Koupae et al. (2018)},
  year={2018},
  note={arXiv:1810.09305}
}
```

- arXiv: 1810.09305