Wikiasp Eval

---
name: wikiasp-eval
description: Evaluates multi-domain aspect-based summarization, requiring models to first discover relevant aspects (Wikipedia section titles) from cited references and then generate domain-specific summaries. It probes content selection, cross-document pronoun resolution, and temporal ordering in multi-source generation. Use when the user wants to benchmark on WikiAsp, or asks about evaluating this task. Reports R-2.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2011.07832
  bibtex_key: [REDACTED]
  confidence: high
---

# wikiasp-eval

> WikiAsp: A Dataset for Multi-domain Aspect-based Summarization — Hayashi et al. (2020) (arXiv:2011.07832, 2020)

## What this evaluates

Evaluates multi-domain aspect-based summarization, requiring models to first discover relevant aspects (Wikipedia section titles) from cited references and then generate domain-specific summaries. It probes content selection, cross-document pronoun resolution, and temporal ordering in multi-source generation.

## Datasets

- **WikiAsp** — total ?; splits: train (-1), test (-1); repo http://github.com/neulab/wikiasp

## Metrics

- `R-2` **(primary)** — range: [0, 1]
  - ROUGE-2 recall/precision/F1 based on bigram overlap between the generated summary and the reference summary.
- `R-L` — range: [0, 1]
  - ROUGE-L longest common subsequence overlap between generated and reference summaries.
- `F-1` — range: [0, 1]
  - Harmonic mean of precision and recall for aspect classification.

## Input / output format

**Input**: Cited reference texts and a target aspect (section title) for summarization; Wikipedia section and cited references for aspect discovery.

**Output**: Aspect label (classification) or generated summary text (summarization).

## Scoring recipe

```python
def score_aspect(preds, gold):
    tp = sum(1 for p, g in zip(preds, gold) if p == g and p == 'aspect')
    fp = sum(1 for p, g in zip(preds, gold) if p == 'aspect' and g != 'aspect')
    fn = sum(1 for p, g in zip(preds, gold) if p != 'aspect' and g == 'aspect')
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

def score_summ(gold, pred):
    return rouge_score(gold, pred, rouge_types=['rouge2', 'rougeL'])
```

## Common pitfalls

- Class imbalance in aspect frequency distributions causes poorly calibrated classifiers that achieve high recall but low precision.
- Low ROUGE scores occur even with oracle baselines because important summary phrases are not rare, making n-gram overlap a poor proxy for content quality.

## Evidence (verbatim from paper)

> The automatic evaluation results are shown in Table 5. Neither baseline unanimously outperformed the other on all domains, but we observe that Pre-Summ (abstractive) performs better than TextRank (extractive) on average. The low R-2 and R-L scores by both models despite the oracle being relatively higher suggest that important phrases to be summarized do not appear rarely.[9] ... We show the aspect discovery results in Table 4. ... Class imbalance also plays a role here; predicting the major classes give high recall due to skew aspect frequency distributions.

## Citation

```bibtex
@misc{hayashi2020wikiasp,
  title={WikiAsp: A Dataset for Multi-domain Aspect-based Summarization},
  author={Hayashi et al. (2020)},
  year={2020},
  note={arXiv:2011.07832}
}
```

- arXiv: 2011.07832