Graphsum Mds Eval

---
name: graphsum-mds-eval
description: Evaluates multi-document summarization performance and model explainability by comparing sentence vs. paragraph inputs and analyzing how attention weights correlate with reference summary similarity to reveal positional bias. Use when the user wants to benchmark on MultiNews, WikiSum, or asks about evaluating this task. Reports ROUGE-F (1/2/L).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2105.11908
  bibtex_key: hickmann2021graphsum
  confidence: high
---

# graphsum-mds-eval

> Analysis of GraphSum's Attention Weights to Improve the Explainability of Multi-Document Summarization — Hickmann et al. (2021) (arXiv:2105.11908, 2021)

## What this evaluates

Evaluates multi-document summarization performance and model explainability by comparing sentence vs. paragraph inputs and analyzing how attention weights correlate with reference summary similarity to reveal positional bias.

## Datasets

- **MultiNews** — total ?; splits: test (-1)
- **WikiSum** — total ?; splits: test (-1)

## Metrics

- `ROUGE-F (1/2/L)` **(primary)** — range: percent
  - Standard ROUGE-F1, F2, and F-L scores measuring n-gram overlap between generated and reference summaries, scaled to 0-100.
- `correlation coefficient` — range: [-1, 1]
  - Pearson correlation coefficient computed between aggregated attention weights per decoding layer and ROUGE scores.

## Input / output format

**Input**: Multiple source documents represented as sequences of tokens or paragraph-level textual units.

**Output**: Abstractive summary generated token-by-token.

## Scoring recipe

```python
# Compute ROUGE-F scores for each generated summary
rouge_scores = compute_rouge_f1_f2_fl(generated_summary, reference_summary)

# Aggregate attention weights per decoding layer
attention_weights = extract_aggregated_attention_weights(model, layer)

# Compute correlation between attention weights and ROUGE scores
correlation = pearson_correlation(attention_weights, rouge_scores)
```

## Common pitfalls

- Positional bias in news summaries heavily influences attention patterns, making later layers appear more correlated with source origin.
- Heatmap visualizations of attention artifacts are skewed by the distribution of summary lengths in the test set.
- WikiSum lacks document/paragraph boundary metadata, preventing positional bias analysis on that dataset.

## Evidence (verbatim from paper)

> Table 1 shows the results for the comparison of sentence-level vs. paragraph-level summaries generated by our MDS models on the test dataset. With a batchsize of 3,000 target tokens, the paragraph-level model outperforms the sentence-level model with regard to the ROUGE score by a small margin.

## Citation

```bibtex
@misc{hickmann2021graphsum,
  title={Analysis of GraphSum's Attention Weights to Improve the Explainability of Multi-Document Summarization},
  author={Hickmann et al. (2021)},
  year={2021},
  note={arXiv:2105.11908}
}
```

- arXiv: 2105.11908