Postersum Eval

---
name: postersum-eval
description: Evaluates multimodal large language models' ability to generate accurate, abstractive summaries from complex, visually dense scientific posters. It probes layout understanding, visual-textual integration, and hierarchical summarization capabilities by measuring how well models extract and synthesize information from combined image and text inputs. Use when the user wants to benchmark on PosterSum, or asks about evaluating this task. Reports ROUGE-L.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.17540
  bibtex_key: saxena2025postersum
  confidence: high
---

# postersum-eval

> PosterSum: A Multimodal Benchmark for Scientific Poster Summarization — Saxena et al. (2025) (arXiv:2502.17540, 2025)

## What this evaluates

Evaluates multimodal large language models' ability to generate accurate, abstractive summaries from complex, visually dense scientific posters. It probes layout understanding, visual-textual integration, and hierarchical summarization capabilities by measuring how well models extract and synthesize information from combined image and text inputs.

## Datasets

- **PosterSum** — total 16305; splits: test (-1); repo https://github.com/saxenarohit/postersum

## Metrics

- `ROUGE-L` **(primary)** — range: percent
  - Computes the longest common subsequence between the reference abstract and the generated summary, normalized by the reference length. Standard NLTK or rouge-score implementation is used.
- `ROUGE-1` — range: percent
  - Measures unigram overlap between reference and generated text, normalized by reference length. Standard implementation.
- `ROUGE-2` — range: percent
  - Measures bigram overlap between reference and generated text, normalized by reference length. Standard implementation.
- `METEOR` — range: percent
  - Matches generated text to reference using exact, stem, synonym, and paraphrase matches, then applies a penalty for fragmentation. Standard implementation.
- `BERTScore-F1` — range: percent
  - Computes token-level similarity using contextual embeddings from a pre-trained BERT model, then calculates precision, recall, and F1 score.
- `SacreBLEU` — range: percent
  - Corpus-level BLEU score with standard tokenization and sentence splitting (paper contains a typo 'ScareBLEU'). Standard implementation.

## Input / output format

**Input**: Multimodal scientific poster image (containing text, figures, charts, and layout) paired with its corresponding ground-truth abstract for reference.

**Output**: A single abstractive summary text (poster abstract) generated by the model.

## Scoring recipe

```python
def compute_metrics(predictions, references):
    rouge_l = rouge_score(references, predictions, rouge_types=['l'])['f'] * 100
    rouge_1 = rouge_score(references, predictions, rouge_types=['r'])['f'] * 100
    rouge_2 = rouge_score(references, predictions, rouge_types=['2'])['f'] * 100
    meteor = meteor_score(references, predictions) * 100
    bertscore = bert_score.score(predictions, references, lang='en')[2].mean() * 100
    sacrebleu = sacrebleu.corpus_sacrebleu(predictions, [references]).score
    return {'ROUGE-L': rouge_l, 'ROUGE-1': rouge_1, 'ROUGE-2': rouge_2, 'METEOR': meteor, 'BERTScore-F1': bertscore, 'SacreBLEU': sacrebleu}
```

## Common pitfalls

- Models often fail to extract text correctly from complex layouts, causing pure OCR baselines to underperform naive multimodal models that ignore visual structure.
- Single-pass generation on the full poster image leads to information loss; the benchmark demonstrates that hierarchical, region-segmented approaches significantly outperform end-to-end generation.
- ROUGE-based metrics heavily penalize paraphrasing, so models that closely mirror the original abstract's phrasing score higher than those that produce semantically equivalent but lexically different summaries.

## Evidence (verbatim from paper)

> Our proposed method outperforms all other models, including closed-source models, on all metrics, achieving ROUGE-1/2/L scores of 46.68, 15.73, and 24.18, respectively, with a 3.14% gain on ROUGE-L compared to open-source models. It also attains a substantially higher ScareBLEU score (12.63) and a BERTScore-F1 of 61.37.

## Citation

```bibtex
@misc{saxena2025postersum,
  title={PosterSum: A Multimodal Benchmark for Scientific Poster Summarization},
  author={Saxena et al. (2025)},
  year={2025},
  note={arXiv:2502.17540}
}
```

- arXiv: 2502.17540