Survey Sum Eval

---
name: survey-sum-eval
description: Evaluates a modular retrieval-augmented generation pipeline for multi-document scientific literature summarization. It probes the model's ability to dynamically generate queries, retrieve relevant papers, and synthesize citation-aware, coherent summaries from multiple sources. Use when the user wants to benchmark on SurveySum, or asks about evaluating this task. Reports Ref-F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.16349
  bibtex_key: achkar2025askretrievesummarize
  confidence: high
---

# survey-sum-eval

> Ask, Retrieve, Summarize: A Modular Pipeline for Scientific Literature Summarization — Achkar et al. (2025) (arXiv:2505.16349, 2025)

## What this evaluates

Evaluates a modular retrieval-augmented generation pipeline for multi-document scientific literature summarization. It probes the model's ability to dynamically generate queries, retrieve relevant papers, and synthesize citation-aware, coherent summaries from multiple sources.

## Datasets

- **SurveySum** — total 79; splits: test (79); repo https://github.com/unicamp-dl/surveysum

## Metrics

- `ROUGE-1` — range: [0, 1]
  - Recall-oriented n-gram overlap measuring unigram matching between generated and reference summaries.
- `ROUGE-2` — range: [0, 1]
  - Recall-oriented n-gram overlap measuring bigram matching between generated and reference summaries.
- `ROUGE-L` — range: [0, 1]
  - Recall-oriented longest common subsequence matching at the sentence level.
- `BERTScore` — range: [0, 1]
  - Semantic similarity computed using contextual embeddings from PLMs (e.g., BERT) between generated and reference texts.
- `Ref-F1` **(primary)** — range: [0, 1]
  - F1 score combining precision (proportion of correctly included references in the generated summary) and recall (proportion of ground-truth references captured).
- `G-Eval` — range: [1, 5]
  - Reference-free LLM-based evaluation using Chain-of-Thought reasoning to score criteria like coherence, coverage, fluency, and relevance on a 1-5 scale.
- `CheckEval` — range: [0, 1]
  - LLM-based checklist evaluation where the final score is the proportion of positive (yes) responses to structured sub-aspect questions.

## Input / output format

**Input**: A collection of full-text scientific papers (average 7.38 per instance) retrieved via dynamic question generation, intended for multi-document summarization.

**Output**: A synthesized multi-document summary with inline citations aligned to academic standards and ground truth references.

## Scoring recipe

```python
def compute_ref_f1(pred_summary, gold_summary):
    pred_refs = extract_citations(pred_summary)
    gold_refs = extract_citations(gold_summary)
    tp = len(pred_refs.intersection(gold_refs))
    precision = tp / len(pred_refs) if pred_refs else 0.0
    recall = tp / len(gold_refs) if gold_refs else 0.0
    f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0.0
    return f1
```

## Common pitfalls

- ROUGE-2 scores are inherently low for abstractive scientific summarization because pipelines prioritize semantic richness and paraphrasing over strict lexical bigram matching.
- LLM-based metrics (G-Eval, CheckEval) are sensitive to evaluator model choice and configuration, which can impact reproducibility and consistency.
- Using identical models for generation and evaluation introduces egocentric bias; the protocol explicitly separates them (GPT-4o-mini for generation, Phi-3-small for evaluation).

## Evidence (verbatim from paper)

> The evaluation employs a mix of traditional and LLM-based metrics to assess the quality of summaries in terms of content coverage, coherence, and citation alignment: ROUGE (Recall-Oriented Understudy for Gisting Evaluation) measures the overlap between the generated summaries and the reference text. It calculates n-gram overlap, word sequence matching, and the longest common subsequences. Reference F1 Score (Ref-F1) measures how accurately the citations in the generated summaries align with those in the ground truth. It computes precision (proportion of correctly included references) and recall (proportion of ground-truth references captured in the generated summary), and combines them into an F1 score.

## Citation

```bibtex
@misc{achkar2025askretrievesummarize,
  title={Ask, Retrieve, Summarize: A Modular Pipeline for Scientific Literature Summarization},
  author={Achkar et al. (2025)},
  year={2025},
  note={arXiv:2505.16349}
}
```

- arXiv: 2505.16349