Feedsum Eval

---
name: feedsum-eval
description: This benchmark evaluates how well LLM-generated feedback aligns with human preferences for text summarization, and tests whether preference learning (DPO) using multi-dimensional feedback improves summary quality over supervised fine-tuning. Use when the user wants to benchmark on FeedSum, or asks about evaluating this task. Reports Spearman correlation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.13116
  bibtex_key: song2024learning
  confidence: high
---

# feedsum-eval

> Learning to Summarize from LLM-generated Feedback — Song et al. (2024) (arXiv:2410.13116, 2024)

## What this evaluates

This benchmark evaluates how well LLM-generated feedback aligns with human preferences for text summarization, and tests whether preference learning (DPO) using multi-dimensional feedback improves summary quality over supervised fine-tuning.

## Datasets

- **FeedSum** — total 125000; splits: train (125000)

## Metrics

- `Spearman correlation` **(primary)** — range: [-1, 1]
  - Computes the rank correlation coefficient between the LLM-generated feedback scores (composite or single-dimension) and human-assigned composite scores. Values closer to 1 indicate stronger alignment with human judgment.

## Input / output format

**Input**: Document text paired with a candidate summary. For preference learning, inputs are formatted as (document, chosen_summary) vs (document, rejected_summary) pairs.

**Output**: For feedback generation: JSON-like scores (e.g., {Faithfulness: 5, Completeness: 3, Conciseness: 3} or percentage scores). For summarization models: a generated summary text.

## Scoring recipe

```python
def compute_spearman(llm_scores, human_scores):
    rank_llm = rankdata(llm_scores)
    rank_human = rankdata(human_scores)
    n = len(rank_llm)
    d_sq = sum((r1 - r2)**2 for r1, r2 in zip(rank_llm, rank_human))
    return 1 - (6 * d_sq) / (n * (n**2 - 1))
```

## Common pitfalls

- The composite score used for DPO pair selection is the average across dimensions, not a sum or max.
- The 125K dataset size is a filtered subset of 182K generated pairs; documents exceeding 8K tokens or yielding malformed LLM feedback are excluded.
- Feedback configurations C1–C4 differ in LLM capacity, dimensionality, and granularity, which must be matched to the correct training setup.

## Evidence (verbatim from paper)

> assessed based on their Spearman correlation with human composite scores in UniSumEval... A summary is selected as the "chosen" one if it scores ≥ 4 on the Likert scale or ≥ 80% in percentage scores... Conversely, a summary is considered "rejected" if its score is at least 1 point lower on the Likert scale or 20 percentage points lower than the chosen one.

## Citation

```bibtex
@misc{song2024learning,
  title={Learning to Summarize from LLM-generated Feedback},
  author={Song et al. (2024)},
  year={2024},
  note={arXiv:2410.13116}
}
```

- arXiv: 2410.13116