Fair Summm Eval

---
name: fair-summm-eval
description: Evaluates the fairness of abstractive summarization by measuring distributional alignment across social attributes (e.g., sentiment, gender, party). It quantifies how well generated summaries preserve the proportion of diverse perspectives present in the source text, penalizing underrepresentation of minority viewpoints. Use when the user wants to benchmark on PERSPECTIVESUMM, or asks about evaluating this task. Reports Binary Unfair Rate (BUR).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2311.07884
  bibtex_key: zhang2023fairsumm
  confidence: high
---

# fair-summm-eval

> Fair Abstractive Summarization of Diverse Perspectives — Zhang et al. (2023) (arXiv:2311.07884, 2023)

## What this evaluates

Evaluates the fairness of abstractive summarization by measuring distributional alignment across social attributes (e.g., sentiment, gender, party). It quantifies how well generated summaries preserve the proportion of diverse perspectives present in the source text, penalizing underrepresentation of minority viewpoints.

## Datasets

- **PERSPECTIVESUMM** — total ?; splits: Claritin (-1), US Election (-1), Yelp (-1), Amazon (-1), SupremeCourt (-1), IQ2 (-1); repo https://github.com/psunlpgroup/FairSumm

## Metrics

- `Binary Unfair Rate (BUR)` **(primary)** — range: percent
  - Measures the proportion of generated summaries that underrepresent a target social attribute or perspective compared to the source distribution. Lower scores indicate better fairness.
- `Unfair Error Rate (UER)` — range: percent
  - Quantifies the magnitude of distributional deviation between the summary and source across social attributes. Lower scores indicate better alignment.

## Input / output format

**Input**: Source text (e.g., product reviews, tweets, political articles) with optional instruction prompts.

**Output**: Abstractive summary generated by the LLM.

## Scoring recipe

```python
def compute_bur_uer(source, summary, values, tau=0.8):
    p_x = count_distribution(source, values)
    p_y = count_distribution(summary, values)
    ratios = [p_y[v] / p_x[v] for v in values if p_x[v] > 0]
    unfair_count = sum(1 for r in ratios if r < 1 - tau)
    BUR = (unfair_count / len(source_samples)) * 100
    UER = mean(abs(ratios - 1)) * 100
    return BUR, UER
```

## Common pitfalls

- BUR and UER are reference-free metrics, so they do not require human-written reference summaries for evaluation.
- Lower scores are better for both BUR and UER (indicated by ↓ in tables).
- Metric values are sensitive to the softmax temperature used in the underlying scoring model (e.g., BARTScore); the paper aligns them using temperature=0.1.

## Evidence (verbatim from paper)

> proposing four reference-free metrics—Binary Unfair Rate (BUR), Unfair Error Rate (UER), and variants—to quantify underrepresentation of perspectives.

## Citation

```bibtex
@misc{zhang2023fairsumm,
  title={Fair Abstractive Summarization of Diverse Perspectives},
  author={Zhang et al. (2023)},
  year={2023},
  note={arXiv:2311.07884}
}
```

- arXiv: 2311.07884