Opinion Summarization Eval

---
name: opinion-summarization-eval
description: Evaluates a model's ability to synthesize multiple product reviews into a single, coherent opinion summary. It probes the model's capacity to capture diverse aspects, maintain factual accuracy, and adhere to domain-specific stylistic constraints without relying on surface-level lexical overlap. Use when the user wants to benchmark on Amazon, Oposum+, Flipkart, or asks about evaluating this task. Reports human evaluation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.15473
  bibtex_key: nath2024leveraging
  confidence: high
---

# opinion-summarization-eval

> Leveraging Domain Knowledge for Efficient Reward Modelling in RLHF: A Case-Study in E-Commerce Opinion Summarization — Nath et al. (2024) (arXiv:2402.15473, 2024)

## What this evaluates

Evaluates a model's ability to synthesize multiple product reviews into a single, coherent opinion summary. It probes the model's capacity to capture diverse aspects, maintain factual accuracy, and adhere to domain-specific stylistic constraints without relying on surface-level lexical overlap.

## Datasets

- **Amazon** — total ?; splits: test (-1)
- **Oposum+** — total ?; splits: test (-1)
- **Flipkart** — total ?; splits: test (-1)

## Metrics

- `human evaluation` **(primary)** — range: other
  - Human domain experts rate the quality of generated opinion summaries given a set of product reviews. The authors explicitly state that overlap-based metrics like ROUGE are inadequate for this task due to first-person bias and aspect coverage issues, making human assessment the primary evaluation method.

## Input / output format

**Input**: A set of product reviews (typically 8-10 per product) from a specific e-commerce domain.

**Output**: A single opinion summary that synthesizes the opinions across the provided reviews.

## Scoring recipe

```python
def score_human_eval(predictions, gold, annotators):
    scores = []
    for pred in predictions:
        # Annotators (domain experts) review input + generated summary
        rating = annotators.rate_quality(pred)
        scores.append(rating)
    return mean(scores)
```

## Common pitfalls

- Over-reliance on ROUGE or other overlap-based metrics, which the authors explicitly note fail to capture first-person bias and comprehensive aspect coverage in opinion summaries.
- Small benchmark scale (32-147 products total), which limits statistical power and may not generalize across broader e-commerce categories.
- Human evaluation relies on domain experts rather than crowd workers, which can introduce specific inductive biases or domain-specific rating thresholds.

## Evidence (verbatim from paper)

> Due to these shortcomings, in our analysis, we do not rely much on overlap-based evaluations, such as Rouge. Rather we rely on human evaluations.

## Citation

```bibtex
@misc{nath2024leveraging,
  title={Leveraging Domain Knowledge for Efficient Reward Modelling in RLHF: A Case-Study in E-Commerce Opinion Summarization},
  author={Nath et al. (2024)},
  year={2024},
  note={arXiv:2402.15473}
}
```

- arXiv: 2402.15473