Domainsum Eval

---
name: domainsum-eval
description: Evaluates abstractive text summarization models under varying degrees of domain shift (genre, style, topic) to measure performance degradation and distributional divergence across hierarchical granularity levels. Use when the user wants to benchmark on DomainSum, or asks about evaluating this task. Reports ROUGE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.15687
  bibtex_key: yuan2024domainsum
  confidence: high
---

# domainsum-eval

> DomainSum: A Hierarchical Benchmark for Fine-Grained Domain Shift in Abstractive Text Summarization — Yuan et al. (2024) (arXiv:2410.15687, 2024)

## What this evaluates

Evaluates abstractive text summarization models under varying degrees of domain shift (genre, style, topic) to measure performance degradation and distributional divergence across hierarchical granularity levels.

## Datasets

- **DomainSum** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/hpzhang94/DomainSum

## Metrics

- `ROUGE` **(primary)** — range: percent
  - Measures n-gram overlap between generated and reference summaries. Typically computed for ROUGE-1, ROUGE-2, and ROUGE-L.
- `BERTScore` — range: [0, 1]
  - Computes cosine similarity between contextual embeddings of generated and reference tokens, aggregated across precision, recall, and F1.

## Input / output format

**Input**: Source document text; optionally, two-shot prompting examples (document-summary pairs).

**Output**: Generated summary text.

## Scoring recipe

```python
def compute_metrics(predictions, references):
    rouge_scores = rouge.compute(predictions=predictions, references=references)
    bert_scores = bertscore.compute(predictions=predictions, references=references)
    return {'ROUGE': rouge_scores, 'BERTScore': bert_scores}
```

## Common pitfalls

- Performance varies significantly between zero-shot and two-shot prompting, with two-shot sometimes degrading performance on certain domains.
- Domain shift granularity (genre vs. style vs. topic) affects metrics non-uniformly; topic shifts show the most consistent distributional change but not necessarily the worst ROUGE scores.
- Models may overfit to in-domain data, making cross-domain evaluation sensitive to training data composition.

## Evidence (verbatim from paper)

> ROUGE | BERTScore | ROUGE | BERTScore | ROUGE | BERTScore | ROUGE | BERTScore | ROUGE | BERTScore | Table 2: Results for zero-shot and two-shot prompting across genre, style, and topic shift levels in DomainSum. Values in parentheses indicate the performance difference between two-shot and zero-shot prompting. The best results for zero-shot prompting are highlighted in red, two-shot in blue, and cases where both achieve the best results are highlighted in purple.

## Citation

```bibtex
@misc{yuan2024domainsum,
  title={DomainSum: A Hierarchical Benchmark for Fine-Grained Domain Shift in Abstractive Text Summarization},
  author={Yuan et al. (2024)},
  year={2024},
  note={arXiv:2410.15687}
}
```

- arXiv: 2410.15687