Orangesum Eval

---
name: orangesum-eval
description: Evaluates abstractive French summarization quality by measuring lexical overlap, semantic similarity, and human judgments of accuracy, informativeness, and fluency. Use when the user wants to benchmark on OrangeSum, or asks about evaluating this task. Reports ROUGE-L.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2010.12321
  bibtex_key: moussa2020barthez
  confidence: high
---

# orangesum-eval

> BARThez: a Skilled Pretrained French Sequence-to-Sequence Model — Moussa Kamal Eddine et al. (2020) (arXiv:2010.12321, 2020)

## What this evaluates

Evaluates abstractive French summarization quality by measuring lexical overlap, semantic similarity, and human judgments of accuracy, informativeness, and fluency.

## Datasets

- **OrangeSum** — total ?; splits: abstract (-1), title (-1)

## Metrics

- `ROUGE-L` **(primary)** — range: [0, 1]
  - F-measure based on the longest common subsequence between generated and reference summaries.
- `BERTScore` — range: [0, 1]
  - Cosine similarity between contextual embeddings of candidate and reference tokens, optionally rescaled.
- `Human Evaluation Score` — range: percent
  - Percentage of times a model's summary was chosen as best minus percentage chosen as worst in Best-Worst Scaling across accuracy, informativeness, and fluency.

## Input / output format

**Input**: French news document text

**Output**: Generated summary text (abstract or title length)

## Scoring recipe

```python
rouge_l = compute_rouge_l(generated, gold)
bertscore = compute_bertscore(generated, gold)
best_count = 0; worst_count = 0; total = 0
for doc, pair in summary_pairs:
    choice = human_judge(doc, pair.sys1, pair.sys2, ['accuracy', 'informativeness', 'fluency'])
    if choice == pair.sys1: best_count += 1
    elif choice == pair.sys2: worst_count += 1
    total += 1
human_score = (best_count / total) * 100 - (worst_count / total) * 100
```

## Common pitfalls

- ROUGE scores poorly correlate with abstractive quality and can be misleading for models that generate irrelevant novel words.
- Human evaluators may prefer model outputs over ground truth if references contain external information (e.g., dates) not present in the source document.
- BERTScore reports two values (with/without rescaling); must specify which convention is used.

## Evidence (verbatim from paper)

> We classically report ROUGE-1, ROUGE-2 and ROUGE-L scores (Lin, 2004) in Table 6. However, since ROUGE is limited to capturing n-gram overlap, which is poorly suited to the abstractive summarization setting, we also report BERTScore scores. BERTScore (Zhang et al., 2019) is a recently introduced metric that leverages the contextual representations of the candidate and gold sentences.

## Citation

```bibtex
@misc{moussa2020barthez,
  title={BARThez: a Skilled Pretrained French Sequence-to-Sequence Model},
  author={Moussa Kamal Eddine et al. (2020)},
  year={2020},
  note={arXiv:2010.12321}
}
```

- arXiv: 2010.12321