Eur Lex Sum Eval

---
name: eur-lex-sum-eval
description: This benchmark evaluates long-form, multi- and cross-lingual summarization capabilities in the legal domain. It probes a model's ability to extract or generate concise summaries from lengthy, structurally complex EU legal documents across 24 official EU languages, including cross-lingual transfer scenarios. Use when the user wants to benchmark on EUR-Lex-Sum, or asks about evaluating this task. Reports ROUGE-1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2210.13448
  bibtex_key: aumiller2022eurlexsum
  confidence: high
---

# eur-lex-sum-eval

> EUR-Lex-Sum: A Multi- and Cross-lingual Dataset for Long-form Summarization in the Legal Domain — Aumiller et al. (2022) (arXiv:2210.13448, 2022)

## What this evaluates

This benchmark evaluates long-form, multi- and cross-lingual summarization capabilities in the legal domain. It probes a model's ability to extract or generate concise summaries from lengthy, structurally complex EU legal documents across 24 official EU languages, including cross-lingual transfer scenarios.

## Datasets

- **EUR-Lex-Sum** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/achouhan93/eur-lex-sum

## Metrics

- `ROUGE-1` **(primary)** — range: percent
  - Unigram overlap F1 score between generated and reference summaries. Computed with stemming disabled to preserve cross-lingual comparability.
- `ROUGE-2` — range: percent
  - Bigram overlap F1 score between generated and reference summaries. Stemming is disabled.
- `ROUGE-L` — range: percent
  - Longest common subsequence F1 score between generated and reference summaries. Stemming is disabled.

## Input / output format

**Input**: Long-form legal document text (chunked by paragraph separators if exceeding model context limits) in a target language.

**Output**: Extractive summary composed of selected paragraphs, or abstractive summary text.

## Scoring recipe

```python
def compute_rouge(pred, ref, rouge_type='rouge1', stem=False):
    p_tokens = tokenize(pred, stem=False)
    r_tokens = tokenize(ref, stem=False)
    if rouge_type == 'rouge1':
        return f1_score(ngrams(p_tokens, 1), ngrams(r_tokens, 1))
    elif rouge_type == 'rouge2':
        return f1_score(ngrams(p_tokens, 2), ngrams(r_tokens, 2))
    elif rouge_type == 'rougel':
        return f1_score(longest_common_subsequence(p_tokens, r_tokens))
```

## Common pitfalls

- Summary lengths often exceed standard transformer context windows (512 or 4096 tokens), requiring paragraph-level chunking that can disrupt coherence if not handled carefully.
- Tokenizer and sentence-splitting availability varies significantly across the 24 EU languages, making fair cross-lingual evaluation difficult for underrepresented languages.
- ROUGE is computed with stemming disabled to ensure cross-lingual comparability, which may penalize morphologically rich languages differently than standard implementations.

## Evidence (verbatim from paper)

> For evaluation, we rely on ROUGE scores (Lin, 2004) with disabled stemming to conserve comparability between languages. We acknowledge that this is not a comprehensive measure and has distinctive shortcomings, but works fairly well at the paragraph level, as such units generally preserve both factual consistency and fluency.

## Citation

```bibtex
@misc{aumiller2022eurlexsum,
  title={EUR-Lex-Sum: A Multi- and Cross-lingual Dataset for Long-form Summarization in the Legal Domain},
  author={Aumiller et al. (2022)},
  year={2022},
  note={arXiv:2210.13448}
}
```

- arXiv: 2210.13448