Xwikis Summarisation Eval

---
name: xwikis-summarisation-eval
description: This benchmark evaluates the ability of multilingual and cross-lingual models to generate accurate English summaries from source documents in German, French, and Czech. It probes supervised, zero-shot, and few-shot cross-lingual transfer capabilities, as well as model robustness on out-of-domain news text. Use when the user wants to benchmark on XWikis, D_en→en, Voxeurop, or asks about evaluating this task. Reports ROUGE-L recall.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2202.09583
  bibtex_key: [REDACTED]
  confidence: high
---

# xwikis-summarisation-eval

> Models and Datasets for Cross-Lingual Summarisation — Perez-Beltrachini et al. (2022) (arXiv:2202.09583, 2022)

## What this evaluates

This benchmark evaluates the ability of multilingual and cross-lingual models to generate accurate English summaries from source documents in German, French, and Czech. It probes supervised, zero-shot, and few-shot cross-lingual transfer capabilities, as well as model robustness on out-of-domain news text.

## Datasets

- **XWikis** — total ?; splits: train (-1), val (-1), test (-1)
- **D_en→en** — total 300000; splits: train (270000), val (15000), test (15000)
- **Voxeurop** — total 2666; splits: test (2000), adaptation (666)

## Metrics

- `ROUGE-L recall` **(primary)** — range: percent
  - Standard ROUGE-L recall computed over the generated summary against the reference summary, measuring the recall of the longest common subsequence. Values in the paper's tables are reported as percentages (0-100).

## Input / output format

**Input**: Source document text in a source language (German, French, Czech, or English), optionally truncated or extracted to a 600-token budget using paragraph-based LEXRANK ranking to handle long documents.

**Output**: A generated English summary text corresponding to the input document.

## Scoring recipe

```python
def compute_rouge_l_recall(predictions, references):
    scores = []
    for pred, ref in zip(predictions, references):
        lcs_len = longest_common_subsequence_length(pred, ref)
        recall = lcs_len / len(ref) if len(ref) > 0 else 0.0
        scores.append(recall * 100)
    return sum(scores) / len(scores)
```

## Common pitfalls

- Evaluating the extractive paragraph selection step using the cross-lingual reference instead of the monolingual reference summary, which the authors explicitly warn against.
- Confusing the XWikis-comparable split (used for training/validation) with the XWikis-parallel split (used for final evaluation).
- Ignoring the 600-token input budget constraint, which significantly alters the effective input length compared to full Wikipedia articles and impacts model performance.

## Evidence (verbatim from paper)

> Table 5 reports ROUGE-L recall of the input against the reference summary (note that to measure this we take the monolingual summary associated with the document rather than the cross-lingual one). As can be seen, the extractive step reduces the document to a manageable size without sacrificing too much content.

## Citation

```bibtex
@misc{perezbeltrachini2022models,
  title={Models and Datasets for Cross-Lingual Summarisation},
  author={Perez-Beltrachini et al. (2022)},
  year={2022},
  note={arXiv:2202.09583}
}
```

- arXiv: 2202.09583