Deepl Supertext Comparison Eval

---
name: deepl-supertext-comparison-eval
description: This evaluation probes the translation quality and contextual consistency of two commercial machine translation systems (DeepL and Supertext) by having professional raters perform blind pairwise comparisons on full documents. It specifically measures whether LLM-based long-context translation yields superior document-level coherence compared to traditional segment-level systems. Use when the user wants to benchmark on Unspecified source documents, or asks about evaluating this task. Reports pairwise preference rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.02577
  bibtex_key: fluckiger2025deeplsupertext
  confidence: high
---

# deepl-supertext-comparison-eval

> A comparison of translation performance between DeepL and Supertext — Flückiger et al. (2025) (arXiv:2502.02577, 2025)

## What this evaluates

This evaluation probes the translation quality and contextual consistency of two commercial machine translation systems (DeepL and Supertext) by having professional raters perform blind pairwise comparisons on full documents. It specifically measures whether LLM-based long-context translation yields superior document-level coherence compared to traditional segment-level systems.

## Datasets

- **Unspecified source documents** — total ?; splits: test (-1); repo https://github.com/supertext/evaluation_deepl_supertext

## Metrics

- `pairwise preference rate` **(primary)** — range: [0, 1]
  - Calculated as the proportion of segments where a given system's translation is chosen as 'better' out of all non-equal judgments. Equal judgments are excluded from the denominator to compute a strict win rate.

## Input / output format

**Input**: Full source document with segments in original order, including formatting such as newlines. Side-by-side translations labeled Translation A and Translation B for each segment.

**Output**: Per segment, a categorical choice: 'Translation A is better', 'Translation B is better', or 'equal quality'.

## Scoring recipe

```python
def compute_preference_rate(wins_a, wins_b, ties):
    decisive = wins_a + wins_b
    if decisive == 0:
        return 0.0
    rate_a = wins_a / decisive
    rate_b = wins_b / decisive
    return rate_a, rate_b
```

## Common pitfalls

- Evaluating segments in isolation rather than using the full document context as instructed.
- Allowing punctuation or formatting differences to bias the preference judgment instead of focusing on content.
- Misinterpreting the 'equal' option as a tie-breaker rather than a valid third category that should be excluded in win-rate calculations.

## Evidence (verbatim from paper)

> For each segment in each document, the assigned rater is asked to choose whether Translation A is better, Translation B is better, or whether both translations are of equal quality.

## Citation

```bibtex
@misc{fluckiger2025deeplsupertext,
  title={A comparison of translation performance between DeepL and Supertext},
  author={Flückiger et al. (2025)},
  year={2025},
  note={arXiv:2502.02577}
}
```

- arXiv: 2502.02577