Wmt24 Docmt Eval

---
name: wmt24-docmt-eval
description: Evaluates document-level machine translation quality using multi-turn conversational prompting strategies with LLMs, measuring contextual coherence and translation accuracy across multiple language directions and domains. Use when the user wants to benchmark on WMT 24 General Track, WMT 23 Chinese-to-English, or asks about evaluating this task. Reports dBLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.10494
  bibtex_key: hu2025sourceprimed
  confidence: high
---

# wmt24-docmt-eval

> Source-primed Multi-turn Conversation Helps Large Language Models Translate Documents — Hu et al. (2025) (arXiv:2503.10494, 2025)

## What this evaluates

Evaluates document-level machine translation quality using multi-turn conversational prompting strategies with LLMs, measuring contextual coherence and translation accuracy across multiple language directions and domains.

## Datasets

- **WMT 24 General Track** — total ?; splits: test (-1); repo https://github.com/ZurichNLP/multiturn-llm-docmt
- **WMT 23 Chinese-to-English** — total ?; splits: test (-1); repo https://github.com/ZurichNLP/multiturn-llm-docmt

## Metrics

- `dBLEU` **(primary)** — range: [0, 100]
  - Document-level BLEU computed using sacreBLEU, considering n-gram matches across the entire translated document rather than per-segment.
- `COMET-22` — range: [0, 1]
  - Neural translation quality metric using the COMET-22 default model. Due to context length limits, each segment is evaluated independently and scores are averaged.
- `BlonDE` — range: [0, 1]
  - Document-level metric measuring correctness of context-dependent features such as tense, pronouns, transliteration, entities, and connectives.

## Input / output format

**Input**: Segmented source document paragraphs with prior conversation turns (context) and target language prompt. For the source-primed variant, the full source document is provided before translation begins.

**Output**: Translated target language paragraphs/documents.

## Scoring recipe

```python
# dBLEU (document-level)
dbleu = sacrebleu.corpus_bleu(predictions, [references], tokenize='none')

# COMET-22 (segment-level average)
comet_scores = []
for pred_seg, ref_seg in zip(predictions, references):
    comet_scores.append(comet22_model.predict(pred_seg, ref_seg))
avg_comet = sum(comet_scores) / len(comet_scores)

# BlonDE (document-level)
blonde_score = blonde_model.predict(doc_prediction, doc_reference)
```

## Common pitfalls

- COMET-22 is evaluated per segment and averaged, which may underestimate document-level coherence due to its context window limit.
- BLEU is computed at the document level (dBLEU), not segment level, altering n-gram statistics and making direct comparison with sentence-level BLEU invalid.
- In-context learning (ICL) with 3 exemplars is used as a baseline setting, which can significantly boost scores compared to zero-shot prompting.

## Evidence (verbatim from paper)

> We use the COMET-22 default model Rei et al. ([2022]) and sacreBLEU Post ([2018]) implementation of BLEU to evaluate translation quality. For COMET, due to its context length limit, we evaluate each segment independently and report average scores; for BLEU, we consider n-gram matches at the document level Liu et al. ([2020]). Additionally, we use BlonDE Jiang et al. ([2022]) to evaluate document-level translation, which specifically measures the correctness of features that are known to benefit from wider context in Chinese-to-English translation, such as tense correctness, pronouns, transliteration, entities, and connectives.

## Citation

```bibtex
@misc{hu2025sourceprimed,
  title={Source-primed Multi-turn Conversation Helps Large Language Models Translate Documents},
  author={Hu et al. (2025)},
  year={2025},
  note={arXiv:2503.10494}
}
```

- arXiv: 2503.10494