Wmt2023 Discourse Literary Eval

---
name: wmt2023-discourse-literary-eval
description: Evaluates machine translation systems on discourse-level literary translation from Chinese to English, focusing on long-range context, cultural adaptation, and stylistic fidelity across entire web novels rather than isolated sentences. Use when the user wants to benchmark on WMT 2023 Discourse-Level Literary Translation Test Set, or asks about evaluating this task. Reports d-BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2311.03127
  bibtex_key: [REDACTED]
  confidence: high
---

# wmt2023-discourse-literary-eval

> Findings of the WMT 2023 Shared Task on Discourse-Level Literary Translation: A Fresh Orb in the Cosmos of LLMs — Wang et al. (2023) (arXiv:2311.03127, 2023)

## What this evaluates

Evaluates machine translation systems on discourse-level literary translation from Chinese to English, focusing on long-range context, cultural adaptation, and stylistic fidelity across entire web novels rather than isolated sentences.

## Datasets

- **WMT 2023 Discourse-Level Literary Translation Test Set** — total ?; splits: test (-1)

## Metrics

- `d-BLEU` **(primary)** — range: [0, 100]
  - Document-level BLEU calculated by concatenating all continuous sentences in a single book into one line, then scoring the concatenated hypothesis against the concatenated reference using sacreBLEU.
- `Human MQM Score` — range: [0, 100]
  - Literary-adapted Multidimensional Quality Metrics (MQM) framework scoring translation quality across Accuracy, Fluency, Style, Terminology, Localization, and Other error types, averaged across four annotators.

## Input / output format

**Input**: Chinese source text from web novels (provided as sentence-level segments for automatic metrics or full documents for d-BLEU).

**Output**: English translated text corresponding to the source segment/document.

## Scoring recipe

```python
# d-BLEU
doc_hyp = " ".join(hypothesis_sentences)
doc_ref = " ".join(reference_sentences)
score = sacreBLEU.corpus_score(doc_hyp, [doc_ref]).score
return score

# Human MQM
scores = [annotator.evaluate(doc_hyp, doc_ref) for annotator in annotators]
return sum(scores) / len(scores)
```

## Common pitfalls

- Automatic metrics like BLEU and COMET fail to capture literary style, cultural nuance, and discourse coherence, leading to poor correlation with human judgment.
- d-BLEU requires concatenating entire books into single lines, which differs from standard sentence-level evaluation and may penalize formatting or paragraph breaks.
- Human evaluation relies on a small sampled subset (5 documents, ~2,194 words) for ranking, which may not fully represent system performance on the broader test set.

## Evidence (verbatim from paper)

> The evaluation metrics includes 1) sentence-level BLEU,chrF,COMET, TER; and 2) document-level d-BLEU. To calculate d-BLEU, we first concatenate all continuous sentences in one book as on line, and then employ sacreBLEU to obtain scorers.

## Citation

```bibtex
@misc{wang2023wmt2023discourse,
  title={Findings of the WMT 2023 Shared Task on Discourse-Level Literary Translation: A Fresh Orb in the Cosmos of LLMs},
  author={Wang et al. (2023)},
  year={2023},
  note={arXiv:2311.03127}
}
```

- arXiv: 2311.03127