Wmt2024 Discourse Literary Eval

---
name: wmt2024-discourse-literary-eval
description: Evaluates document-level and literary-domain machine translation quality, focusing on discourse-aware metrics like consistency, anaphora resolution, and term consistency across long-form Chinese web novels. The protocol combines automated n-gram and neural metrics with a structured human judgment framework to capture cross-sentence coherence and literary style. Use when the user wants to benchmark on WMT 2024 Discourse-Level Literary Translation Shared Task, or asks about evaluating this task. Reports d-BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.11732
  bibtex_key: wang2024wmt2024discourse
  confidence: high
---

# wmt2024-discourse-literary-eval

> Findings of the WMT 2024 Shared Task on Discourse-Level Literary Translation — Wang et al. (2024) (arXiv:2412.11732, 2024)

## What this evaluates

Evaluates document-level and literary-domain machine translation quality, focusing on discourse-aware metrics like consistency, anaphora resolution, and term consistency across long-form Chinese web novels. The protocol combines automated n-gram and neural metrics with a structured human judgment framework to capture cross-sentence coherence and literary style.

## Datasets

- **WMT 2024 Discourse-Level Literary Translation Shared Task** — total ?; splits: test (-1); repo https://github.com/longyuewangdcu/GuoFeng-Webnovel

## Metrics

- `d-BLEU` **(primary)** — range: [0, 100]
  - Document-level sacreBLEU calculated by concatenating all sentences in a document into a single line and computing n-gram matches case-sensitively using sacrebleu tool with signature nrefs:2|case:mixed|eff:no|tok:13a|smooth:exp|version:2.3.1.
- `Human Quality Score` — range: [0, 5]
  - 0–5 scale assessing general quality (fluency, adequacy) and discourse-aware quality (consistency, word choice, anaphora). Scores are averaged across two professional evaluators.

## Input / output format

**Input**: Source Chinese document (concatenated sentences for automated metrics) or window of neighboring sentences with document context for human evaluation.

**Output**: Translated target language document (German, Russian, or English).

## Scoring recipe

```python
def score_d_bleu(hypothesis_sentences, reference_sentences):
    doc_hyp = ' '.join(hypothesis_sentences)
    doc_ref = ' '.join(reference_sentences)
    return sacrebleu.corpus_bleu(doc_hyp, [doc_ref], tokenize='13a', case_mixed=True).score

def score_human(hyp_text, ref_text, context):
    # Evaluators assign 0-5 based on rubric for general & discourse quality
    score_1 = evaluate(hyp_text, ref_text, context, evaluator=1)
    score_2 = evaluate(hyp_text, ref_text, context, evaluator=2)
    return (score_1 + score_2) / 2
```

## Common pitfalls

- d-BLEU requires concatenating all sentences into a single line before scoring, unlike standard sentence-level BLEU.
- Human evaluation uses a 0-5 scale for two distinct dimensions (general quality and discourse-aware quality), not a single MQM score.
- Automatic metrics show significant discrepancies with human judgments, especially for discourse properties like anaphora and consistency.

## Evidence (verbatim from paper)

> For document-level evaluation, we employ document-level sacreBLEU (d-BLEU) Liu et al. ([2020]), which assesses n-gram matches across complete documents. This involves concatenating all sentences in a document into a single line before applying the sacreBLEU metric. Evaluations are conducted in a case-sensitive manner. We establish two sets of evaluation criteria: 1) general quality, covering aspects such as fluency and adequacy; 2) discourse-aware quality, including factors such as consistency, word choice, and anaphora. The detailed scoring criteria are listed in Table[6]. Accordingly, each output will be assigned two distinct scores (0∼5).

## Citation

```bibtex
@misc{wang2024wmt2024discourse,
  title={Findings of the WMT 2024 Shared Task on Discourse-Level Literary Translation},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2412.11732}
}
```

- arXiv: 2412.11732