Wmt24 Chat Translation Eval

---
name: wmt24-chat-translation-eval
description: Evaluates machine translation systems for bilingual customer support conversations, focusing on context utilization, discourse coherence, and turn-level versus conversation-level translation quality across five language pairs. Use when the user wants to benchmark on MAIA 2.0, or asks about evaluating this task. Reports COMET.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.11624
  bibtex_key: mohammed2024wmt24chattranslation
  confidence: high
---

# wmt24-chat-translation-eval

> Findings of the WMT 2024 Shared Task on Chat Translation — Mohammed et al. (2024) (arXiv:2410.11624, 2024)

## What this evaluates

Evaluates machine translation systems for bilingual customer support conversations, focusing on context utilization, discourse coherence, and turn-level versus conversation-level translation quality across five language pairs.

## Datasets

- **MAIA 2.0** — total ?; splits: train (79400), dev (12300), test (10000)

## Metrics

- `COMET` **(primary)** — range: [-1, 1]
  - Neural reference-based metric trained on human judgments to predict translation quality.
- `BLEU` — range: [0, 1]
  - Standard n-gram overlap metric computed with SacreBLEU.
- `chrF` — range: [0, 1]
  - Character n-gram F-score computed with SacreBLEU.
- `ContextCometQE` — range: [-1, 1]
  - Reference-free metric that uses bilingual context (previous two turns) to assess translation quality.
- `MuDA F1` — range: [0, 1]
  - F1 accuracy on four context-dependent discourse phenomena tags: lexical cohesion, formality, pronoun resolution, and verb forms.
- `DA+SQM` — range: [0, 100]
  - Human assessment using Direct Assessment and Scalar Quality Metric on a continuous scale from 0 to 100 based on accuracy and grammatical correctness.
- `ContextMQM` — range: other
  - LLM-based fine-grained error score aggregating minor, major, and critical errors weighted by 1, 5, and 10 respectively.

## Input / output format

**Input**: Source text in the customer's or agent's language, accompanied by preceding conversation context (full context for human eval, up to 8 turns for LLM eval, or 2 turns for ContextCometQE).

**Output**: Translated text in the target language (agent's or customer's language).

## Scoring recipe

```python
def compute_metrics(predictions, gold, sources):
    comet_scores = comet_model.predict(hypotheses=predictions, references=gold, sources=sources)
    comet_val = comet_scores['scores'].mean()
    bleu_val = sacrebleu.corpus_bleu(predictions, [gold]).score / 100
    chrf_val = sacrebleu.corpus_chrf(predictions, [gold]).score / 100
    muda_f1 = f1_score(tag_discourse(predictions), tag_discourse(gold))
    human_val = sum(linguist_scores) / len(linguist_scores)
    mqm_val = sum(minor*1 + major*5 + critical*10 for errors in llm_mqm_outputs)
    return {'COMET': comet_val, 'BLEU': bleu_val, 'chrF': chrf_val, 'MuDA_F1': muda_f1, 'DA+SQM': human_val, 'ContextMQM': mqm_val}
```

## Common pitfalls

- Turn-level metrics (e.g., BLEU, COMET) often overestimate system capability because performance degrades significantly at the conversation level and in later turns.
- Standard MT metrics fail to capture discourse coherence; context-dependent phenomena like formality and pronoun resolution require specialized tagging or context-aware metrics.
- LLM-based ContextMQM evaluation was restricted to the en-de language pair due to budget constraints, limiting cross-lingual generalizability of fine-grained error analysis.

## Evidence (verbatim from paper)

> We use Comet Rei et al. ([2022]) as our primary evaluation metric for assessing translation quality of the submitted systems. Additionally, we report lexical metrics: BLEU and chrF using the SacreBLEU library Post ([2018]). We also include ContextCometQE (Agrawal et al., [2024]), a reference-free metric that uses bilingual context (previous two turns) to assess the translation quality of the current turn.

## Citation

```bibtex
@misc{mohammed2024wmt24chattranslation,
  title={Findings of the WMT 2024 Shared Task on Chat Translation},
  author={Mohammed et al. (2024)},
  year={2024},
  note={arXiv:2410.11624}
}
```

- arXiv: 2410.11624