Commt Mt Eval

---
name: commt-mt-eval
description: Evaluates machine translation quality across multiple language pairs and specialized tasks (general translation, terminology-constrained, and automatic post-editing). It measures how well encoder-decoder and decoder-only models generate accurate and fluent target sentences. Use when the user wants to benchmark on ComMT, or asks about evaluating this task. Reports SacreBLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.06594
  bibtex_key: luo2025beyond
  confidence: high
---

# commt-mt-eval

> Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation — Yingfeng Luo et al. (arXiv:2503.06594, 2025)

## What this evaluates

Evaluates machine translation quality across multiple language pairs and specialized tasks (general translation, terminology-constrained, and automatic post-editing). It measures how well encoder-decoder and decoder-only models generate accurate and fluent target sentences.

## Datasets

- **ComMT** — total ?; splits: test (-1)

## Metrics

- `SacreBLEU` **(primary)** — range: percent
  - Standard BLEU score computed using the SacreBLEU toolkit, typically with case-sensitive tokenization and no smoothing by default.
- `COMET (wmt22-comet-da)` — range: [0, 1]
  - Reference-free neural machine translation quality estimation model trained on WMT22 data, outputting a continuous score between 0 and 1.
- `Terminology Success Rate (TSR)` — range: percent
  - Percentage of required domain-specific terms that are correctly translated in the model output.
- `Human Translation Edit Rate (HTER)` — range: percent
  - Measures the amount of editing operations required to align a machine translation with a human reference translation, expressed as a percentage.

## Input / output format

**Input**: Source language sentence (English, German, Czech, Russian, or Chinese).

**Output**: Target language translation sentence.

## Scoring recipe

```python
def compute_mt_metrics(predictions, references, sources=None):
    # SacreBLEU
    bleu = sacrebleu.corpus_bleu(predictions, [references])
    # COMET (requires source, prediction, reference)
    comet_scores = comet_model.predict(predictions, references, sources)
    # TSR & HTER require external terminology lists and edit logs
    return {
        'sacrebleu': bleu.score,
        'comet': float(comet_scores.mean())
    }
```

## Common pitfalls

- SacreBLEU scores are highly sensitive to tokenization and case-sensitivity conventions; evaluators must use the exact SacreBLEU flags specified in the paper.
- COMET (wmt22-comet-da) requires the source sentence alongside predictions and references; omitting the source will cause evaluation failure or invalid scores.
- TSR and HTER require strict alignment between predicted terms and reference terminology; minor spelling variations or tokenization mismatches can drastically lower TSR.

## Evidence (verbatim from paper)

> We conducted tests on ComMT and evaluated the model’s translation performance across all tasks using COMET (wmt22-comet-da) (Rei et al., [2020]) and SacreBLEU (Post, [2018]). Additionally, we used Terminology Success Rate (TSR) for the terminology-constrained translation task and Human Translation Edit Rate (HTER) for the automatic post-editing task.

## Citation

```bibtex
@misc{luo2025beyond,
  title={Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation},
  author={Yingfeng Luo et al.},
  year={2025},
  note={arXiv:2503.06594}
}
```

- arXiv: 2503.06594