Opus Mt Eval

---
name: opus-mt-eval
description: Evaluates the translation quality of OPUS-MT models across diverse language pairs using standard automatic metrics and specialized linguistic test suites. It probes general-purpose translation capability, lexical ambiguity disambiguation, and cross-lingual robustness. Use when the user wants to benchmark on Flores, Tatoeba, MuCoW, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2212.01936
  bibtex_key: [REDACTED]
  confidence: high
---

# opus-mt-eval

> Democratizing Neural Machine Translation with OPUS-MT — Tiedemann et al. (2022) (arXiv:2212.01936, 2022)

## What this evaluates

Evaluates the translation quality of OPUS-MT models across diverse language pairs using standard automatic metrics and specialized linguistic test suites. It probes general-purpose translation capability, lexical ambiguity disambiguation, and cross-lingual robustness.

## Datasets

- **Flores** — total ?; splits: test (-1)
- **Tatoeba** — total ?; splits: test (-1)
- **MuCoW** — total ?; splits: test (-1)

## Metrics

- `BLEU` **(primary)** — range: [0, 100] percent
  - Standard n-gram overlap metric between predicted and reference translations. The paper also reports spBLEU (sentence-piece tokenized), chrF/chrf++ (character n-gram F-score), and COMET (cross-lingual neural metric trained on human judgments).
- `spBLEU` — range: [0, 100] percent
  - Sentence-piece tokenized BLEU score, used for more consistent cross-lingual comparison.
- `chrF++` — range: [0, 100] percent
  - Character n-gram F-score, noted as more reliable across languages than BLEU.
- `COMET` — range: [0, 1]
  - Cross-lingual neural evaluation metric trained on human judgments to better correlate with human quality assessments.

## Input / output format

**Input**: Source sentence in the source language

**Output**: Target sentence in the target language

## Scoring recipe

```python
def evaluate(predictions, references, metric='BLEU'):
    scores = []
    for pred, ref in zip(predictions, references):
        if metric == 'BLEU':
            scores.append(calculate_bleu(pred, ref))
        elif metric == 'spBLEU':
            scores.append(calculate_spbleu(pred, ref))
        elif metric == 'chrF++':
            scores.append(calculate_chrf(pred, ref))
        elif metric == 'COMET':
            scores.append(comet_model.predict(pred, ref))
    return sum(scores) / len(scores) if scores else 0
```

## Common pitfalls

- BLEU scores are problematic to compare across languages; chrF is preferred for reliability.
- Models are general-purpose and not directly comparable to WMT shared task systems due to different training data and domain tuning.
- MuCoW only evaluates lexical ambiguity for nouns and does not cover broader linguistic phenomena or fluency.

## Evidence (verbatim from paper)

> Currently, we support BLEU [33], spBLEU [32], chrF [34], chrf++ [35] and COMET [36] scores but other measures may be added once they become available from our systematic test procedures.

## Citation

```bibtex
@misc{tiedemann2022opusmt,
  title={Democratizing Neural Machine Translation with OPUS-MT},
  author={Tiedemann et al. (2022)},
  year={2022},
  note={arXiv:2212.01936}
}
```

- arXiv: 2212.01936