Phomt Eval

---
name: phomt-eval
description: This benchmark evaluates Vietnamese-English machine translation quality by comparing neural baselines and commercial engines. It probes translation accuracy across multiple domains and sentence lengths using both automatic metrics and human preference judgments. Use when the user wants to benchmark on PhoMT, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2110.12199
  bibtex_key: doan2021phomt
  confidence: high
---

# phomt-eval

> PhoMT: A High-Quality and Large-Scale Benchmark Dataset for Vietnamese-English Machine Translation — Doan et al. (2021) (arXiv:2110.12199, 2021)

## What this evaluates

This benchmark evaluates Vietnamese-English machine translation quality by comparing neural baselines and commercial engines. It probes translation accuracy across multiple domains and sentence lengths using both automatic metrics and human preference judgments.

## Datasets

- **PhoMT** — total 3020000; splits: val (-1), test (-1); repo https://github.com/VinAIResearch/PhoMT

## Metrics

- `BLEU` **(primary)** — range: [0, 100]
  - Case-sensitive BLEU score computed using SacreBLEU. Higher values indicate better translation quality.
- `TER` — range: [0, 100]
  - Translation Edit Rate (TER). Lower values indicate better performance.

## Input / output format

**Input**: Source sentence in Vietnamese or English.

**Output**: Translated sentence in the target language (English or Vietnamese).

## Scoring recipe

```python
def compute_automatic_metrics(predictions, references):
    bleu = sacrebleu.corpus_bleu(predictions, [references])
    ter = tercom(predictions, references)  # or equivalent TER implementation
    return {'BLEU': bleu.score, 'TER': ter.score}

def compute_human_eval(predictions, references, annotators):
    # Sample 100 test sentences, shuffle 5 system outputs per sentence
    # 3 annotators pick best per sentence
    votes = count_votes(annotators.choices)
    return {sys: votes[sys]/100 for sys in systems}
```

## Common pitfalls

- BLEU is computed case-sensitively using SacreBLEU, not standard tokenized BLEU.
- Human evaluation uses a 5-system shuffle protocol with 3 annotators per sentence, not a simple pairwise comparison.
- Automatic metrics (BLEU/TER) do not always correlate with human preference, especially for commercial engines vs. neural models.

## Evidence (verbatim from paper)

> We report standard metrics TER (Snover et al., 2006) and BLEU (Papineni et al., 2002), in which lower TER and higher BLEU indicate better performances. We compute the case-sensitive BLEU score using SacreBLEU (Post, 2018).

## Citation

```bibtex
@misc{doan2021phomt,
  title={PhoMT: A High-Quality and Large-Scale Benchmark Dataset for Vietnamese-English Machine Translation},
  author={Doan et al. (2021)},
  year={2021},
  note={arXiv:2110.12199}
}
```

- arXiv: 2110.12199