Flores101 Mt Eval

---
name: flores101-mt-eval
description: Evaluates the translation quality of Neural Machine Translation (NMT) models trained on filtered pseudo-parallel corpora. It measures how well few-shot Quality Estimation (QE) based corpus filtering improves MT performance across low-resource and mid-resource language pairs compared to baselines and other filtering methods. Use when the user wants to benchmark on FLORES101, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.03507
  bibtex_key: batheja2023little
  confidence: high
---

# flores101-mt-eval

> "A Little is Enough": Few-Shot Quality Estimation based Corpus Filtering improves Machine Translation — Batheja et al. (2023) (arXiv:2306.03507, 2023)

## What this evaluates

Evaluates the translation quality of Neural Machine Translation (NMT) models trained on filtered pseudo-parallel corpora. It measures how well few-shot Quality Estimation (QE) based corpus filtering improves MT performance across low-resource and mid-resource language pairs compared to baselines and other filtering methods.

## Datasets

- **FLORES101** — total 1012; splits: test (1012)

## Metrics

- `BLEU` **(primary)** — range: percent
  - Standard sentence-level BLEU score computed over the FLORES101 test set. It measures n-gram precision with a brevity penalty to penalize overly short translations.
- `Pearson Correlation` — range: [-1, 1]
  - Pairwise Pearson correlation coefficient computed between human-annotated Domain Adaptation (DA) quality scores and model-predicted quality scores on the QE test set.

## Input / output format

**Input**: Source sentence in the source language (e.g., English, Marathi, Chinese, Hindi, or Bengali).

**Output**: Translated sentence in the target language.

## Scoring recipe

```python
def compute_bleu(predictions, references):
    # predictions: list of translated strings
    # references: list of reference strings (one per prediction)
    bleu_score = sacrebleu.corpus_bleu(predictions, [references])
    return bleu_score.score
```

## Common pitfalls

- BLEU scores are reported separately for each translation direction (e.g., En→Mr vs Mr→En) and should not be averaged without explicit instruction.
- The FLORES101 test set contains exactly 1,012 sentence pairs per language pair, not the full 101 languages.
- QE filtering thresholds differ by language pair (-0.5 for En-Mr, -0.4 for Zh-En, 0 for Hi-Bn), so filtered corpus sizes vary significantly across methods and are not directly comparable.

## Evidence (verbatim from paper)

> For evaluation, we use the FLORES 101 test set which contains 1,012 sentence pairs for each language pair. Table 3: BLEU scores of En→Mr and Mr→En NMT models on FLORES101 test data.

## Citation

```bibtex
@misc{batheja2023little,
  title={"A Little is Enough": Few-Shot Quality Estimation based Corpus Filtering improves Machine Translation},
  author={Batheja et al. (2023)},
  year={2023},
  note={arXiv:2306.03507}
}
```

- arXiv: 2306.03507