Indic Nmt Eval

---
name: indic-nmt-eval
description: Evaluates neural machine translation models for Indic languages by measuring translation quality against reference texts across multiple standard benchmarks. It specifically tests the effectiveness of training on the Samanantar parallel corpus compared to commercial systems and existing open-source baselines. Use when the user wants to benchmark on WAT2020 Indic task, WAT2021 Multi-IndicMT task, WMT test sets (2014, 2019, 2020), UFAL Entam, FLORES test set, PMIndia en-as testset, or asks about evaluating this task. Reports BLEU (SacreBLEU).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2104.05596
  bibtex_key: ramesh2021samanantar
  confidence: high
---

# indic-nmt-eval

> Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages — Gowtham Ramesh et al. (2021) (arXiv:2104.05596, 2021)

## What this evaluates

Evaluates neural machine translation models for Indic languages by measuring translation quality against reference texts across multiple standard benchmarks. It specifically tests the effectiveness of training on the Samanantar parallel corpus compared to commercial systems and existing open-source baselines.

## Datasets

- **WAT2020 Indic task** — total ?; splits: test (-1)
- **WAT2021 Multi-IndicMT task** — total ?; splits: test (-1)
- **WMT test sets (2014, 2019, 2020)** — total ?; splits: test (-1)
- **UFAL Entam** — total ?; splits: test (-1)
- **FLORES test set** — total ?; splits: test (-1)
- **PMIndia en-as testset** — total 3000; splits: val (1000), test (2000)

## Metrics

- `BLEU (SacreBLEU)` **(primary)** — range: percent
  - SacreBLEU corpus score computed with direction-specific tokenization: mteval-v13a tokenizer for Indic→English, and IndicNLP tokenizer followed by SacreBLEU's default tokenizer for English→Indic.

## Input / output format

**Input**: Source sentence in English or an Indic language

**Output**: Translated target sentence

## Scoring recipe

```python
def compute_bleu(preds, refs, direction):
    if direction == 'indic_to_en':
        tok = mteval_v13a_tokenize
    else:
        tok = lambda x: sacrebleu_tokenize(indicnlp_tokenize(x))
    return sacrebleu.corpus_bleu(
        [tok(p) for p in preds],
        [[tok(r) for r in ref] for ref in refs]
    ).score
```

## Common pitfalls

- Tokenization differs by direction: Indic→English uses mteval-v13a, while English→Indic requires IndicNLP tokenizer before SacreBLEU.
- Strict train/test separation must be enforced when training baselines on existing parallel data to prevent data leakage.

## Evidence (verbatim from paper)

> We use BLEU scores for evaluating the models. To ensure consistency and reproducibility across the models, we provide SacreBLEU signatures in the footnote for Indic-English and English-Indic evaluations. For Indic-English, we use the in-built, default mteval-v13a tokenizer. For En-Indic, since SacreBLEU tokenizer does not support Indic languages, we first tokenize using the IndicNLP tokenizer before running SacreBLEU.

## Citation

```bibtex
@misc{ramesh2021samanantar,
  title={Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages},
  author={Gowtham Ramesh et al. (2021)},
  year={2021},
  note={arXiv:2104.05596}
}
```

- arXiv: 2104.05596