Pmindia Nmt Eval

---
name: pmindia-nmt-eval
description: This evaluation probes the quality of automatic machine translation between English and 13 Indian languages using a parallel corpus. It measures how well NMT systems can handle diverse linguistic structures, including abugida scripts and agglutinative morphology, across low-resource language pairs. Use when the user wants to benchmark on PMIndia, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2001.09907
  bibtex_key: haddow2020pmindia
  confidence: high
---

# pmindia-nmt-eval

> PMIndia -- A Collection of Parallel Corpora of Languages of India — Haddow et al. (2020) (arXiv:2001.09907, 2020)

## What this evaluates

This evaluation probes the quality of automatic machine translation between English and 13 Indian languages using a parallel corpus. It measures how well NMT systems can handle diverse linguistic structures, including abugida scripts and agglutinative morphology, across low-resource language pairs.

## Datasets

- **PMIndia** — total ?; splits: dev (1000), test (1000); repo https://github.com/bhoddow/pmindia-crawler

## Metrics

- `BLEU` **(primary)** — range: percent
  - Standard n-gram precision-based metric computed over tokenized text. The paper explicitly uses the Moses multi-bleu.perl script rather than sacrebleu due to lack of support for South Asian scripts and languages.

## Input / output format

**Input**: Source sentence in English or one of the 13 Indian languages, preprocessed with language-specific normalization, tokenization, and BPE subword splitting (10000 merges).

**Output**: Translated target sentence in the corresponding language, evaluated on the same tokenized/subword format.

## Scoring recipe

```python
def compute_bleu(predictions, references):
    # predictions, references: list of tokenized strings
    # Uses Moses multi-bleu.perl script
    import subprocess
    with open('pred.txt', 'w') as f:
        f.write('\n'.join(predictions))
    cmd = f"multi-bleu.perl {' '.join(references)} < pred.txt"
    out = subprocess.run(cmd, shell=True, capture_output=True, text=True)
    return float(out.stdout.split()[0])
```

## Common pitfalls

- Using sacrebleu instead of the Moses multi-bleu.perl script, as sacrebleu lacks support for South Asian scripts and languages used in this benchmark.
- Relying solely on BLEU for agglutinative Dravidian languages, as word-based n-gram matching is known to be less reliable for morphologically rich languages.
- Evaluating on raw text without applying the specified BPE subword splits (10000 merges) and language-specific tokenization, which significantly affects score comparability.

## Evidence (verbatim from paper)

> The MT results (BLEU scores) are show in Table 5. We evaluated on tokenized text, using the multi-bleu.perl script from Moses, since sacrebleu (Post, 2018) does not currently support South Asian languages.

## Citation

```bibtex
@misc{haddow2020pmindia,
  title={PMIndia -- A Collection of Parallel Corpora of Languages of India},
  author={Haddow et al. (2020)},
  year={2020},
  note={arXiv:2001.09907}
}
```

- arXiv: 2001.09907