Indictrans2 Eval

---
name: indictrans2-eval
description: This benchmark evaluates multilingual machine translation quality across 22 scheduled Indian languages and English. It probes a model's ability to handle diverse domains (news, web, conversation, legal, etc.) and both Indic-to-English and English-to-Indic translation directions in an n-way parallel setting. Use when the user wants to benchmark on IN22, FLORES-200, NTREX, WMT (2014, 2019, 2020), WAT (2020, 2021), UFAL, or asks about evaluating this task. Reports chrF++.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.16307
  bibtex_key: gala2023indictrans2
  confidence: high
---

# indictrans2-eval

> IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages — Gala et al. (2023) (arXiv:2305.16307, 2023)

## What this evaluates

This benchmark evaluates multilingual machine translation quality across 22 scheduled Indian languages and English. It probes a model's ability to handle diverse domains (news, web, conversation, legal, etc.) and both Indic-to-English and English-to-Indic translation directions in an n-way parallel setting.

## Datasets

- **IN22** — total ?; splits: test (-1); repo https://github.com/AI4Bharat/IndicTrans2
- **FLORES-200** — total ?; splits: test (-1); HF `facebook/flores`
- **NTREX** — total ?; splits: test (-1)
- **WMT (2014, 2019, 2020)** — total ?; splits: test (-1)
- **WAT (2020, 2021)** — total ?; splits: test (-1)
- **UFAL** — total ?; splits: test (-1)

## Metrics

- `chrF++` **(primary)** — range: [0, 100]
  - Character n-gram F-score extended with word unigrams and bigrams. Computed via sacreBLEU with character order 6, word order 2, and no space normalization.
- `BLEU` — range: [0, 100]
  - Standard n-gram precision metric. Computed using sacreBLEU with mteval-v13a tokenizer for Indic-to-English and IndicNLP/Urduhack tokenizers for English-to-Indic.
- `COMET-DA` — range: [0, 1]
  - Reference-based model-based metric using the COMET-22 DA model (XLM-RoBERTa backbone). Only reported for 13 Indic languages supported by the underlying XLM-R model.

## Input / output format

**Input**: Source sentence in English or one of the 22 scheduled Indian languages.

**Output**: Translated target sentence in the corresponding target language.

## Scoring recipe

```python
def compute_chrF2_plus(predictions, references, direction):
    if direction == 'Indic-En':
        tok = 'mteval-v13a'
    else:
        tok = 'indicnlp_urduhack'
    
    tokenized_preds = tokenize(predictions, method=tok)
    tokenized_refs = tokenize(references, method=tok)
    
    score = sacrebleu.corpus_chrf(
        tokenized_preds, [tokenized_refs],
        char_order=6, word_order=2, space=False
    )
    return score.score
```

## Common pitfalls

- Tokenization dependency: BLEU and chrF++ scores are highly sensitive to tokenization. Standard Moses tokenizers fail on Indic scripts; specific tokenizers (IndicNLP, Urduhack) must be used for En-Indic directions.
- COMET coverage gap: COMET-22 DA relies on XLM-RoBERTa and only supports 13 of the 22 Indic languages, leaving 9 languages unmeasured by this metric.
- Proprietary model opacity: Commercial systems (Google Translate, Azure, GPT-3.5) are evaluated via API without access to training data or internal parameters, making fair comparison an approximation.

## Evidence (verbatim from paper)

> In this work, we, therefore, primarily rely on chrF++ as our primary metric for evaluating translation quality. We also report additional metrics such as BLEU (Papineni et al., 2002) and COMET (Rei et al., 2022).

## Citation

```bibtex
@misc{gala2023indictrans2,
  title={IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languages},
  author={Gala et al. (2023)},
  year={2023},
  note={arXiv:2305.16307}
}
```

- arXiv: 2305.16307