Wmt24 Indic Mt Eval

---
name: wmt24-indic-mt-eval
description: Evaluates machine translation quality for low-resource Northeast Indian languages (Assamese, Khasi, Mizo, Manipuri) paired with English. It probes cross-lingual transfer capabilities, model adaptation under data scarcity, and the effectiveness of architectural constraints like layer freezing and script-based language grouping. Use when the user wants to benchmark on IndicNECorp1.0, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.03215
  bibtex_key: [REDACTED]
  confidence: high
---

# wmt24-indic-mt-eval

> NLIP_Lab-IITH Low-Resource MT System for WMT24 Indic MT Shared Task — Sahoo et al. (2024) (arXiv:2410.03215, 2024)

## What this evaluates

Evaluates machine translation quality for low-resource Northeast Indian languages (Assamese, Khasi, Mizo, Manipuri) paired with English. It probes cross-lingual transfer capabilities, model adaptation under data scarcity, and the effectiveness of architectural constraints like layer freezing and script-based language grouping.

## Datasets

- **IndicNECorp1.0** — total ?; splits: dev (-1), test (-1)

## Metrics

- `BLEU` **(primary)** — range: percent
  - Standard n-gram overlap metric. Computed using SacreBLEU v2.3.1 with mixed case, no tokenization (13a), exponential smoothing, and nrefs:1.
- `chrF` — range: percent
  - Character n-gram F-score metric. Computed with a character n-gram order of 2.
- `chrF++` — range: percent
  - Extension of chrF that incorporates word n-grams alongside character n-grams to improve correlation with human judgment.
- `TER` — range: percent
  - Translation Edit Rate measuring the minimum number of edits (insertions, deletions, substitutions, shifts) required to transform the hypothesis into the reference.
- `RIBES` — range: [0, 1]
  - Rank-based Index evaluating translation quality based on rank correlation of n-gram matches, robust to word order variations.
- `COMET` — range: [0, 1]
  - Cross-lingual Evaluation Framework using a pre-trained multilingual model to predict human-like quality scores based on source, reference, and hypothesis.

## Input / output format

**Input**: Source sentence in English or a Northeast Indian language (Assamese, Khasi, Mizo, Manipuri).

**Output**: Translated target sentence in the corresponding target language.

## Scoring recipe

```python
import sacrebleu
# predictions and gold are lists of strings
bleu = sacrebleu.corpus_bleu(
    predictions, [gold],
    tokenize='none', smooth_method='exp',
    lowercase=False, force=False, lowercase_mixed=True,
    use_effective_order=True
)
return bleu.score
```

## Common pitfalls

- SacreBLEU tokenization and smoothing settings must exactly match the paper's signature (tok:13a, smooth:exp, case:mixed) to reproduce scores.
- chrF word order parameter defaults to 6 in many implementations, but the paper explicitly sets it to 2.
- COMET and RIBES scores are highly sensitive to the specific model checkpoint and preprocessing pipeline used; default HuggingFace pipelines may diverge from reported values.

## Evidence (verbatim from paper)

> We evaluate using BLEU Papineni et al. ([2002]), chrF Popović ([2015]), and chrF++ Popović ([2017]) metrics. We use the SacreBLEU toolkit Post ([2018]) to perform our evaluation777SacreBLEU signature: nrefs:1|case:mixed|eff:no|tok:13a|smooth:exp|version:2.3.1 with a chrF word order of 2. Additionally, as per the evaluation metrics used by the organizers, we report results on TER Snover et al. ([2006]), RIBES Isozaki et al. ([2010]), and COMET Rei et al. ([2022]) for our primary and contrastive submissions.

## Citation

```bibtex
@misc{sahoo2024nliplabiith,
  title={NLIP_Lab-IITH Low-Resource MT System for WMT24 Indic MT Shared Task},
  author={Sahoo et al. (2024)},
  year={2024},
  note={arXiv:2410.03215}
}
```

- arXiv: 2410.03215