Hinmix Robust Cmt Eval

---
name: hinmix-robust-cmt-eval
description: Evaluates machine translation models on code-mixed and noisy Hindi-English and Bengali-English text, measuring robustness to script variations, romanization, and synthetic noise. The protocol tests both in-domain performance on the HINMIX corpus and out-of-domain generalizability on LinCE, SpokenTutorial, and IITB Hi-En. It also assesses zero-shot transfer to unseen code-mixed Bengali-English translation. Use when the user wants to benchmark on HINMIX, or asks about evaluating this task. Reports SacreBLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.16771
  bibtex_key: kartik2024synthetic
  confidence: high
---

# hinmix-robust-cmt-eval

> Synthetic Data Generation and Joint Learning for Robust Code-Mixed Translation — Kartik et al. (2024) (arXiv:2403.16771, 2024)

## What this evaluates

Evaluates machine translation models on code-mixed and noisy Hindi-English and Bengali-English text, measuring robustness to script variations, romanization, and synthetic noise. The protocol tests both in-domain performance on the HINMIX corpus and out-of-domain generalizability on LinCE, SpokenTutorial, and IITB Hi-En. It also assesses zero-shot transfer to unseen code-mixed Bengali-English translation.

## Datasets

- **HINMIX** — total ?; splits: test (-1); repo https://github.com/LCS2-IIITD/Robust_CodeMIX_MT

## Metrics

- `SacreBLEU` **(primary)** — range: [0, 100]
  - Standard BLEU score computed with the SacreBLEU toolkit, using standard tokenization, corpus-level aggregation, and default smoothing.
- `METEOR` — range: [0, 100]
  - Alignment-based metric that computes precision, recall, and penalty for fragmentation, incorporating synonymy matching and stemming.

## Input / output format

**Input**: Source sentence in code-mixed Hindi/English or Bengali/English, presented in Devanagari, Romanized, or noisy (typo/phonetic) variants.

**Output**: Target English sentence.

## Scoring recipe

```python
# SacreBLEU
bleu = sacrebleu.corpus_bleu(predictions, [references])
# METEOR
meteor = meteor_score(predictions, references)
# Report both as percentages (0-100)
```

## Common pitfalls

- Baselines are fine-tuned from scratch on HINMIX for fair comparison, not evaluated in a zero-shot or pre-trained setting.
- Noise robustness is tested on three distinct corpus variants (clean code-mixed, romanized, and noisy), not a single mixed test set.
- Zero-shot Bengali evaluation uses a completely unseen language pair without any parallel code-mixed Bengali training data.

## Evidence (verbatim from paper)

> Finally, we evaluate the quality of models on SacreBLEU (Ott et al., 2019) and METEOR (Banerjee and Lavie, 2005) metrics.

## Citation

```bibtex
@misc{kartik2024synthetic,
  title={Synthetic Data Generation and Joint Learning for Robust Code-Mixed Translation},
  author={Kartik et al. (2024)},
  year={2024},
  note={arXiv:2403.16771}
}
```

- arXiv: 2403.16771