Hintedbt Eval

---
name: hintedbt-eval
description: Evaluates cross-script machine translation quality for low-resource Indian languages (Hindi, Gujarati, Tamil) translating to English. It specifically probes how well models leverage back-translation augmented with quality and transliteration hints to handle noisy data and script conversion challenges. Use when the user wants to benchmark on IIT Bombay en-hi Corpus, WMT-2019 gu-en, TED2020, GNOME & Ubuntu, OPUS, WMT-2020 ta-en, GNOME, OPUS, WMT-2014 hi→en test, WMT-2019 gu→en test, WMT-2020 ta→en test, or asks about evaluating this task. Reports SacreBLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2109.04443
  bibtex_key: ramnath2021hintedbt
  confidence: high
---

# hintedbt-eval

> HintedBT: Augmenting Back-Translation with Quality and Transliteration Hints — Ramnath et al. (2021) (arXiv:2109.04443, 2021)

## What this evaluates

Evaluates cross-script machine translation quality for low-resource Indian languages (Hindi, Gujarati, Tamil) translating to English. It specifically probes how well models leverage back-translation augmented with quality and transliteration hints to handle noisy data and script conversion challenges.

## Datasets

- **IIT Bombay en-hi Corpus** — total 1680000; splits: train (1680000)
- **WMT-2019 gu-en, TED2020, GNOME & Ubuntu, OPUS** — total 162000; splits: train (162000)
- **WMT-2020 ta-en, GNOME, OPUS** — total 630000; splits: train (630000)
- **WMT-2014 hi→en test** — total 2500; splits: dev (520), test (2500)
- **WMT-2019 gu→en test** — total 1000; splits: dev (3400), test (1000)
- **WMT-2020 ta→en test** — total 1000; splits: dev (2000), test (1000)

## Metrics

- `SacreBLEU` **(primary)** — range: percent
  - Standard n-gram precision-based metric computed using the SacreBLEU toolkit. It uses tokenized text (WordPiece) and applies sentence-level BLEU averaged over the test set.

## Input / output format

**Input**: Source sentence in Hindi, Gujarati, or Tamil.

**Output**: Target sentence in English.

## Scoring recipe

```python
import sacrebleu
predictions = [pred.strip() for pred in model_outputs]
references = [[ref.strip()] for ref in gold_targets]
score = sacrebleu.corpus_bleu(predictions, references).score
```

## Common pitfalls

- BLEU scores are computed on WordPiece tokenized text, not raw characters.
- Human evaluation uses a 0-6 scale for quality assessment but is not the primary reported metric.
- Back-translation data is generated from WMT Newscrawl 2013 English monolingual corpus, which is separate from the test sets.

## Evidence (verbatim from paper)

> We use SacreBLEU² (Post, 2018) to evaluate our models.

## Citation

```bibtex
@misc{ramnath2021hintedbt,
  title={HintedBT: Augmenting Back-Translation with Quality and Transliteration Hints},
  author={Ramnath et al. (2021)},
  year={2021},
  note={arXiv:2109.04443}
}
```

- arXiv: 2109.04443