Scb Mt En Th 2020 Eval

---
name: scb-mt-en-th-2020-eval
description: This protocol evaluates neural machine translation quality between English and Thai. It measures translation accuracy on a newly curated 1M-parallel corpus (SCB_1M) and a filtered OPUS corpus, while also testing cross-domain generalization on the IWSLT 2015 Thai-English benchmark. Use when the user wants to benchmark on SCB_1M, MT_OPUS, IWSLT 2015 Thai-English, or asks about evaluating this task. Reports SacreBLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2007.03541
  bibtex_key: lowphansirikul2020scbmtenth2020
  confidence: high
---

# scb-mt-en-th-2020-eval

> scb-mt-en-th-2020: A Large English-Thai Parallel Corpus — Lalita Lowphansirikul et al. (arXiv:2007.03541, 2020)

## What this evaluates

This protocol evaluates neural machine translation quality between English and Thai. It measures translation accuracy on a newly curated 1M-parallel corpus (SCB_1M) and a filtered OPUS corpus, while also testing cross-domain generalization on the IWSLT 2015 Thai-English benchmark.

## Datasets

- **SCB_1M** — total 1001752; splits: train (-1), val (-1), test (100177)
- **MT_OPUS** — total 3318153; splits: train (-1), val (-1), test (297874)
- **IWSLT 2015 Thai-English** — total 4242; splits: test (4242)

## Metrics

- `SacreBLEU` **(primary)** — range: percent
  - Corpus-level BLEU score computed with the SacreBLEU toolkit. Uses 13a tokenization, exponential smoothing, and exactly one reference per hypothesis. Case handling is mixed-case for Thai-to-English and lowercase for English-to-Thai.

## Input / output format

**Input**: Source sentence in either Thai or English.

**Output**: Target sentence in either English or Thai.

## Scoring recipe

```python
import sacrebleu

def compute_bleu(hypotheses, references, direction):
    if direction == 'th_en':
        return sacrebleu.corpus_bleu(
            hypotheses, references,
            lowercase=False, tokenize='13a',
            smooth_method='exp', smooth_value=0
        ).score
    else: # en_th
        return sacrebleu.corpus_bleu(
            hypotheses, references,
            lowercase=True, tokenize='13a',
            smooth_method='exp', smooth_value=0
        ).score
```

## Common pitfalls

- Detokenization must strictly follow direction-specific rules: Moses detokenizer for word-level th→en, SentencePiece for subword outputs, and space-joining for word-level en→th.
- Cross-domain evaluation (training on SCB_1M, testing on MT_OPUS or vice versa) yields drastically lower BLEU scores (~6–13) compared to in-domain evaluation (~25–42), which can mislead model selection if splits are not stratified by source domain.
- Case sensitivity conventions differ between directions in the reported results, affecting direct score comparisons across language pairs.

## Evidence (verbatim from paper)

> SacreBLEU [Post, 2018] is used to evaluate translation quality in both directions. For th → en translation, word-level outputs are detokenized with Moses detokenizer and subword outputs for both Thai and English are detokenized Sentencepiece [Kudo and Richardson, 2018]. The version string used for computing BLEU score for case-sensitive and case-insertive are BLEU + case.mixed + numrefs.1 + smooth.exp + tok.13a + version.1.2.10 and BLEU + case.lc + numrefs.1 + smooth.exp + tok.13a + version.1.2.12 respectively.

## Citation

```bibtex
@misc{lowphansirikul2020scbmtenth2020,
  title={scb-mt-en-th-2020: A Large English-Thai Parallel Corpus},
  author={Lalita Lowphansirikul et al.},
  year={2020},
  note={arXiv:2007.03541}
}
```

- arXiv: 2007.03541