Chren Bleu Eval

---
name: chren-bleu-eval
description: Evaluates machine translation quality between Cherokee and English, focusing on low-resource, morphologically complex translation. It probes both in-domain and out-of-domain generalization, as well as the reliability of automatic metrics versus human judgment for polysynthetic languages. Use when the user wants to benchmark on ChrEn, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2010.04791
  bibtex_key: zhang2020chren
  confidence: high
---

# chren-bleu-eval

> ChrEn: Cherokee-English Machine Translation for Endangered Language Revitalization — Zhang et al. (2020) (arXiv:2010.04791, 2020)

## What this evaluates

Evaluates machine translation quality between Cherokee and English, focusing on low-resource, morphologically complex translation. It probes both in-domain and out-of-domain generalization, as well as the reliability of automatic metrics versus human judgment for polysynthetic languages.

## Datasets

- **ChrEn** — total 14000; splits: dev (-1), test (-1), out-dev (-1), out-test (-1); repo https://github.com/ZhangShiyue/ChrEn

## Metrics

- `BLEU` **(primary)** — range: percent
  - Detokenized and case-sensitive BLEU score computed using SacreBLEU. Standard n-gram precision with brevity penalty, applied to raw token sequences without lowercasing.

## Input / output format

**Input**: Source sentence in Cherokee or English.

**Output**: Target sentence in English or Cherokee.

## Scoring recipe

```python
import sacrebleu
# predictions and references are lists of detokenized strings
score = sacrebleu.corpus_bleu(predictions, [references], lowercase=False, tokenize='none')
return score.score
```

## Common pitfalls

- BLEU scores may not correlate with human preference for Cherokee due to its polysynthetic morphology and complex word formation.
- Out-of-domain performance trends differ significantly from in-domain; SMT often outperforms NMT on out-of-domain data despite NMT being better in-domain.

## Evidence (verbatim from paper)

> As shown in Table 5, human preference does not always follow the trends of BLEU scores. For English-Cherokee translation, though the RNN-NMT+BERT (N5) has a better BLEU score than SMT+BT (S3) (12.2 vs. 9.9), it is liked less by humans (21 vs. 29), indicating that BLEU is possibly not a suitable for Cherokee evaluation.

## Citation

```bibtex
@misc{zhang2020chren,
  title={ChrEn: Cherokee-English Machine Translation for Endangered Language Revitalization},
  author={Zhang et al. (2020)},
  year={2020},
  note={arXiv:2010.04791}
}
```

- arXiv: 2010.04791