Ethiomt Eval

---
name: ethiomt-eval
description: Machine translation performance across multiple low-resource Ethiopian languages paired with English, evaluating both English-to-Ethiopian and Ethiopian-to-English directions. It probes how model initialization (training from scratch vs. fine-tuning a multilingual model) and available corpus size impact translation quality. Use when the user wants to benchmark on EthioMT, or asks about evaluating this task. Reports spBLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.19365
  bibtex_key: tonja2024ethiomt
  confidence: high
---

# ethiomt-eval

> EthioMT: Parallel Corpus for Low-resource Ethiopian Languages — Tonja et al. (2024) (arXiv:2403.19365, 2024)

## What this evaluates

Machine translation performance across multiple low-resource Ethiopian languages paired with English, evaluating both English-to-Ethiopian and Ethiopian-to-English directions. It probes how model initialization (training from scratch vs. fine-tuning a multilingual model) and available corpus size impact translation quality.

## Datasets

- **EthioMT** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `spBLEU` **(primary)** — range: [0, 100]
  - Sentence-level BLEU score computed using the SacreBLEU library, which standardizes tokenization and reference handling. It is typically reported on a 0-100 scale.

## Input / output format

**Input**: Source sentence in an Ethiopian language or English, paired with a target sentence in English or an Ethiopian language respectively.

**Output**: Translated sentence in the target language.

## Scoring recipe

```python
import sacrebleu
# predictions: list of generated translations
# references: list of lists of reference translations
score = sacrebleu.corpus_bleu(predictions, [references]).score
```

## Common pitfalls

- Performance varies drastically with corpus size, so low scores for some languages may reflect data scarcity rather than model failure.
- Fine-tuning a pre-trained multilingual model (m2m100) yields significantly higher scores than training a transformer from scratch, making baseline selection critical for fair comparison.
- Absolute spBLEU scores are extremely low (0.01–17.8), indicating that even the best models struggle with these low-resource languages.

## Evidence (verbatim from paper)

> We evaluated the above approaches in bidirectional translation from Ethiopian languages to English and From English to Ethiopian languages. We used Sacrebleu Post ([2018]) evaluation metrics to evaluate translation models. ... the performance of the transformer model also varies in the ranges of 0.01 – 17.8 spBLEU from language to language with different corpus sizes.

## Citation

```bibtex
@misc{tonja2024ethiomt,
  title={EthioMT: Parallel Corpus for Low-resource Ethiopian Languages},
  author={Tonja et al. (2024)},
  year={2024},
  note={arXiv:2403.19365}
}
```

- arXiv: 2403.19365