Igbo English Mt Eval

---
name: igbo-english-mt-eval
description: This benchmark evaluates bidirectional machine translation quality between Igbo and English. It probes a model's ability to accurately translate news and contemporary media content across two directions (Igbo-to-English and English-to-Igbo) using human-validated parallel sentences. Use when the user wants to benchmark on Igbo-English MT Benchmark, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2004.00648
  bibtex_key: ezeani2020igboenglish
  confidence: medium
---

# igbo-english-mt-eval

> Igbo-English Machine Translation: An Evaluation Benchmark — Ezeani et al. (2020) (arXiv:2004.00648, 2020)

## What this evaluates

This benchmark evaluates bidirectional machine translation quality between Igbo and English. It probes a model's ability to accurately translate news and contemporary media content across two directions (Igbo-to-English and English-to-Igbo) using human-validated parallel sentences.

## Datasets

- **Igbo-English MT Benchmark** — total 11584; splits: dev (10000), test (1000), hidden_test (584); repo https://github.com/IgnatiusEzeani/IGBONLP

## Metrics

- `BLEU` **(primary)** — range: [0, 100]
  - Standard n-gram overlap metric for machine translation. The paper does not explicitly define the metric in the provided section, but BLEU is the standard evaluation measure for MT benchmarks of this type.

## Input / output format

**Input**: A single source sentence in either Igbo or English.

**Output**: A single target sentence in the corresponding target language (English or Igbo).

## Scoring recipe

```python
```python
import sacrebleu
def compute_bleu(predictions, references):
    # predictions and references are lists of strings
    score = sacrebleu.corpus_bleu(predictions, [references])
    return score.score  # Returns 0-100
```
```

## Common pitfalls

- The hidden test set (584 pairs) is not publicly released, preventing direct leaderboard comparison or full-dataset evaluation.
- The dataset is bidirectional (IG-EN and EN-IG); models must be evaluated separately for each direction to avoid cross-contamination.
- The ~380k monolingual Igbo sentences are provided for pre-training or unsupervised MT, not for direct supervised evaluation.

## Evidence (verbatim from paper)

> The splits of the parallel data into development, text and hidden test sets is shown in Table 2. Table 1: Breakdown of the Benchmark Evaluation Parallel Data ... Total 11,584

## Citation

```bibtex
@misc{ezeani2020igboenglish,
  title={Igbo-English Machine Translation: An Evaluation Benchmark},
  author={Ezeani et al. (2020)},
  year={2020},
  note={arXiv:2004.00648}
}
```

- arXiv: 2004.00648