Ebible Benchmarks Eval

---
name: ebible-benchmarks-eval
description: Machine translation performance on low-resource languages using verse-aligned Bible texts. It probes model robustness across different biblical book genres (Gospels, Epistles, OT books) and the utility of related language data for translation. Use when the user wants to benchmark on eBible Corpus, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2304.09919
  bibtex_key: akerman2023ebible
  confidence: high
---

# ebible-benchmarks-eval

> The eBible Corpus: Data and Model Benchmarks for Bible Translation for Low-Resource Languages — Åkerman et al. (2023) (arXiv:2304.09919, 2023)

## What this evaluates

Machine translation performance on low-resource languages using verse-aligned Bible texts. It probes model robustness across different biblical book genres (Gospels, Epistles, OT books) and the utility of related language data for translation.

## Datasets

- **eBible Corpus** — total ?; splits: train (-1), test (-1); repo https://github.com/BibleNLP/ebible

## Metrics

- `BLEU` **(primary)** — range: percent
  - Standard n-gram precision metric with brevity penalty. Calculated at the sentence/verse level.
- `spBLEU` — range: percent
  - Sentence-piece BLEU, which computes BLEU over subword tokenizations to handle morphologically rich or low-resource languages better.
- `chrF3` — range: percent
  - Character n-gram F-score with n=3, measuring character-level precision and recall.

## Input / output format

**Input**: Source language Bible verse text.

**Output**: Target language translated Bible verse text.

## Scoring recipe

```python
def compute_metrics(predictions, references):
    bleu = sacrebleu.corpus_bleu(predictions, [references]).score
    sp_bleu = sacrebleu.corpus_bleu(predictions, [references], tokenize='spm').score
    chrf3 = sacrebleu.corpus_chrf(predictions, [references], char_order=3).score
    return {'BLEU': bleu, 'spBLEU': sp_bleu, 'chrF3': chrf3}
```

## Common pitfalls

- BLEU scores can vary significantly from subword metrics (spBLEU) for morphologically complex languages, so relying solely on BLEU may misrepresent performance.
- The benchmark uses verse-aligned data, so evaluation must be done at the verse level, not paragraph or chapter level.
- Performance varies widely across biblical book genres (Gospels vs. Epistles vs. OT), so reporting a single aggregate score without genre breakdown is misleading.

## Evidence (verbatim from paper)

> Figure 6 shows a bar chart of the median BLEU, spBLEU, and chrF3 scores for all eight translation pairings. Interestingly, we find no clear correlation between the scope (NT-only, NT with partial OT, or full Bible) of the translation pairing and our selected scoring metrics.

## Citation

```bibtex
@misc{akerman2023ebible,
  title={The eBible Corpus: Data and Model Benchmarks for Bible Translation for Low-Resource Languages},
  author={Åkerman et al. (2023)},
  year={2023},
  note={arXiv:2304.09919}
}
```

- arXiv: 2304.09919