Kreyol Mt Eval

---
name: kreyol-mt-eval
description: Evaluates machine translation performance across 41 Creole languages, testing cross-lingual transfer and the impact of data cleaning and scale on translation quality. Use when the user wants to benchmark on Kreyol-MT, or asks about evaluating this task. Reports BLEU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.05376
  bibtex_key: robinson2024kreyolmt
  confidence: medium
---

# kreyol-mt-eval

> Krey\`ol-MT: Building MT for Latin American, Caribbean and Colonial African Creole Languages — Robinson et al. (2024) (arXiv:2405.05376, 2024)

## What this evaluates

Evaluates machine translation performance across 41 Creole languages, testing cross-lingual transfer and the impact of data cleaning and scale on translation quality.

## Datasets

- **Kreyol-MT** — total ?; splits: train (-1), dev (-1), test (-1); repo https://github.com/JHU-CLSP/Kreyol-MT

## Metrics

- `BLEU` **(primary)** — range: [0, 100]
  - Standard sentence-level BLEU score (Papineni et al., 2002), computed with tokenizer normalization. (Note: exact metric not explicitly stated in the provided section, but BLEU is the standard evaluation metric for machine translation.)

## Input / output format

**Input**: Source language Creole sentence.

**Output**: Target language Creole sentence.

## Scoring recipe

```python
def compute_bleu(predictions, references):
    # Tokenize predictions and references using standard MT tokenizer
    # Compute n-gram precision for n=1..4
    # Apply brevity penalty based on reference and prediction lengths
    # Return geometric mean of precisions * brevity penalty
    # Note: Exact implementation details are not provided in the text.
```

## Common pitfalls

- Test set contamination: must explicitly remove any sentence pairs from train/dev that overlap with pre-existing test sets.
- Cleaning configuration mismatch: models trained on cleaned vs. non-cleaned data use different test splits, making direct comparison invalid.
- Zero-shot evaluations: language pairs with removed training data are still evaluated zero-shot, which may skew aggregate results.

## Evidence (verbatim from paper)

> After filtering, we prepare a train/dev/test split for each language pair of the remaining data by aggregating all sentences and splitting randomly with a fixed random seed, and a target ratio of 85 / 5 / 10, with minimum 50 and maximum 2000 sentences for the dev and test sets. We discard train sets with fewer than 100 sentences.

## Citation

```bibtex
@misc{robinson2024kreyolmt,
  title={Krey\`ol-MT: Building MT for Latin American, Caribbean and Colonial African Creole Languages},
  author={Robinson et al. (2024)},
  year={2024},
  note={arXiv:2405.05376}
}
```

- arXiv: 2405.05376