Thiomi Baseline Eval

---
name: thiomi-baseline-eval
description: Evaluates the quality and utility of a multimodal corpus for low-resource African languages. It does so by training and testing baseline models for automatic speech recognition, machine translation, and text-to-speech across multiple languages. Use when the user wants to benchmark on Thiomi Dataset, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.29244
  bibtex_key: [REDACTED]
  confidence: high
---

# thiomi-baseline-eval

> The Thiomi Dataset: A Large-Scale Multimodal Corpus for Low-Resource African Languages — Mutisya et al. (2026) (arXiv:2603.29244, 2026)

## What this evaluates

Evaluates the quality and utility of a multimodal corpus for low-resource African languages. It does so by training and testing baseline models for automatic speech recognition, machine translation, and text-to-speech across multiple languages.

## Datasets

- **Thiomi Dataset** — total ?; splits: test (-1)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate calculated as the sum of insertions, deletions, and substitutions divided by the total number of reference words, reported as a percentage.
- `BLEU` — range: percent
  - Standard n-gram precision metric with a brevity penalty to penalize overly short translations, reported on a 0–100 scale.
- `MOS` — range: [1, 5]
  - Mean Opinion Score derived from averaging 1–5 Likert scale ratings given by human evaluators for naturalness and intelligibility of synthesized speech.

## Input / output format

**Input**: Audio recordings for ASR; parallel source-target text pairs for MT; raw text utterances for TTS.

**Output**: Predicted transcriptions for ASR; translated text for MT; synthesized audio waveforms for TTS.

## Scoring recipe

```python
# ASR: Word Error Rate
wer = (insertions + deletions + substitutions) / len(reference_words) * 100

# MT: BLEU (0-100 scale)
bleu = 100 * math.exp(sum(math.log(p_n) for p_n in ngram_precisions) / num_ngrams) * brevity_penalty

# TTS: Mean Opinion Score
mos = sum(rating for rating in evaluator_ratings) / len(evaluator_ratings)
```

## Common pitfalls

- Tonal languages like Maasai and Kipsigis use complex phonology not fully captured by Latin orthography, which artificially inflates WER.
- BLEU scores are computed on a narrow, domain-constrained sentence set, so high scores do not generalize to open-domain translation.
- MOS evaluations rely on only 5 native speakers per language rating 50 utterances, which may not capture broader listener preferences or dialectal variation.

## Evidence (verbatim from paper)

> The Swahili result (3.24% WER on Common Voice) represents the best reported academic WER for Swahili ASR, surpassing the prior best system (XLS-R finetuned, 8.3%) by 61% relative improvement.

## Citation

```bibtex
@misc{mutisya2026thiomi,
  title={The Thiomi Dataset: A Large-Scale Multimodal Corpus for Low-Resource African Languages},
  author={Mutisya et al. (2026)},
  year={2026},
  note={arXiv:2603.29244}
}
```

- arXiv: 2603.29244