Sagalee Asr Eval

---
name: sagalee-asr-eval
description: Evaluates automatic speech recognition (ASR) performance on the Oromo language using real-world, crowd-sourced audio data. It measures how well different model architectures (Conformer trained from scratch, Whisper fine-tuned) transcribe spoken Oromo into text under varying acoustic conditions. Use when the user wants to benchmark on Sagalee, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.00421
  bibtex_key: turi2025sagalee
  confidence: high
---

# sagalee-asr-eval

> Sagalee: an Open Source Automatic Speech Recognition Dataset for Oromo Language — Turi Abu et al. (2025) (arXiv:2502.00421, 2025)

## What this evaluates

Evaluates automatic speech recognition (ASR) performance on the Oromo language using real-world, crowd-sourced audio data. It measures how well different model architectures (Conformer trained from scratch, Whisper fine-tuned) transcribe spoken Oromo into text under varying acoustic conditions.

## Datasets

- **Sagalee** — total 100; splits: train (94), dev (4), test (2); repo https://github.com/turinaf/sagalee

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate (WER) is the standard ASR metric calculated as the minimum number of word edits (insertions, deletions, substitutions) required to transform the predicted transcription into the reference transcription, divided by the total number of words in the reference. Expressed as a percentage.

## Input / output format

**Input**: Acoustic features extracted from Oromo speech audio recordings (x).

**Output**: Text transcription corresponding to the input audio (y).

## Scoring recipe

```python
def compute_wer(predictions, references):
    total_words = sum(len(ref.split()) for ref in references)
    if total_words == 0: return 0.0
    edits = 0
    for pred, ref in zip(predictions, references):
        # Standard Levenshtein distance on word tokens
        dist = levenshtein_distance(pred.split(), ref.split())
        edits += dist
    return (edits / total_words) * 100
```

## Common pitfalls

- The dataset size is reported in hours (93.6h/4.2h/2.4h) rather than sample counts, which can cause confusion when comparing to text-only benchmarks.
- Model evaluation uses an ensemble of checkpoints (average of 10 best for Conformer, average of last 3 for Whisper) rather than a single final checkpoint, which inflates reported performance compared to standard single-model baselines.
- WER is highly sensitive to tokenization; the paper uses BPE (nbpe=500) for training, so evaluation must use the exact same tokenizer to be reproducible.

## Evidence (verbatim from paper)

> The dataset is split into train, dev and test sets with 93.6hrs, 4.2hrs, and 2.4hrs size respectively. The performance of the trained models in terms of Word Error Rate (WER) is summarized in Table [IV]... Conformer AED and Conformer CTC models achieve WERs of 15.32% and 18.74%, respectively... The Whisper model achieves a significantly lower WER... 10.82%

## Citation

```bibtex
@misc{turi2025sagalee,
  title={Sagalee: an Open Source Automatic Speech Recognition Dataset for Oromo Language},
  author={Turi Abu et al. (2025)},
  year={2025},
  note={arXiv:2502.00421}
}
```

- arXiv: 2502.00421