Swivuriso Asr Eval

---
name: swivuriso-asr-eval
description: Evaluates automatic speech recognition (ASR) capabilities across seven South African languages. It measures how well pre-trained speech models can transcribe spontaneous and scripted audio in low-resource, domain-specific contexts (agriculture, healthcare, general). Use when the user wants to benchmark on Swivuriso, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.02201
  bibtex_key: [REDACTED]
  confidence: high
---

# swivuriso-asr-eval

> Swivuriso: The South African Next Voices Multilingual Speech Dataset — Marivate et al. (2025) (arXiv:2512.02201, 2025)

## What this evaluates

Evaluates automatic speech recognition (ASR) capabilities across seven South African languages. It measures how well pre-trained speech models can transcribe spontaneous and scripted audio in low-resource, domain-specific contexts (agriculture, healthcare, general).

## Datasets

- **Swivuriso** — total 3000; splits: test (-1)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate: the number of substitutions, deletions, and insertions required to transform the predicted transcript into the reference transcript, divided by the total number of words in the reference. Lower is better.
- `CER` — range: percent
  - Character Error Rate: identical to WER but computed at the character level instead of word level. Lower is better.

## Input / output format

**Input**: Raw audio recordings (speech) in seven South African languages, spanning scripted and unscripted domains.

**Output**: Predicted text transcript corresponding to the input audio.

## Scoring recipe

```python
def compute_wer(predictions, references):
    # Align predicted and reference word sequences
    # Count substitutions (S), deletions (D), insertions (I)
    # N = number of words in reference
    wer = (S + D + I) / N
    return wer * 100  # reported as percent
```

## Common pitfalls

- Zero-shot WER can exceed 100% for low-resource languages due to high insertion/deletion rates, which may be misinterpreted as model failure rather than a known ASR artifact.
- Cross-dataset comparisons require careful downsampling to equal durations (~6 hours), which can introduce sampling bias if not stratified by domain or speaker.
- Training is reported in optimization steps rather than epochs; comparing step counts across models with different learning rates or batch sizes requires normalization.

## Evidence (verbatim from paper)

> Table[6] summarises the Word Error Rate (WER) achieved by each model at early (1,000 steps) and final (10,000 steps) training checkpoints. Continued fine-tuning yields substantial improvements across all models, with Whisper-large-v3-Turbo achieving the largest absolute reduction of 0.22, followed by Wav2Vec-BERT at 0.27 and MMS at 0.15.

## Citation

```bibtex
@misc{marivate2025swivuriso,
  title={Swivuriso: The South African Next Voices Multilingual Speech Dataset},
  author={Marivate et al. (2025)},
  year={2025},
  note={arXiv:2512.02201}
}
```

- arXiv: 2512.02201