Kunkado Nyana Eval Eval

---
name: kunkado-nyana-eval-eval
description: Evaluates automatic speech recognition (ASR) models on spontaneous, radio-based Bambara speech containing real-world artifacts like code-switching, overlapping speakers, and background noise. It measures transcription accuracy under pragmatic normalization conditions. Use when the user wants to benchmark on Kunkado Test, Nyana-Eval, or asks about evaluating this task. Reports WER (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.19400
  bibtex_key: diarra2025kunnafonidilaw
  confidence: high
---

# kunkado-nyana-eval-eval

> Kunnafonidilaw ka Cadeau: an ASR dataset of present-day Bambara — Diarra et al. (2025) (arXiv:2512.19400, 2025)

## What this evaluates

Evaluates automatic speech recognition (ASR) models on spontaneous, radio-based Bambara speech containing real-world artifacts like code-switching, overlapping speakers, and background noise. It measures transcription accuracy under pragmatic normalization conditions.

## Datasets

- **Kunkado Test** — total ?; splits: test (-1)
- **Nyana-Eval** — total ?; splits: test (-1)

## Metrics

- `WER (%)` **(primary)** — range: percent
  - Word Error Rate: percentage of words incorrectly transcribed. Calculated as (Substitutions + Deletions + Insertions) / Total Words in reference.
- `CER (%)` — range: percent
  - Character Error Rate: percentage of characters incorrectly transcribed. Calculated as (Substitutions + Deletions + Insertions) / Total Characters in reference.

## Input / output format

**Input**: Raw audio recordings of spontaneous Bambara speech (radio broadcasts).

**Output**: Normalized text transcription. Acoustic event tags, code-switching markers, punctuation, and diacritics are removed from both reference and prediction before scoring.

## Scoring recipe

```python
def score_asr(predictions, references):
    # Apply normalization: remove tags, code-switching markers, punctuation, diacritics
    pred_clean = normalize_text(predictions)
    ref_clean = normalize_text(references)
    # Calculate edit distances
    wer = edit_distance(pred_clean, ref_clean) / len(ref_clean.split())
    cer = edit_distance(pred_clean, ref_clean) / len(ref_clean)
    return wer * 100, cer * 100
```

## Common pitfalls

- Failing to strip acoustic event tags and code-switching markers from both reference and hypothesis before scoring, which artificially inflates error rates.
- Applying training-time normalizations (e.g., removing numbers/diacritics) inconsistently during evaluation, leading to non-comparable WER/CER values.
- Confusing the 33.47-hour human-reviewed training subset with the 5-hour test set used for reporting.

## Evidence (verbatim from paper)

> We evaluated all the models on a 5 hour test set taken from the Kunkado data, and Nyana-Eval, a small, stratified human evaluation dataset with only 45 entries of 3 minutes total duration... We apply the same normalization steps as explained in section 3 and remove the tags from both the reference and the prediction before calculating the WER and CER.

## Citation

```bibtex
@misc{diarra2025kunnafonidilaw,
  title={Kunnafonidilaw ka Cadeau: an ASR dataset of present-day Bambara},
  author={Diarra et al. (2025)},
  year={2025},
  note={arXiv:2512.19400}
}
```

- arXiv: 2512.19400