Common Voice Asr Eval

---
name: common-voice-asr-eval
description: Evaluates multilingual automatic speech recognition (ASR) capabilities, specifically testing speaker generalization and low-resource language adaptation via transfer learning from an English model. It measures how well a model can transcribe audio from diverse, crowdsourced speakers across multiple languages with varying data sizes. Use when the user wants to benchmark on Common Voice, or asks about evaluating this task. Reports character error rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1912.06670
  bibtex_key: ardila2019commonvoice
  confidence: high
---

# common-voice-asr-eval

> Common Voice: A Massively-Multilingual Speech Corpus — Ardila et al. (2019) (arXiv:1912.06670, 2019)

## What this evaluates

Evaluates multilingual automatic speech recognition (ASR) capabilities, specifically testing speaker generalization and low-resource language adaptation via transfer learning from an English model. It measures how well a model can transcribe audio from diverse, crowdsourced speakers across multiple languages with varying data sizes.

## Datasets

- **Common Voice** — total ?; splits: train (-1), dev (-1), test (-1)

## Metrics

- `character error rate` **(primary)** — range: percent
  - Standard character-level edit distance (Levenshtein) between the predicted transcription and the ground truth text, normalized by the length of the reference text.

## Input / output format

**Input**: Audio clips in MPEG-3 format, processed into 494-dimensional vectors (19 spliced frames of 26 MFCC features each), paired with ground truth text transcriptions.

**Output**: Predicted character sequence corresponding to the input audio clip.

## Scoring recipe

```python
def compute_cer(predictions, golds):
    total_errors = 0
    total_refs = 0
    for pred, gold in zip(predictions, golds):
        errors = levenshtein_distance(pred, gold)
        total_errors += errors
        total_refs += len(gold)
    return total_errors / total_refs
```

## Common pitfalls

- Splits enforce strict speaker separation (one speaker only in one split) to test speaker generalization, resulting in very small training sets per language.
- Early stopping uses a custom criterion based on a 5-epoch window and a mean loss threshold of 0.5 with std < 0.5, rather than standard patience.
- Transfer learning setup copies layers from a pre-trained English model and initializes new target-language layers, which must be fine-tuned jointly.

## Evidence (verbatim from paper)

> The corpus enables end-to-end ASR experiments, achieving an average 5.99% character error rate improvement across 12 low-resource languages through transfer learning from an English model... We made dataset splits (c.f. Table (2)) such that one speaker's recordings are only present in one data split. This allows us to make a fair evaluation of speaker generalization... The splits per language were made as close as possible to 80% train, 10% development, and 10% test.

## Citation

```bibtex
@misc{ardila2019commonvoice,
  title={Common Voice: A Massively-Multilingual Speech Corpus},
  author={Ardila et al. (2019)},
  year={2019},
  note={arXiv:1912.06670}
}
```

- arXiv: 1912.06670