Mls Asr Eval

---
name: mls-asr-eval
description: Evaluates automatic speech recognition (ASR) systems on multilingual read speech data. It measures word error rate across multiple languages and decoding strategies to assess model performance and data scaling effects. Use when the user wants to benchmark on MLS, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2012.03411
  bibtex_key: pratap2020mls
  confidence: high
---

# mls-asr-eval

> MLS: A Large-Scale Multilingual Dataset for Speech Research — Pratap et al. (2020) (arXiv:2012.03411, 2020)

## What this evaluates

Evaluates automatic speech recognition (ASR) systems on multilingual read speech data. It measures word error rate across multiple languages and decoding strategies to assess model performance and data scaling effects.

## Datasets

- **MLS** — total ?; splits: dev (-1), test (-1)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate, calculated as the number of substitutions, deletions, and insertions required to transform the predicted transcript into the reference transcript, divided by the total number of words in the reference. Reported as a percentage.

## Input / output format

**Input**: 80-channel log-mel filterbanks extracted from audio utterances (10–20 seconds long).

**Output**: Predicted word or grapheme sequence (decoded via Viterbi or beam-search).

## Scoring recipe

```python
def compute_wer(predictions, references):
    total_errors = 0
    total_words = 0
    for pred, ref in zip(predictions, references):
        dist = levenshtein_distance(pred.split(), ref.split())
        total_errors += dist
        total_words += len(ref.split())
    return (total_errors / total_words) * 100 if total_words > 0 else 0.0
```

## Common pitfalls

- WER is reported separately for development and test sets; results should not be averaged across them without explicit instruction.
- Decoding strategy (Viterbi vs. beam-search with/without LM) significantly impacts WER, so comparisons must match the decoding configuration.
- For Polish, high OOV rates (13%) with constrained lexicons negatively impact LM decoding WER, which is a known dataset-specific artifact.

## Evidence (verbatim from paper)

> The AMs take 80-channel log-mel filterbanks as input and are trained end-to-end with Connectionist Temporal Classification (CTC) loss. SpecAugment is used for data augmentation in training: there are two frequency masks, and ten time masks with maximum time mask ratio of p = 0.1; frequency and time mask parameters are tuned separately for each model; time warping is not used. The maximum frequency bands masked by one frequency mask is 30, and the maximum frames masked by the time mask is 30, too. We use the set of graphemes from the corresponding training set of the language as the token set. The viterbi WER on development and test for all the languages are shown in Table 6.

## Citation

```bibtex
@misc{pratap2020mls,
  title={MLS: A Large-Scale Multilingual Dataset for Speech Research},
  author={Pratap et al. (2020)},
  year={2020},
  note={arXiv:2012.03411}
}
```

- arXiv: 2012.03411