Asr Robustness Eval

---
name: asr-robustness-eval
description: Evaluates the robustness and cross-domain generalization of Automatic Speech Recognition (ASR) models by measuring Word Error Rate (WER) across multiple public and in-house English speech datasets with varying acoustic conditions, sampling rates, and speech types. Use when the user wants to benchmark on LibriSpeech, SwitchBoard & Fisher, WSJ, Common Voice, TED-LIUM v3, Robust Video, CHiME-6, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2010.11745
  bibtex_key: likhomanenko2020rethinking
  confidence: high
---

# asr-robustness-eval

> Rethinking Evaluation in ASR: Are Our Models Robust Enough? — Likhomanenko et al. (2020) (arXiv:2010.11745, 2020)

## What this evaluates

Evaluates the robustness and cross-domain generalization of Automatic Speech Recognition (ASR) models by measuring Word Error Rate (WER) across multiple public and in-house English speech datasets with varying acoustic conditions, sampling rates, and speech types.

## Datasets

- **LibriSpeech** — total ?; splits: train (960), dev-clean (5), dev-other (5), test-clean (5), test-other (5)
- **SwitchBoard & Fisher** — total ?; splits: train (2300), dev (6), test (4)
- **WSJ** — total ?; splits: train (81), dev (1), test (1)
- **Common Voice** — total ?; splits: train (693), dev (27), test (26)
- **TED-LIUM v3** — total ?; splits: train (452), dev (2), test (3)
- **Robust Video** — total ?; splits: train (5000), dev (14), test (75)
- **CHiME-6** — total 40; splits: dev (-1), eval (-1)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate computed as (Substitutions + Deletions + Insertions) / Total Words in reference transcript. Evaluated using the standard Kaldi recipe for data processing and alignment.

## Input / output format

**Input**: 16kHz audio resampled from original datasets, processed into 80 log-mel spectrogram features (25ms window, 10ms shift), normalized to zero mean and unit variance per input sequence.

**Output**: Transcribed text sequence using a fixed token set of 26 English letters, apostrophe, and word boundary token; punctuation removed for WSJ.

## Scoring recipe

```python
def compute_wer(reference, hypothesis):
    ref_words = reference.split()
    hyp_words = hypothesis.split()
    edit_dist = levenshtein_distance(ref_words, hyp_words)
    total_words = len(ref_words)
    if total_words == 0:
        return 0.0
    return (edit_dist / total_words) * 100
```

## Common pitfalls

- The reported 'average WER' is calculated as the average of per-dataset average WERs, not a global average across all test words, to prevent dataset size bias.
- Audio must be resampled to 16kHz and features normalized per sequence before evaluation to ensure consistency across datasets with different original sampling rates.
- Punctuation is stripped from transcriptions (e.g., WSJ) and tokenization uses a fixed alphabet set rather than word-pieces, which affects alignment and error counting.

## Evidence (verbatim from paper)

> Table 4: WER of models evaluated on all datasets (downsampled to 16kHz) with a greedy decoding and no LM (top row), with in-domain n-gram LM beam-search decoding (middle row) and with additional second-pass rescoring by in-domain Transformer LM (below row). Joint models are also decoded with CC LM with either a single-pass (top row) or a two-pass (bottom row) decoding. State-of-the-art (SOTA) models are given from WSJ [Hadian et al., 2018], TEDLIUM [Zhou et al., 2020], LibriSpeech [Gulati et al., 2020], SwitchBoard & Fisher [Han et al., 2017]. The SOTA models are all decoded with in-domain LMs. The average is computed as average of averages for LibriSpeech's validations/tests, and SwitchBoard's tests (SB, CH) sets, so as not to weight them more heavily.

## Citation

```bibtex
@misc{likhomanenko2020rethinking,
  title={Rethinking Evaluation in ASR: Are Our Models Robust Enough?},
  author={Likhomanenko et al. (2020)},
  year={2020},
  note={arXiv:2010.11745}
}
```

- arXiv: 2010.11745