Robust Asr Wer Eval

---
name: robust-asr-wer-eval
description: Evaluates the robustness of end-to-end automatic speech recognition models against various stationary and non-stationary noise types at different signal-to-noise ratios (SNR). It also measures the degradation of recognition accuracy on clean speech when noise-adaptation techniques are applied. Use when the user wants to benchmark on Custom noisy speech dataset (7 noise types), or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2102.06237
  bibtex_key: prasad2021robust
  confidence: high
---

# robust-asr-wer-eval

> An Investigation of End-to-End Models for Robust Speech Recognition — Prasad et al. (2021) (arXiv:2102.06237, 2021)

## What this evaluates

Evaluates the robustness of end-to-end automatic speech recognition models against various stationary and non-stationary noise types at different signal-to-noise ratios (SNR). It also measures the degradation of recognition accuracy on clean speech when noise-adaptation techniques are applied.

## Datasets

- **Custom noisy speech dataset (7 noise types)** — total ?; splits: test (-1); repo https://github.com/archiki/Robust-E2E-ASR

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate, calculated as the minimum number of word edits (insertions, deletions, substitutions) required to change the predicted transcription into the reference transcription, divided by the total number of words in the reference. Reported as a percentage.

## Input / output format

**Input**: Noisy speech audio recordings mixed with one of seven noise types (Babble, Airport/Station, AC/Vacuum, Cafe, Traffic, Metro, Car) at SNR levels of 0, 5, 10, 15, or 20 dB, plus clean speech recordings.

**Output**: Predicted text transcription of the audio input.

## Scoring recipe

```python
def compute_wer(predictions, references):
    total_words = sum(len(ref.split()) for ref in references)
    if total_words == 0: return 0.0
    edits = 0
    for pred, ref in zip(predictions, references):
        edits += levenshtein_distance(pred.split(), ref.split())
    return (edits / total_words) * 100
```

## Common pitfalls

- Ignoring clean speech WER degradation when evaluating noise-adaptation techniques, as some methods (e.g., SE-VCAE, AvT) significantly harm clean speech performance.
- Failing to distinguish between stationary noise types (Car, Metro, Traffic) and non-stationary noise types (Babble, Airport/Station, Cafe, AC/Vacuum), as adaptation strategies perform differently on each.
- Assuming higher objective speech enhancement scores directly translate to lower ASR WER, as content distortion in enhanced speech can negatively impact recognition.

## Evidence (verbatim from paper)

> Table 2 lists an exhaustive comparison of all previously mentioned techniques on seven different noise types and five different SNR values. We also report the WER on clean speech to observe degradation with each noise adaptation technique in place.

## Citation

```bibtex
@misc{prasad2021robust,
  title={An Investigation of End-to-End Models for Robust Speech Recognition},
  author={Prasad et al. (2021)},
  year={2021},
  note={arXiv:2102.06237}
}
```

- arXiv: 2102.06237