Deepspeech Wer Eval

---
name: deepspeech-wer-eval
description: This evaluation probes an end-to-end speech recognition system's ability to accurately transcribe conversational telephone speech and robustly handle background noise without phoneme-level modeling or explicit speaker adaptation. It measures transcription accuracy against ground truth references using standard error rates. Use when the user wants to benchmark on Switchboard Hub5’00 (LDC2002S23), Custom Noisy Speech Test Set, or asks about evaluating this task. Reports word error rate (WER).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1412.5567
  bibtex_key: hannun2014deepspeech
  confidence: high
---

# deepspeech-wer-eval

> Deep Speech: Scaling up end-to-end speech recognition — Awni Hannun et al. (2014) (arXiv:1412.5567, 2014)

## What this evaluates

This evaluation probes an end-to-end speech recognition system's ability to accurately transcribe conversational telephone speech and robustly handle background noise without phoneme-level modeling or explicit speaker adaptation. It measures transcription accuracy against ground truth references using standard error rates.

## Datasets

- **Switchboard Hub5’00 (LDC2002S23)** — total ?; splits: SWB (-1), CH (-1), Full (-1)
- **Custom Noisy Speech Test Set** — total 200; splits: clean (100), noisy (100)

## Metrics

- `word error rate (WER)` **(primary)** — range: percent
  - Calculated as (S + D + I) / N * 100, where S=substitutions, D=deletions, I=insertions, and N=number of words in the reference transcription. Standard Levenshtein-based alignment is used.

## Input / output format

**Input**: Log filter bank spectrograms (80 or 160 channels) + energy term, computed over 20ms windows with 10ms stride. Audio is resampled to 8kHz or 16kHz. Per-speaker or per-utterance normalization is applied.

**Output**: Word-level transcription string generated by decoding character-level probability vectors with a language model.

## Scoring recipe

```python
def compute_wer(predictions, references):
    total_errors = 0
    total_words = 0
    for pred, ref in zip(predictions, references):
        edits = levenshtein_distance(ref.split(), pred.split())
        total_errors += edits
        total_words += len(ref.split())
    return (total_errors / total_words) * 100 if total_words > 0 else 0.0
```

## Common pitfalls

- Researchers often split Hub5’00 into 'easy' (Switchboard) and 'hard' (CallHome) subsets and report results only on the easier portion, whereas this protocol requires evaluating on the full combined set.
- When comparing against commercial APIs, the evaluation only includes utterances where all systems returned a non-empty result, which artificially inflates accuracy compared to penalizing API failures as 100% error.

## Evidence (verbatim from paper)

> The predicted probability vectors and language model are then fed into our decoder to yield a word-level transcription, which is compared with the ground truth transcription to yield the word error rate (WER).

## Citation

```bibtex
@misc{hannun2014deepspeech,
  title={Deep Speech: Scaling up end-to-end speech recognition},
  author={Awni Hannun et al. (2014)},
  year={2014},
  note={arXiv:1412.5567}
}
```

- arXiv: 1412.5567