Librispeech Voice Search Wer Eval

---
name: librispeech-voice-search-wer-eval
description: Evaluates the word error rate of streaming speech recognition systems using a first-pass RNN-T model followed by a second-pass rescorer. It probes the ability of parallel Transformer rescoring to improve transcription accuracy while maintaining low-latency streaming constraints on-device. Use when the user wants to benchmark on Librispeech, Google Voice Search, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2008.13093
  bibtex_key: [REDACTED]
  confidence: high
---

# librispeech-voice-search-wer-eval

> Parallel Rescoring with Transformer for Streaming On-Device Speech Recognition — Li et al. (2020) (arXiv:2008.13093, 2020)

## What this evaluates

Evaluates the word error rate of streaming speech recognition systems using a first-pass RNN-T model followed by a second-pass rescorer. It probes the ability of parallel Transformer rescoring to improve transcription accuracy while maintaining low-latency streaming constraints on-device.

## Datasets

- **Librispeech** — total ?; splits: train (-1), test-clean (-1), test-other (-1)
- **Google Voice Search** — total 14000; splits: test (14000)

## Metrics

- `WER` **(primary)** — range: percent
  - Standard Word Error Rate: (S + D + I) / N * 100%, where S=substitutions, D=deletions, I=insertions, and N=number of words in the reference transcription.

## Input / output format

**Input**: Acoustic features and top-4 hypothesis sequences from a first-pass RNN-T model.

**Output**: Rescored probability distributions over 4096 word pieces, yielding a final transcription hypothesis.

## Scoring recipe

```python
def wer(predictions, references):
    # predictions and references are lists of word strings
    # Align predictions to references using standard edit distance
    edit_dist = levenshtein_distance(predictions, references)
    ref_len = len(references)
    if ref_len == 0:
        return 0.0
    return (edit_dist / ref_len) * 100.0
```

## Common pitfalls

- Using causal self-attention during training with ground truth makes the task trivial; random token swapping (15%) is required for CE training.
- N-best size for MWER training must exactly match evaluation (top 4 hypotheses).
- Evaluation must use Exponential Moving Average (EMA) weights, not the final training checkpoint.

## Evidence (verbatim from paper)

> As is shown in Table 1, both the LSTM rescorer and the Transformer rescorer significantly improve the WER of the clean and noisy test sets compared to the RNN-T only model with 10-20% relative improvement, alleviating the limited context problem for the 1st-pass model while still maintaining low-latency streaming recognition. The Transformer rescorer further improves the WER slightly over the LSTM rescorer, and also significantly reduce the 2nd-pass latency, which is studied in detail in Section 4.

## Citation

```bibtex
@misc{li2020parallelrescoring,
  title={Parallel Rescoring with Transformer for Streaming On-Device Speech Recognition},
  author={Li et al. (2020)},
  year={2020},
  note={arXiv:2008.13093}
}
```

- arXiv: 2008.13093