Voice Search Wer Eval

---
name: voice-search-wer-eval
description: Evaluates speech recognition accuracy and latency trade-offs for streaming vs. non-streaming decoding on a proprietary voice-search dataset. It probes the model's ability to maintain low word error rate while minimizing output delay and computational overhead. Use when the user wants to benchmark on Voice Search, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2010.03192
  bibtex_key: tripathi2020transformertransducer
  confidence: high
---

# voice-search-wer-eval

> Transformer Transducer: One Model Unifying Streaming and Non-streaming Speech Recognition — Tripathi et al. (2020) (arXiv:2010.03192, 2020)

## What this evaluates

Evaluates speech recognition accuracy and latency trade-offs for streaming vs. non-streaming decoding on a proprietary voice-search dataset. It probes the model's ability to maintain low word error rate while minimizing output delay and computational overhead.

## Datasets

- **Voice Search** — total ?; splits: test (14000)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate: the percentage of words incorrectly recognized relative to the reference transcript.
- `RTF` — range: other
  - Real-Time Factor: the ratio of processing time to audio duration.
- `Alignment Delay` — range: other
  - Mean difference in word alignment timestamps between reference and model: D = (1/N) * sum(T_ref - T_Y).

## Input / output format

**Input**: 30ms logmel energy features (128-dim, stacked 4x, subsampled 3x) derived from speech waveforms using a 32ms window with 10ms shift.

**Output**: Transcribed text sequence and word alignment timestamps.

## Scoring recipe

```python
def compute_wer(predictions, references):
    errors = sum(edit_distance(p, r) for p, r in zip(predictions, references))
    total = sum(len(r) for r in references)
    return 100 * errors / total

def compute_alignment_delay(ref_times, model_times):
    return sum(t_ref - t_model for t_ref, t_model in zip(ref_times, model_times)) / len(ref_times)
```

## Common pitfalls

- Confusing fixed lookahead latency (e.g., 240ms buffer) with actual word alignment delay, which varies by model configuration.
- Reporting RTF without specifying hardware (TPU vs CPU) and batch size, as inference speed heavily depends on these factors.
- Overlooking that constrained alignment training improves delay but degrades low-latency WER.

## Evidence (verbatim from paper)

> Table 1 shows WER and benchmarking results for different label encoder architectures.

## Citation

```bibtex
@misc{tripathi2020transformertransducer,
  title={Transformer Transducer: One Model Unifying Streaming and Non-streaming Speech Recognition},
  author={Tripathi et al. (2020)},
  year={2020},
  note={arXiv:2010.03192}
}
```

- arXiv: 2010.03192