Wsj Timit Speech Recognition Eval

---
name: wsj-timit-speech-recognition-eval
description: Evaluates speech recognition models on their ability to accurately transcribe spoken audio into text (WER) and characters (LER). It probes the effectiveness of unsupervised pre-training on raw audio for downstream acoustic modeling and decoding. Use when the user wants to benchmark on TIMIT, WSJ, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1904.05862
  bibtex_key: schneider2019wav2vec
  confidence: high
---

# wsj-timit-speech-recognition-eval

> wav2vec: Unsupervised Pre-training for Speech Recognition — Schneider et al. (2019) (arXiv:1904.05862, 2019)

## What this evaluates

Evaluates speech recognition models on their ability to accurately transcribe spoken audio into text (WER) and characters (LER). It probes the effectiveness of unsupervised pre-training on raw audio for downstream acoustic modeling and decoding.

## Datasets

- **TIMIT** — total ?; splits: train (-1), dev (-1), test (-1)
- **WSJ** — total 81; splits: si284 (-1), nov93dev (-1), nov92 (-1)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate: (Substitutions + Deletions + Insertions) / Total Reference Words. Expressed as a percentage.
- `LER` — range: percent
  - Letter Error Rate: (Substitutions + Deletions + Insertions) / Total Reference Letters. Expressed as a percentage.

## Input / output format

**Input**: 80-dimensional log-mel filterbank coefficients extracted from raw audio using a 25ms window with 10ms stride, or pre-trained contextual embeddings.

**Output**: Sequence of words or characters decoded via beam search, optimized using acoustic model probabilities, language model scores, word penalty, and silence penalty.

## Scoring recipe

```python
def compute_wer(reference, hypothesis):
    ref_words = reference.split()
    hyp_words = hypothesis.split()
    if not ref_words:
        return 0.0
    dist = levenshtein_distance(ref_words, hyp_words)
    return (dist / len(ref_words)) * 100
```

## Common pitfalls

- Language model hyperparameters (α, β, γ) and beam search settings are tuned separately for word-based vs character-based LMs, significantly affecting final scores.
- Evaluation splits for WSJ are nov92 (test) and nov93dev (validation), which differ from the standard WSJ 0.38/0.92 hour sets used in other benchmarks.
- Pre-training involves cropping audio sequences, removing ~25% of training data, which can impact downstream performance if not accounted for.

## Evidence (verbatim from paper)

> Final models are evaluated in terms of both word error rate (WER) and letter error rate (LER).

## Citation

```bibtex
@misc{schneider2019wav2vec,
  title={wav2vec: Unsupervised Pre-training for Speech Recognition},
  author={Schneider et al. (2019)},
  year={2019},
  note={arXiv:1904.05862}
}
```

- arXiv: 1904.05862