Librispeech Chime4 Asr Eval

---
name: librispeech-chime4-asr-eval
description: Evaluates the robustness of automatic speech recognition (ASR) models under various noise conditions and signal-to-noise ratios (SNRs) using simulated and real-world noisy speech datasets. Use when the user wants to benchmark on LibriSpeech, CHiME-4, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2304.04974
  bibtex_key: hu2023wav2code
  confidence: high
---

# librispeech-chime4-asr-eval

> Wav2code: Restore Clean Speech Representations via Codebook Lookup for Noise-Robust ASR — Hu et al. (2023) (arXiv:2304.04974, 2023)

## What this evaluates

Evaluates the robustness of automatic speech recognition (ASR) models under various noise conditions and signal-to-noise ratios (SNRs) using simulated and real-world noisy speech datasets.

## Datasets

- **LibriSpeech** — total ?; splits: train (-1), val (-1), test (4200)
- **CHiME-4** — total ?; splits: train (8738), val (3280), test (2640); repo http://spandh.dcs.shef.ac.uk/chime_challenge/CHiME4/index.html

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate, calculated as the number of substitutions, deletions, and insertions required to transform the predicted transcript into the reference transcript, divided by the total number of words in the reference.

## Input / output format

**Input**: Noisy audio waveform (16 kHz) and corresponding reference transcript.

**Output**: Predicted text transcript.

## Scoring recipe

```python
def compute_wer(predictions, references):
    total_errors = 0
    total_words = 0
    for pred, ref in zip(predictions, references):
        total_errors += levenshtein_distance(pred.split(), ref.split())
        total_words += len(ref.split())
    return (total_errors / total_words) * 100 if total_words > 0 else 0.0
```

## Common pitfalls

- The paper explicitly states no language model (LM) is used during inference for LibriSpeech, whereas many baselines and prior works do use one.
- CHiME-4 evaluation uses only the one-channel track for validation and testing, despite training on all six channels.
- WER is averaged across SNR levels in the tables, which can mask performance degradation at lower SNRs.

## Evidence (verbatim from paper)

> “Avg” denotes the averaged WER results on all type-A noise conditions, and “Clean” denotes the WER results on clean test set.

## Citation

```bibtex
@misc{hu2023wav2code,
  title={Wav2code: Restore Clean Speech Representations via Codebook Lookup for Noise-Robust ASR},
  author={Hu et al. (2023)},
  year={2023},
  note={arXiv:2304.04974}
}
```

- arXiv: 2304.04974