Librispeech Eval

---
name: librispeech-eval
description: Evaluates speech recognition performance under varying amounts of labeled data (1h, 10h, 100h) and different model sizes. It probes the ability of self-supervised speech models to adapt to downstream transcription tasks with limited supervision. Use when the user wants to benchmark on LibriSpeech, or asks about evaluating this task. Reports Word Error Rate (WER).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.02720
  bibtex_key: shi2023multiresolutionhubert
  confidence: high
---

# librispeech-eval

> Multi-resolution HuBERT: Multi-resolution Speech Self-Supervised Learning with Masked Unit Prediction — Shi et al. (2023) (arXiv:2310.02720, 2023)

## What this evaluates

Evaluates speech recognition performance under varying amounts of labeled data (1h, 10h, 100h) and different model sizes. It probes the ability of self-supervised speech models to adapt to downstream transcription tasks with limited supervision.

## Datasets

- **LibriSpeech** — total ?; splits: dev-clean (-1), dev-other (-1), test-clean (-1), test-other (-1); HF `librispeech_asr`

## Metrics

- `Word Error Rate (WER)` **(primary)** — range: percent
  - Percentage of words incorrectly recognized compared to the reference transcript. Calculated as (Substitutions + Deletions + Insertions) / Total Words. Lower is better.

## Input / output format

**Input**: Audio waveform and reference transcript.

**Output**: Predicted word sequence.

## Scoring recipe

```python
def compute_wer(predictions, references):
    # Align predicted and reference word sequences
    # Count substitutions (S), deletions (D), insertions (I)
    # N = total words in reference
    wer = (S + D + I) / N * 100
    return wer
```

## Common pitfalls

- Evaluating on different labeled data subsets (1h, 10h, 100h) changes the fine-tuning regime, not just the test set.
- Results are reported with and without a 4-gram language model joint decoding, which significantly impacts WER and must be specified.

## Evidence (verbatim from paper)

> We evaluate the proposed methods using a variety of speech processing tasks, segmented into four key categories: speech recognition on the LibriSpeech benchmarks... Our chosen evaluation metric is the Word Error Rate (WER).

## Citation

```bibtex
@misc{shi2023multiresolutionhubert,
  title={Multi-resolution HuBERT: Multi-resolution Speech Self-Supervised Learning with Masked Unit Prediction},
  author={Shi et al. (2023)},
  year={2023},
  note={arXiv:2310.02720}
}
```

- arXiv: 2310.02720