Gigaspeech Eval

---
name: gigaspeech-eval
description: Evaluates automatic speech recognition (ASR) systems on a large-scale, multi-domain English corpus containing both read and spontaneous speech. It benchmarks transcription accuracy across tiered training subsets and professionally re-transcribed evaluation sets using word error rate. Use when the user wants to benchmark on GigaSpeech, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2106.06909
  bibtex_key: chen2021gigaspeech
  confidence: high
---

# gigaspeech-eval

> GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio — Chen et al. (2021) (arXiv:2106.06909, 2021)

## What this evaluates

Evaluates automatic speech recognition (ASR) systems on a large-scale, multi-domain English corpus containing both read and spontaneous speech. It benchmarks transcription accuracy across tiered training subsets and professionally re-transcribed evaluation sets using word error rate.

## Datasets

- **GigaSpeech** — total 10000; splits: train (-1), dev (-1), test (-1); repo https://github.com/SpeechColab/GigaSpeech

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate, calculated as the number of insertions, deletions, and substitutions required to transform the predicted transcription into the reference transcription, divided by the total number of words in the reference.

## Input / output format

**Input**: Audio recordings (typically processed into acoustic features such as 80-dim log Fbanks)

**Output**: Text transcription corresponding to the audio input

## Scoring recipe

```python
def calculate_wer(predictions, references):
    total_errors = 0
    total_words = 0
    for pred, ref in zip(predictions, references):
        edit_dist = levenshtein_distance(pred.split(), ref.split())
        total_errors += edit_dist
        total_words += len(ref.split())
    return (total_errors / total_words) * 100 if total_words > 0 else 0.0
```

## Common pitfalls

- Baseline results are toolkit-specific and explicitly stated as not suitable for cross-toolkit performance comparison.
- The XS training subset (10 hours) is intended only for system building and debugging, not for evaluating strong model performance.
- Evaluation sets (DEV and TEST) are professionally re-transcribed to ensure high fidelity, unlike training data which relies on forced alignment with strict WER caps.

## Evidence (verbatim from paper)

> Table 4: GigaSpeech baselines for the XL training subset (WER in %) ... Table 5: Kaldi baselines for GigaSpeech training subsets (WER in %)

## Citation

```bibtex
@misc{chen2021gigaspeech,
  title={GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio},
  author={Chen et al. (2021)},
  year={2021},
  note={arXiv:2106.06909}
}
```

- arXiv: 2106.06909