Whisper Zero Shot Eval

---
name: whisper-zero-shot-eval
description: Evaluates the zero-shot generalization capability of a speech recognition model across diverse English and multilingual domains. It measures robustness to out-of-distribution audio, varying noise levels, and translation tasks without any dataset-specific fine-tuning. Use when the user wants to benchmark on LibriSpeech, Common Voice, Fleurs, CoVoST2, Multilingual LibriSpeech (MLS), VoxPopuli, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2212.04356
  bibtex_key: radford2022whisper
  confidence: high
---

# whisper-zero-shot-eval

> Robust Speech Recognition via Large-Scale Weak Supervision — Radford et al. (2022) (arXiv:2212.04356, 2022)

## What this evaluates

Evaluates the zero-shot generalization capability of a speech recognition model across diverse English and multilingual domains. It measures robustness to out-of-distribution audio, varying noise levels, and translation tasks without any dataset-specific fine-tuning.

## Datasets

- **LibriSpeech** — total ?; splits: test (-1)
- **Common Voice** — total ?; splits: test (-1)
- **Fleurs** — total ?; splits: test (-1)
- **CoVoST2** — total ?; splits: test (-1)
- **Multilingual LibriSpeech (MLS)** — total ?; splits: test (-1)
- **VoxPopuli** — total ?; splits: test (-1)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate calculated as the minimum number of insertions, deletions, and substitutions of words required to transform the predicted transcript into the reference transcript, divided by the number of words in the reference. Applied after a custom text normalization step to ignore formatting/style differences.
- `BLEU` — range: percent
  - Standard n-gram based BLEU score measuring the precision of predicted translations against reference English transcripts.

## Input / output format

**Input**: Raw audio recordings in various formats, languages, and acoustic conditions (e.g., clean speech, noisy environments, meetings, read speech).

**Output**: Raw text transcript corresponding to the spoken audio, without any special formatting or delimiters.

## Scoring recipe

```python
def compute_wer(predictions, references):
    normalized_preds = [normalize_text(p) for p in predictions]
    normalized_refs = [normalize_text(r) for r in references]
    total_errors = 0
    total_words = 0
    for pred, ref in zip(normalized_preds, normalized_refs):
        dist = edit_distance(pred.split(), ref.split())
        total_errors += dist
        total_words += len(ref.split())
    return (total_errors / total_words) * 100
```

## Common pitfalls

- Naive WER calculation heavily penalizes minor formatting or stylistic differences (e.g., contractions, punctuation), requiring the paper's custom text normalizer for fair comparison.
- Zero-shot evaluation explicitly forbids fine-tuning on the target dataset's training split, contrasting with standard supervised benchmarks that often report in-distribution performance.
- Human performance baselines are measured out-of-distribution, while many supervised models are evaluated in-distribution, making direct human-vs-machine comparisons misleading without this protocol.

## Evidence (verbatim from paper)

> Speech recognition research typically evaluates and compares systems based on the word error rate (WER) metric. However, WER, which is based on string edit distance, penalizes all differences between the model’s output and the reference transcript including innocuous differences in transcript style. As a result, systems that output transcripts that would be judged as correct by humans can still have a large WER due to minor formatting differences.

## Citation

```bibtex
@misc{radford2022whisper,
  title={Robust Speech Recognition via Large-Scale Weak Supervision},
  author={Radford et al. (2022)},
  year={2022},
  note={arXiv:2212.04356}
}
```

- arXiv: 2212.04356