Voxpopuli Eval

---
name: voxpopuli-eval
description: Benchmarks multilingual speech representation learning and semi-supervised ASR/ST performance across multiple languages and domains, measuring phoneme discriminability, recognition accuracy, and translation quality. Use when the user wants to benchmark on VoxPopuli, Common Voice, ZeroSpeech 2017, EuroParl-ST, CoVoST 2, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2101.00390
  bibtex_key: wang2021voxpopuli
  confidence: high
---

# voxpopuli-eval

> VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation — Wang et al. (2021) (arXiv:2101.00390, 2021)

## What this evaluates

Benchmarks multilingual speech representation learning and semi-supervised ASR/ST performance across multiple languages and domains, measuring phoneme discriminability, recognition accuracy, and translation quality.

## Datasets

- **VoxPopuli** — total 400000; splits: unlabeled (400000), transcribed (1800)
- **Common Voice** — total ?; splits: train (3600), dev (1200), test (3600)
- **ZeroSpeech 2017** — total ?; splits: test (10)
- **EuroParl-ST** — total ?; splits: train (-1)
- **CoVoST 2** — total ?; splits: test (-1)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate: the ratio of insertions, deletions, and substitutions to the total number of words in the reference transcript. Lower values indicate better performance.
- `PER` — range: percent
  - Phone Error Rate: identical to WER but computed over phoneme sequences using a phoneme vocabulary. Lower values indicate better performance.
- `BLEU` — range: percent
  - Bilingual Evaluation Understudy: geometric mean of modified n-gram precisions between predicted and reference translations, with brevity penalty. Higher values indicate better performance.
- `ABX discriminability score` — range: other
  - Measures phoneme discriminability of unsupervised features by computing the error rate of a linear classifier distinguishing phoneme B from A and X, where A and X share the same phoneme label. Lower values indicate better feature discriminability.

## Input / output format

**Input**: Raw audio utterances (often segmented by VAD and speaker diarized), optionally paired with transcribed text for fine-tuning or decoding. Features may be 80-dim log-mel filterbanks with CMVN or raw samples.

**Output**: Predicted phoneme sequences, word sequences, or translated text, depending on the task and vocabulary used.

## Scoring recipe

```python
def compute_metric(predictions, gold, task):
    if task == 'ASR':
        return edit_distance(predictions, gold) / len(gold)  # PER or WER based on vocab
    elif task == 'ST':
        return bleu_score(gold, predictions)
    elif task == 'ABX':
        return abx_discriminability(audio_features, phoneme_boundaries)
    return None
```

## Common pitfalls

- Domain mismatch: VoxPopuli pre-training data consists of political oral speeches, while fine-tuning/test sets like Common Voice contain read speech, which can skew generalization metrics.
- Checkpoint selection varies by experiment: best validation loss checkpoint is used for standard fine-tuning, but the average of the 10 best checkpoints is used for self-training experiments.
- Vocabulary choice directly impacts metrics: phoneme vocabularies are required for PER evaluation, while character or subword vocabularies are used for WER and BLEU.

## Evidence (verbatim from paper)

> We follow the setting in Rivière et al. (2020) to evaluate unsupervised speech representations by phoneme discriminability on 3 languages (English, French and Mandarin), and report ABX discriminability score on the 10s test set from ZeroSpeech 2017. ... We report test WER on Common Voice (CV). ... Left: test BLEU for ST models. Right: test WER for ASR models.

## Citation

```bibtex
@misc{wang2021voxpopuli,
  title={VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation},
  author={Wang et al. (2021)},
  year={2021},
  note={arXiv:2101.00390}
}
```

- arXiv: 2101.00390