Wavenet Audio Eval

---
name: wavenet-audio-eval
description: Evaluates the capability of autoregressive generative models to synthesize high-quality raw audio waveforms for speech and music, and to perform discriminative tasks like speech recognition directly on raw audio without intermediate feature extraction. Use when the user wants to benchmark on VCTK (CSTR Voice Cloning Toolkit), Google TTS (NA English & Mandarin), MagnaTagATune, YouTube Piano, TIMIT, or asks about evaluating this task. Reports Mean Opinion Score (MOS).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1609.03499
  bibtex_key: vandenoord2016wavenet
  confidence: high
---

# wavenet-audio-eval

> WaveNet: A Generative Model for Raw Audio — van den Oord et al. (2016) (arXiv:1609.03499, 2016)

## What this evaluates

Evaluates the capability of autoregressive generative models to synthesize high-quality raw audio waveforms for speech and music, and to perform discriminative tasks like speech recognition directly on raw audio without intermediate feature extraction.

## Datasets

- **VCTK (CSTR Voice Cloning Toolkit)** — total ?; splits: train (-1)
- **Google TTS (NA English & Mandarin)** — total ?; splits: train (-1)
- **MagnaTagATune** — total ?; splits: train (-1)
- **YouTube Piano** — total ?; splits: train (-1)
- **TIMIT** — total ?; splits: train (-1), test (-1)

## Metrics

- `Mean Opinion Score (MOS)` **(primary)** — range: other
  - Average rating on a 5-point Likert scale (1: Bad to 5: Excellent) assigned by human listeners to judge the naturalness of synthesized speech samples.
- `Subjective Paired Comparison` — range: percent
  - Percentage of times a WaveNet-generated sample is preferred over a baseline sample in forced-choice listening tests.
- `Phone Error Rate (PER)` — range: percent
  - Standard speech recognition metric calculated as (substitutions + deletions + insertions) / total phonemes, expressed as a percentage.

## Input / output format

**Input**: Raw audio waveforms quantized via μ-law companding (8-bit categorical levels), optionally conditioned on speaker IDs (one-hot), linguistic features, log fundamental frequency (F0), or music genre/instrument tags (binary vector).

**Output**: Autoregressively generated next audio sample (categorical distribution over μ-law levels), or synthesized full audio waveform, or phoneme/class labels for recognition tasks.

## Scoring recipe

```python
def compute_mos(ratings):
    return sum(ratings) / len(ratings)

def compute_paired_comparison(wins, total_pairs):
    return (wins / total_pairs) * 100

def compute_per(errors, total_phonemes):
    return (errors / total_phonemes) * 100

# MOS: Average 1-5 scale ratings from human listeners
# Paired: % preference over baseline
# PER: Standard ASR error rate on TIMIT test set
```

## Common pitfalls

- Receptive field limitations (e.g., 240ms for TTS) cause unnatural prosody and stress patterns due to insufficient long-range dependency modeling.
- Subjective MOS scores are heavily influenced by quantization; μ-law (8-bit) naturally scores lower than 16-bit linear PCM, requiring careful baseline comparison.
- Music generation lacks second-to-second consistency in genre/instrumentation despite harmonic quality, often misinterpreted as model failure rather than a receptive field constraint.

## Evidence (verbatim from paper)

> To evaluate the performance of WaveNets for the TTS task, subjective paired comparison tests and mean opinion score (MOS) tests were conducted. In the paired comparison tests, after listening to each pair of samples, the subjects were asked to choose which they preferred, though they could choose "neutral" if they did not have any preference. In the MOS tests, after listening to each stimulus, the subjects were asked to rate the naturalness of the stimulus in a five-point Likert scale score (1: Bad, 2: Poor, 3: Fair, 4: Good, 5: Excellent). Please refer to Appendix B for details.

## Citation

```bibtex
@misc{vandenoord2016wavenet,
  title={WaveNet: A Generative Model for Raw Audio},
  author={van den Oord et al. (2016)},
  year={2016},
  note={arXiv:1609.03499}
}
```

- arXiv: 1609.03499