Tacotron2 Mos Eval

---
name: tacotron2-mos-eval
description: Evaluates the perceptual naturalness and quality of text-to-speech synthesis. It probes the model's ability to generate high-fidelity audio waveforms that are indistinguishable from human speech. Use when the user wants to benchmark on Internal US English Test Set, Custom 100-Sentence Test Set, News Headlines Test Set, or asks about evaluating this task. Reports MOS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1712.05884
  bibtex_key: shen2017tacotron2
  confidence: high
---

# tacotron2-mos-eval

> Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions — Shen et al. (2017) (arXiv:1712.05884, 2017)

## What this evaluates

Evaluates the perceptual naturalness and quality of text-to-speech synthesis. It probes the model's ability to generate high-fidelity audio waveforms that are indistinguishable from human speech.

## Datasets

- **Internal US English Test Set** — total ?; splits: test (100)
- **Custom 100-Sentence Test Set** — total 100; splits: test (100)
- **News Headlines Test Set** — total 37; splits: test (37)

## Metrics

- `MOS` **(primary)** — range: other
  - Subjective Mean Opinion Score calculated as the arithmetic mean of human ratings. Each audio sample is rated by at least 8 independent raters on a 1 to 5 scale with 0.5 point increments. Scores are averaged across all raters and all test samples.
- `Side-by-Side Score` — range: other
  - Mean of human ratings on a -3 to 3 scale comparing synthesized audio against ground truth. -3 indicates synthesized is much worse, 3 indicates synthesized is much better.
- `Error Count` — range: other
  - Manual count of specific error categories per 100 sentences: repeated words, mispronunciations, skipped words, unnatural prosody, and end-point prediction failures.

## Input / output format

**Input**: Normalized text sequence (e.g., numbers spelled out like 'sixteen', abbreviations expanded).

**Output**: Audio waveform (speech).

## Scoring recipe

```python
def compute_mos(predictions, gold, num_raters=8):
    all_scores = []
    for pred in predictions:
        rater_scores = [r.rate(pred) for r in range(num_raters)] # scale 1-5, step 0.5
        all_scores.extend(rater_scores)
    return sum(all_scores) / len(all_scores)

def compute_sbs(predictions, gold):
    scores = []
    for pred, gt in zip(predictions, gold):
        score = r.compare(pred, gt) # scale -3 to 3
        scores.append(score)
    return sum(scores) / len(scores)
```

## Common pitfalls

- Confusing teacher-forcing training (ground truth alignment) with autoregressive inference (predicted features), which drastically affects quality.
- Using non-normalized text (e.g., digits, abbreviations) without explicit spelling rules, leading to severe pronunciation errors.
- Assuming MOS generalizes to out-of-domain text; the internal test set shares recurring patterns/words with training data, potentially inflating scores.

## Evidence (verbatim from paper)

> Audio generated on this set are sent to a human rating service similar to Amazon’s Mechanical Turk where each sample is rated by at least 8 raters on a scale from 1 to 5 with 0.5 point increments, from which a subjective mean opinion score (MOS) is calculated.

## Citation

```bibtex
@misc{shen2017tacotron2,
  title={Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions},
  author={Shen et al. (2017)},
  year={2017},
  note={arXiv:1712.05884}
}
```

- arXiv: 1712.05884