Css10 Tts Eval

---
name: css10-tts-eval
description: Evaluates the quality of synthesized speech from single-speaker TTS models trained on the CSS10 datasets across 10 languages. It probes how well models can reproduce natural-sounding audio and accurate pronunciation for held-out test sentences. Use when the user wants to benchmark on CSS10, or asks about evaluating this task. Reports MOS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1903.11269
  bibtex_key: park2019css10
  confidence: high
---

# css10-tts-eval

> CSS10: A Collection of Single Speaker Speech Datasets for 10 Languages — Park et al. (2019) (arXiv:1903.11269, 2019)

## What this evaluates

Evaluates the quality of synthesized speech from single-speaker TTS models trained on the CSS10 datasets across 10 languages. It probes how well models can reproduce natural-sounding audio and accurate pronunciation for held-out test sentences.

## Datasets

- **CSS10** — total ?; splits: train (-1); repo https://github.com/Kyubyong/CSS10

## Metrics

- `MOS` **(primary)** — range: [1, 5]
  - Absolute Category Rating (ACR) scale from 1 to 5. Two sub-metrics are reported: Speech Naturalness and Pronunciation Accuracy. Scores are averaged per model-language pair, with 95% confidence intervals computed using the crowdMOS method.

## Input / output format

**Input**: Text (or phonetic transcription for Chinese/Japanese) of 20 held-out sentences per language, used as input to TTS models for synthesis.

**Output**: Synthesized audio clips corresponding to each test sentence.

## Scoring recipe

```python
# For each language and model:
naturalness_scores = []
pronunciation_scores = []
for worker in mturk_workers:
    if worker.transcribed_reference_correctly():
        naturalness_scores.append(worker.rate_naturalness(audio_clip)) # 1-5
        pronunciation_scores.append(worker.rate_pronunciation(audio_clip)) # 1-5
mean_naturalness = sum(naturalness_scores) / len(naturalness_scores)
mean_pronunciation = sum(pronunciation_scores) / len(pronunciation_scores)
ci_95 = compute_confidence_interval(naturalness_scores, method="crowdMOS")
```

## Common pitfalls

- MOS is a Likert scale; taking the arithmetic mean can be semantically misleading because category distances aren't uniform.
- Worker validation relies on a single reference transcription task, which may not fully guarantee language proficiency for all 10 languages.
- Test sentences are fixed (20 per language) and chosen for phoneme coverage, so results may not generalize to arbitrary text.

## Evidence (verbatim from paper)

> Although it's common to use MOS as a performance metric for TTS models, we recognize that it may not be appropriate to take the mean of Likert scores, because each score belongs to a category and the semantic meaning of these categories need not be evenly distributed along a number line as we (and others) have implied in our rubrics.

## Citation

```bibtex
@misc{park2019css10,
  title={CSS10: A Collection of Single Speaker Speech Datasets for 10 Languages},
  author={Park et al. (2019)},
  year={2019},
  note={arXiv:1903.11269}
}
```

- arXiv: 1903.11269