Clip Tts Eval

---
name: clip-tts-eval
description: Evaluates the naturalness and quality of synthesized speech across single-speaker, multi-speaker, and multi-emotion TTS models. It measures how closely generated audio matches human ground truth in terms of overall speech quality and emotional similarity. Use when the user wants to benchmark on Baker, AISHELL3, LJSpeech, LibriTTS, Emotional Speech Dataset (ESD), or asks about evaluating this task. Reports MOS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.18889
  bibtex_key: liu2025cliptts
  confidence: high
---

# clip-tts-eval

> Clip-TTS: Contrastive Text-content and Mel-spectrogram, A High-Quality Text-to-Speech Method based on Contextual Semantic Understanding — Liu (2025) (arXiv:2502.18889, 2025)

## What this evaluates

Evaluates the naturalness and quality of synthesized speech across single-speaker, multi-speaker, and multi-emotion TTS models. It measures how closely generated audio matches human ground truth in terms of overall speech quality and emotional similarity.

## Datasets

- **Baker** — total 10000; splits: test (-1)
- **AISHELL3** — total ?; splits: test (-1)
- **LJSpeech** — total 13100; splits: test (-1)
- **LibriTTS** — total ?; splits: test (-1)
- **Emotional Speech Dataset (ESD)** — total ?; splits: test (-1)

## Metrics

- `MOS` **(primary)** — range: [1, 5]
  - Mean Opinion Score: human listeners rate the overall naturalness and quality of synthesized speech on a standard 1–5 scale. Scores are averaged across listeners with 95% confidence intervals reported.
- `CMOS` — range: [-3, 3]
  - Comparative Mean Opinion Score: listeners compare synthesized speech to a reference/original speech on a 6-level scale from -3 to +3. Positive scores indicate the synthesized speech is better/more similar, negative scores indicate it is worse/less similar.

## Input / output format

**Input**: Text transcription (and optional emotion/style conditioning) provided to the TTS model.

**Output**: Mel-spectrograms generated by the model, subsequently converted to waveform audio via a pre-trained HiFi-GAN vocoder.

## Scoring recipe

```python
def compute_mos(predictions, gold):
    ratings = [r for r in predictions if r is not None]
    return sum(ratings) / len(ratings) if ratings else 0.0

def compute_cmos(predictions, gold):
    ratings = [r for r in predictions if r is not None]
    return sum(ratings) / len(ratings) if ratings else 0.0
```

## Common pitfalls

- MOS scores are heavily influenced by the inherent recording quality and speaker characteristics of the source dataset, not just the model's architecture.
- CMOS measures emotional similarity to the original speech rather than absolute synthesis quality, so negative values do not necessarily mean poor audio fidelity.
- Human evaluation protocols vary widely in listener count, instructions, and confidence interval calculation, making cross-paper comparisons difficult without standardized guidelines.

## Evidence (verbatim from paper)

> The speech quality evaluation method I used is the MOS [37] score. ... On the emotional speech dataset, I used CMOS scores to evaluate the differences between the speech generated by Clip-TTS for different emotions and the original speech. The CMOS scores are divided into 6 levels, ranging from -3 to 3. A score between -1 and -3 indicates that the quality of the synthesized speech is progressively worse than the original speech, while a score between +1 and +3 indicates the opposite.

## Citation

```bibtex
@misc{liu2025cliptts,
  title={Clip-TTS: Contrastive Text-content and Mel-spectrogram, A High-Quality Text-to-Speech Method based on Contextual Semantic Understanding},
  author={Liu (2025)},
  year={2025},
  note={arXiv:2502.18889}
}
```

- arXiv: 2502.18889