Emilia Tts Eval

---
name: emilia-tts-eval
description: Evaluates the effectiveness of the Emilia dataset for Text-to-Speech generation by comparing models trained on Emilia versus MLS. It probes intelligibility, speaker similarity, and naturalness across formal and spontaneous speaking styles in both English and multilingual settings. Use when the user wants to benchmark on LibriSpeech-Test, Emilia-Test, Aishell-3, Common Voice, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.05361
  bibtex_key: he2024emilia
  confidence: high
---

# emilia-tts-eval

> Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation — He et al. (2024) (arXiv:2407.05361, 2024)

## What this evaluates

Evaluates the effectiveness of the Emilia dataset for Text-to-Speech generation by comparing models trained on Emilia versus MLS. It probes intelligibility, speaker similarity, and naturalness across formal and spontaneous speaking styles in both English and multilingual settings.

## Datasets

- **LibriSpeech-Test** — total 1200; splits: test (1200)
- **Emilia-Test** — total 600; splits: test (600)
- **Aishell-3** — total ?; splits: test (-1)
- **Common Voice** — total ?; splits: test (-1)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate calculated by comparing the ASR transcription of the synthesized speech against the ground truth input text. Lower is better.
- `SIM-O` — range: [0, 1]
  - Cosine similarity between speaker embeddings extracted from the generated speech and the reference speech prompt using a WavLM-TDCNN model. Higher is better.
- `FSD` — range: other
  - Fréchet Speech Distance measuring the distributional similarity between generated and real speech samples in an emotion2vec feature space. Lower is better.
- `CMOS` — range: [-3, 3]
  - Comparative Mean Opinion Score from human judges rating the naturalness of synthesized speech relative to a reference prompt. Higher is better.
- `SMOS` — range: [1, 5]
  - Similarity Mean Opinion Score from human judges rating speaker similarity between generated speech and the reference prompt. Higher is better.

## Input / output format

**Input**: Text transcript and a reference audio prompt (for zero-shot voice cloning).

**Output**: Synthesized speech audio waveform.

## Scoring recipe

```python
# Objective metrics
pred_text = asr_model(generated_audio)
wer = compute_wer(gold_text, pred_text)

gold_emb = wavlm_tdcnn(gold_prompt_audio)
pred_emb = wavlm_tdcnn(generated_audio)
sim_o = cosine_similarity(gold_emb, pred_emb)

gold_feats = emotion2vec(gold_real_audio)
pred_feats = emotion2vec(generated_audio)
fsd = frechet_distance(gold_feats, pred_feats)

# Subjective metrics (human evaluation)
cmos = mean([judge.compare(generated_audio, gold_prompt_audio) for judge in judges])
smos = mean([judge.rate_similarity(generated_audio, gold_prompt_audio) for judge in judges])
```

## Common pitfalls

- Confusing SIM-O (objective embedding similarity) with SMOS (subjective human rating for similarity), as both assess speaker likeness but use different scales and methods.
- Using inconsistent ASR backbones for WER calculation across datasets (HuBERT-Large for LibriSpeech vs. Whisper-Medium for others), which can introduce measurement bias when comparing cross-dataset results.
- FSD is computed in emotion2vec feature space rather than raw audio or standard mel-spectrogram space, making absolute values incomparable to standard FSD benchmarks.

## Evidence (verbatim from paper)

> For the objective evaluation, we consider the following aspects: (1) Intelligibility: Measured by the Word Error Rate (WER) of the synthesized speech's transcription compared to the input text. For LibriSpeech-Test, we use a finetuned HuBERT-Large ASR model.[12] For other testsets, we use the Whisper-Medium model.[13] (2) Coherence: Assessed by speaker similarity between generated speech and the speech prompt using the WavLM-TDCNN speaker embedding model. We report similarity to the original speech prompt (SIM-O). (3) Naturalness: Evaluated using the Fréchet Speech Distance (FSD), which measures the similarity between the distributions of generated and real samples in a feature space.

## Citation

```bibtex
@misc{he2024emilia,
  title={Emilia: An Extensive, Multilingual, and Diverse Speech Dataset for Large-Scale Speech Generation},
  author={He et al. (2024)},
  year={2024},
  note={arXiv:2407.05361}
}
```

- arXiv: 2407.05361