Emergent Tts Eval

---
name: emergent-tts-eval
description: Evaluates text-to-speech models on six complex linguistic and prosodic dimensions: emotions, paralinguistics, syntactic complexity, questions, foreign words, and complex pronunciation. It uses a model-as-a-judge framework to measure pairwise preference against a baseline, alongside automated speech quality metrics. Use when the user wants to benchmark on EmergentTTS-Eval, or asks about evaluating this task. Reports win-rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.23009
  bibtex_key: manku2025emergentts
  confidence: high
---

# emergent-tts-eval

> EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge — Manku et al. (2025) (arXiv:2505.23009, 2025)

## What this evaluates

Evaluates text-to-speech models on six complex linguistic and prosodic dimensions: emotions, paralinguistics, syntactic complexity, questions, foreign words, and complex pronunciation. It uses a model-as-a-judge framework to measure pairwise preference against a baseline, alongside automated speech quality metrics.

## Datasets

- **EmergentTTS-Eval** — total 1645; splits: test (1645); repo https://github.com/boson-ai/EmergentTTS-Eval-public

## Metrics

- `win-rate` **(primary)** — range: percent
  - Percentage of pairwise comparisons where the evaluated model's audio output is preferred over the baseline (gpt-4o-mini-tts, Alloy voice) by a judge LALM. Computed per category and overall.
- `WER` — range: percent
  - Word Error Rate computed using Whisper-v3-large to measure transcription accuracy of the generated audio.
- `MOS` — range: other
  - Mean Opinion Score estimated using a fine-tuned wav2vec2.0 model to predict human-like quality ratings.

## Input / output format

**Input**: Text utterance. For 'Strong Prompting', input is augmented with category-specific instructions (e.g., 'be emotionally expressive') passed via style descriptors or user messages depending on the model type.

**Output**: Audio waveform (TTS output).

## Scoring recipe

```python
wins = 0
total = 0
for audio_gen, audio_baseline in zip(generated_audios, baseline_audios):
    judge_response = judge_lalm.compare(audio_gen, audio_baseline)
    if judge_response == "preferred_gen":
        wins += 1
    total += 1
win_rate = (wins / total) * 100
wer = whisper_v3_large.transcribe(audio_gen).word_error_rate
mos = wav2vec2_mos_model.predict(audio_gen)
```

## Common pitfalls

- Win-rate scores are highly sensitive to the specific voice used by the evaluated TTS model; results can vary significantly across different voice clones.
- Judge parsing failures (due to incorrect JSON formatting or token limits in reasoning loops) must be filtered out, as they can artificially deflate win-rates.
- Performance gains from 'Strong Prompting' are substantial for some models, so comparisons must explicitly state whether basic or strong prompting was used.

## Evidence (verbatim from paper)

> In addition to the win-rate as described in Section 3.2, we follow standard practice by computing WER using Whisper-v3-large [27], and MOS scores are calculated using a fine-tuned wav2vec2.0 model [7].

## Citation

```bibtex
@misc{manku2025emergentts,
  title={EmergentTTS-Eval: Evaluating TTS Models on Complex Prosodic, Expressiveness, and Linguistic Challenges Using Model-as-a-Judge},
  author={Manku et al. (2025)},
  year={2025},
  note={arXiv:2505.23009}
}
```

- arXiv: 2505.23009