Zero Shot Tts Eval

---
name: zero-shot-tts-eval
description: Evaluates zero-shot text-to-speech synthesis capability across English and Chinese. It measures intelligibility, speaker similarity, and naturalness against reference prompts without fine-tuning on target speakers. Use when the user wants to benchmark on Seed-TTS test-en, Seed-TTS test-zh, AISHELL-3 test set, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.04720
  bibtex_key: wang2025m3tts
  confidence: high
---

# zero-shot-tts-eval

> M3-TTS: Multi-modal DiT Alignment & Mel-latent for Zero-shot High-fidelity Speech Synthesis — Wang et al. (2025) (arXiv:2512.04720, 2025)

## What this evaluates

Evaluates zero-shot text-to-speech synthesis capability across English and Chinese. It measures intelligibility, speaker similarity, and naturalness against reference prompts without fine-tuning on target speakers.

## Datasets

- **Seed-TTS test-en** — total 1088; splits: test (1088)
- **Seed-TTS test-zh** — total 2020; splits: test (2020)
- **AISHELL-3 test set** — total 1000; splits: test (1000)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate measured using Whisper-large-v3 for English and Paraformer-zh for Chinese.
- `SIM-o` — range: [0, 1]
  - Speaker similarity computed as cosine similarity between WavLM-based ECAPATDNN embeddings extracted from the reference prompt and the synthesized speech.
- `UTMOS` — range: [1, 5]
  - Automatic naturalness score estimated by the UTMOS model.
- `NMOS` — range: [1, 5]
  - Human naturalness Mean Opinion Score collected via evaluation surveys.
- `QMOS` — range: [1, 5]
  - Human quality Mean Opinion Score collected via evaluation surveys.

## Input / output format

**Input**: Target text transcription and a reference audio prompt containing the target speaker's voice.

**Output**: Synthesized audio waveform matching the target speaker and text content.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    # WER
    wer = asr_backend.transcribe(predictions['audio']).word_error_rate(gold['text'])
    # SIM-o
    emb_prompt = ecapatdnn_wavlm.encode(gold['prompt_audio'])
    emb_pred = ecapatdnn_wavlm.encode(predictions['audio'])
    sim_o = cosine_similarity(emb_prompt, emb_pred)
    # UTMOS
    utmos = utmos_model.predict(predictions['audio'])
    # NMOS / QMOS
    nmos = average_human_mos(predictions['audio'], type='naturalness')
    qmos = average_human_mos(predictions['audio'], type='quality')
    return {'WER': wer, 'SIM-o': sim_o, 'UTMOS': utmos, 'NMOS': nmos, 'QMOS': qmos}
```

## Common pitfalls

- Using different ASR backends for English and Chinese WER can introduce cross-lingual bias if not normalized or compared carefully.
- SIM-o relies on WavLM-based ECAPATDNN embeddings which may not perfectly correlate with human perception of speaker identity, especially for non-English voices.
- UTMOS and MOS scores are automatic/subjective proxies that may not fully capture real-world listening experience or artifacts like clipping.

## Evidence (verbatim from paper)

> Intelligibility is measured by WER using ASR backends: Whisper-large-v3 [29] for English and Paraformer-zh [30] for Chinese. Speaker similarity (SIM-o) is computed as the cosine similarity between WavLM-based ECAPATDNN embeddings [31] extracted from the prompt and synthesized speech. Naturalness is estimated by UTMOS [32]. For human evaluation, we report NMOS (naturalness MOS) and QMOS (quality MOS).

## Citation

```bibtex
@misc{wang2025m3tts,
  title={M3-TTS: Multi-modal DiT Alignment & Mel-latent for Zero-shot High-fidelity Speech Synthesis},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2512.04720}
}
```

- arXiv: 2512.04720