Cosyvoice Tts Eval

---
name: cosyvoice-tts-eval
description: Evaluates zero-shot text-to-speech synthesis quality, focusing on content consistency (how well generated speech matches input text) and speaker similarity (how well the cloned voice matches the reference speaker) across English and Chinese. It also probes emotion controllability and the utility of synthesized speech for augmenting ASR training data. Use when the user wants to benchmark on LibriTTS, AISHELL-3, or asks about evaluating this task. Reports WER (%), CER (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.05407
  bibtex_key: du2024cosyvoice
  confidence: high
---

# cosyvoice-tts-eval

> CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens — Du et al. (2024) (arXiv:2407.05407, 2024)

## What this evaluates

Evaluates zero-shot text-to-speech synthesis quality, focusing on content consistency (how well generated speech matches input text) and speaker similarity (how well the cloned voice matches the reference speaker) across English and Chinese. It also probes emotion controllability and the utility of synthesized speech for augmenting ASR training data.

## Datasets

- **LibriTTS** — total ?; splits: test-clean (-1)
- **AISHELL-3** — total ?; splits: test (-1)

## Metrics

- `WER (%)` **(primary)** — range: percent
  - Word Error Rate computed by transcribing the generated speech with Whisper-Large V3 and calculating the edit distance against the ground-truth English text.
- `CER (%)` **(primary)** — range: percent
  - Character Error Rate computed by transcribing the generated speech with Paraformer and calculating the edit distance against the ground-truth Chinese text.
- `Speaker Similarity (SS)` — range: other
  - Raw cosine similarity between speaker embeddings extracted from the generated utterance and the reference prompt using the ERes2Net model.

## Input / output format

**Input**: Input text (and optional style instruction) + reference audio prompt for zero-shot voice cloning.

**Output**: Synthesized speech waveform.

## Scoring recipe

```python
def evaluate(text, prompt_audio, gen_speech, lang):
    asr = whisper_large_v3 if lang == 'en' else paraformer
    transcript = asr.transcribe(gen_speech)
    wer_cer = compute_edit_distance(transcript, text) / len(text)
    gen_emb = eres2net(gen_speech)
    prompt_emb = eres2net(prompt_audio)
    ss = cosine_similarity(gen_emb, prompt_emb)
    return wer_cer, ss
```

## Common pitfalls

- ASR re-ranking drastically lowers WER/CER but is only reported for offline mode, not real-time synthesis.
- Speaker similarity is computed via cosine similarity of ERes2Net embeddings, which may not correlate perfectly with human perceptual ratings.
- Chinese evaluation uses Character Error Rate (CER) instead of WER, making direct cross-lingual metric comparison invalid.
- Random sampling decoding introduces variance; metrics are averaged over 5 seeds and reported with standard deviations.

## Evidence (verbatim from paper)

> We report the word error rate (WER), and the number of insertion, deletion and substation errors. As for the speaker similarity, we employ the ERes2Net model (Chen et al., 2023) to extract speaker embeddings of prompt and generated utterances, and their raw cosine similarity is treated as the speaker similarity.

## Citation

```bibtex
@misc{du2024cosyvoice,
  title={CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens},
  author={Du et al. (2024)},
  year={2024},
  note={arXiv:2407.05407}
}
```

- arXiv: 2407.05407