Seed Tts Eval

---
name: seed-tts-eval
description: Evaluates zero-shot voice conversion systems on linguistic preservation, speaker identity retention, and audio naturalness across English, Chinese, and cross-lingual settings. It also measures computational efficiency and latency for both streaming and offline inference modes. Use when the user wants to benchmark on Seed-TTS-Eval, or asks about evaluating this task. Reports WER (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.12456
  bibtex_key: zheng2026xvc
  confidence: high
---

# seed-tts-eval

> X-VC: Zero-shot Streaming Voice Conversion in Codec Space — Zheng et al. (2026) (arXiv:2604.12456, 2026)

## What this evaluates

Evaluates zero-shot voice conversion systems on linguistic preservation, speaker identity retention, and audio naturalness across English, Chinese, and cross-lingual settings. It also measures computational efficiency and latency for both streaming and offline inference modes.

## Datasets

- **Seed-TTS-Eval** — total ?; splits: test-en (1000), test-zh (1000), cross-en-zh (1000), cross-zh-en (1000)

## Metrics

- `WER (%)` **(primary)** — range: percent
  - Word Error Rate measuring linguistic preservation. Computed using Whisper-large-v3 for English and Paraformer-zh for Chinese.
- `SIM` — range: [-1, 1]
  - Speaker Similarity computed as the cosine similarity between speaker embeddings extracted by a WavLM-based model for the converted speech and the target reference.
- `UTMOS` — range: [1, 5]
  - Objective estimate of speech naturalness without human annotations, predicted by the UTMOS model.
- `RTF` — range: [0, inf)
  - Real-Time Factor calculated as the ratio of total end-to-end processing time to the input audio duration.
- `SMOS` — range: [1, 5]
  - Speaker Similarity Mean Opinion Score from a subjective test with 15 listeners rating on a 5-point scale (1: completely different, 5: identical speaker).

## Input / output format

**Input**: Source speech audio and reference speech audio (target speaker).

**Output**: Converted speech audio.

## Scoring recipe

```python
def evaluate(predictions, gold_refs, langs):
    wer_scores = []
    for pred, lang in zip(predictions, langs):
        asr = whisper_large_v3 if lang == 'en' else paraformer_zh
        wer_scores.append(asr.word_error_rate(pred))
    sim_scores = [cosine_similarity(wavlm_embed(pred), wavlm_embed(ref)) 
                  for pred, ref in zip(predictions, gold_refs)]
    utmos_scores = [utmos_predict(pred) for pred in predictions]
    rtf_scores = [processing_time(pred) / audio_duration(pred) for pred in predictions]
    return {
        'WER (%)': mean(wer_scores),
        'SIM': mean(sim_scores),
        'UTMOS': mean(utmos_scores),
        'RTF': mean(rtf_scores)
    }
```

## Common pitfalls

- MeanVC's offline RTF is measured using a chunkwise inference script rather than full-utterance processing, which inflates latency compared to true offline RTF.
- WER is computed using different ASR models for English (Whisper-large-v3) and Chinese (Paraformer-zh), making cross-lingual WER comparisons potentially biased.
- SMOS relies on a small panel of 15 listeners and random sample ordering, which may limit statistical reliability and introduce listener bias.

## Evidence (verbatim from paper)

> To assess speech quality, we use Word Error Rate (WER) to measure linguistic preservation, employing Whisper-large-v3 for English and Paraformer-zh for Chinese. Speaker identity consistency is evaluated via Speaker Similarity (SIM), computed as the cosine similarity between speaker embeddings extracted by a WavLM-based model, while UTMOS is utilized to objectively estimate speech naturalness without human annotations. For inference efficiency, all metrics are measured end-to-end on a single NVIDIA RTX 3090 GPU.

## Citation

```bibtex
@misc{zheng2026xvc,
  title={X-VC: Zero-shot Streaming Voice Conversion in Codec Space},
  author={Zheng et al. (2026)},
  year={2026},
  note={arXiv:2604.12456}
}
```

- arXiv: 2604.12456