Qwen3 Tts Eval

---
name: qwen3-tts-eval
description: This evaluation protocol assesses the speech synthesis and tokenization capabilities of the Qwen3-TTS model. It probes zero-shot voice cloning, multilingual and cross-lingual generation, controllable voice design, and long-form audio stability across multiple languages. Use when the user wants to benchmark on CommonVoice & Fleurs, LibriSpeech test-clean, Seed-TTS test set, TTS multilingual test set, CV3-Eval, InstructTTSEval, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.15621
  bibtex_key: hu2026qwen3tts
  confidence: high
---

# qwen3-tts-eval

> Qwen3-TTS Technical Report — Hangrui Hu et al. (2026) (arXiv:2601.15621, 2026)

## What this evaluates

This evaluation protocol assesses the speech synthesis and tokenization capabilities of the Qwen3-TTS model. It probes zero-shot voice cloning, multilingual and cross-lingual generation, controllable voice design, and long-form audio stability across multiple languages.

## Datasets

- **CommonVoice & Fleurs** — total ?; splits: test (-1)
- **LibriSpeech test-clean** — total 2620; splits: test (2620)
- **Seed-TTS test set** — total ?; splits: test (-1)
- **TTS multilingual test set** — total ?; splits: test (-1)
- **CV3-Eval** — total ?; splits: test (-1)
- **InstructTTSEval** — total ?; splits: test (-1)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate: (Substitutions + Deletions + Insertions) / Total Reference Words. Lower is better.
- `Speaker Similarity (SIM)` — range: [0, 1]
  - Cosine similarity between speaker embeddings extracted from reference and generated audio using a WavLM-based verification model. Higher is better.
- `PESQ` — range: [0, 4.5]
  - Perceptual Evaluation of Speech Quality: standard objective metric for speech quality and intelligibility. Higher is better.
- `STOI` — range: [0, 1]
  - Short-Time Objective Intelligibility: predicts speech intelligibility based on temporal envelope correlation. Higher is better.
- `UTMOS` — range: [1, 5]
  - Unsupervised MOS: neural network-based prediction of mean opinion score for naturalness. Higher is better.
- `APS` — range: [0, 100]
  - Attribute Perception and Synthesis accuracy: measures how well the model captures specified voice attributes. Higher is better.
- `DSD` — range: [0, 100]
  - Description–Speech Consistency: evaluates alignment between text descriptions and generated acoustic output. Higher is better.
- `RP` — range: [0, 100]
  - Response Precision: measures adherence to voice control instructions. Higher is better.

## Input / output format

**Input**: Text instructions (including reference audio/text for voice cloning) and target transcript. For controllable tasks, inputs follow the ChatML format with explicit voice design or editing instructions.

**Output**: Synthesized speech audio waveform.

## Scoring recipe

```python
def compute_wer(generated_audio, reference_text, asr_model):
    transcribed = asr_model.transcribe(generated_audio)
    return wer_score(transcribed, reference_text)  # (S+D+I)/N

def compute_sim(ref_audio, gen_audio, wavlm_model):
    ref_emb = wavlm_model.encode(ref_audio)
    gen_emb = wavlm_model.encode(gen_audio)
    return cosine_similarity(ref_emb, gen_emb)

# For controllable tasks, APS/DSD/RP are computed via automated or human evaluation pipelines aligned with InstructTTSEval guidelines.
```

## Common pitfalls

- ASR transcription models vary in accuracy, directly impacting WER scores; consistent ASR backends are required for fair comparison.
- Speaker similarity (SIM) relies on WavLM embeddings, which may exhibit language bias or require identical sampling rates for reference and generated audio.
- Long-form evaluation uses an internal 100-sample dataset, making independent replication difficult.

## Evidence (verbatim from paper)

> We evaluate the model’s ability to clone unseen voices by measuring content consistency—specifically Word Error Rate (WER)—on the public Seed-TTS test set (Anastassiou et al., [2024]).

## Citation

```bibtex
@misc{hu2026qwen3tts,
  title={Qwen3-TTS Technical Report},
  author={Hangrui Hu et al. (2026)},
  year={2026},
  note={arXiv:2601.15621}
}
```

- arXiv: 2601.15621