Zero Shot Tts Vietnamese Eval

---
name: zero-shot-tts-vietnamese-eval
description: Evaluates zero-shot text-to-speech models on Vietnamese speech generation, measuring intelligibility, speaker similarity, and naturalness across long-form and short-form text inputs. Use when the user wants to benchmark on viVoice, PAB-S, PAB-U, VIVOS, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.01322
  bibtex_key: vu2025vietnamesetts
  confidence: high
---

# zero-shot-tts-vietnamese-eval

> Zero-Shot Text-to-Speech for Vietnamese — Vu et al. (2025) (arXiv:2506.01322, 2025)

## What this evaluates

Evaluates zero-shot text-to-speech models on Vietnamese speech generation, measuring intelligibility, speaker similarity, and naturalness across long-form and short-form text inputs.

## Datasets

- **viVoice** — total ?; splits: test (-1)
- **PAB-S** — total ?; splits: test (-1)
- **PAB-U** — total ?; splits: test (-1)
- **VIVOS** — total ?; splits: test (-1)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate: percentage of words incorrectly recognized or generated compared to the reference transcript.
- `SMOS` — range: [0, 1]
  - Speaker Mean Opinion Score: subjective rating of how closely the generated speech matches the target speaker's voice.
- `RMSEF0` — range: other
  - Root Mean Square Error of F0: measures the deviation of the fundamental frequency contour between generated and reference speech.
- `MCD` — range: other
  - Mel Cepstral Distortion: measures the spectral distortion between generated and reference audio frames.
- `MOS` — range: [0, 1]
  - Mean Opinion Score: subjective rating of overall speech naturalness and quality.

## Input / output format

**Input**: Input text prompt and a reference audio clip of a target speaker.

**Output**: Synthesized audio waveform corresponding to the input text.

## Scoring recipe

```python
def evaluate(predictions, gold):
    wer = compute_wer(predictions.text, gold.text)
    smos = compute_smos(predictions.audio, gold.audio)
    rmsef0 = compute_rmsef0(predictions.f0, gold.f0)
    mcd = compute_mcd(predictions.spectra, gold.spectra)
    mos = compute_mos(predictions.audio)
    return {'WER': wer, 'SMOS': smos, 'RMSEF0': rmsef0, 'MCD': mcd, 'MOS': mos}
```

## Common pitfalls

- XTTS-v2-based models tend to generate redundant or rambling speech on short text inputs (e.g., VIVOS set).
- Performance varies significantly by input length; models optimized for long-form data may underperform on short sentences.

## Evidence (verbatim from paper)

> For instance, on the viVoice set, XTTS-v2PAB achieves the best WER of 8.32, which is substantially lower than the 12.54 WER of viXTTS, even though viXTTS is tested on its own training data. Additionally, XTTS-v2PAB also produces substantially higher SMOS and RMSEF0 scores compared to viXTTS in all test sets, indicating that the speech it generates more closely resembles the reference speaker.

## Citation

```bibtex
@misc{vu2025vietnamesetts,
  title={Zero-Shot Text-to-Speech for Vietnamese},
  author={Vu et al. (2025)},
  year={2025},
  note={arXiv:2506.01322}
}
```

- arXiv: 2506.01322