Breezyvoice Tts Eval

---
name: breezyvoice-tts-eval
description: Evaluates the phonetic accuracy, audio quality, and speaker similarity of a Taiwanese Mandarin TTS system, with a focus on voice cloning robustness and code-switching scenarios. The benchmark probes the model's ability to handle long-tail speaker variability and context-dependent pronunciation ambiguities in both monolingual and bilingual contexts. Use when the user wants to benchmark on FormosaSpeech (subset), Spontaneous Recordings, Traditional Chinese Monologue Dataset (TCMD), Traditional Chinese Code-switching Dataset (TCCSD), or asks about evaluating this task. Reports PER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.17790
  bibtex_key: hsu2025breezyvoice
  confidence: medium
---

# breezyvoice-tts-eval

> BreezyVoice: Adapting TTS for Taiwanese Mandarin with Enhanced Polyphone Disambiguation -- Challenges and Insights — Hsu et al. (2025) (arXiv:2501.17790, 2025)

## What this evaluates

Evaluates the phonetic accuracy, audio quality, and speaker similarity of a Taiwanese Mandarin TTS system, with a focus on voice cloning robustness and code-switching scenarios. The benchmark probes the model's ability to handle long-tail speaker variability and context-dependent pronunciation ambiguities in both monolingual and bilingual contexts.

## Datasets

- **FormosaSpeech (subset)** — total 15; splits: test (15)
- **Spontaneous Recordings** — total 100; splits: test (100)
- **Traditional Chinese Monologue Dataset (TCMD)** — total ?; splits: test (-1)
- **Traditional Chinese Code-switching Dataset (TCCSD)** — total ?; splits: test (-1)

## Metrics

- `PER` **(primary)** — range: percent
  - Phone Error Rate; measures the normalized edit distance between predicted and reference phoneme sequences. Formula not explicitly provided in the text.
- `Speaker Similarity` — range: [0, 1]
  - Quantifies the acoustic similarity between generated and reference speech, typically via embedding cosine similarity. Specific implementation details are not provided in the text.

## Input / output format

**Input**: Traditional Chinese text (monologue or code-switching), conditioning speech utterances (5–15 seconds from diverse speakers), and optionally augmented Mandarin phonetic symbols.

**Output**: Generated audio waveform (derived from mel-spectrograms via a static algorithmic vocoder).

## Scoring recipe

```python
def evaluate(predictions, gold):
    per_scores = []
    sim_scores = []
    for pred_audio, ref_audio, ref_text in zip(predictions, gold['audio'], gold['text']):
        # Compute PER between generated and reference phonemes
        pred_phones = extract_phonemes(pred_audio)
        ref_phones = extract_phonemes(ref_audio)
        per_scores.append(edit_distance(pred_phones, ref_phones) / len(ref_phones))
        # Compute Speaker Similarity between generated and reference audio
        sim_scores.append(cosine_similarity(embed(pred_audio), embed(ref_audio)))
    return {'PER': mean(per_scores), 'Speaker Similarity': mean(sim_scores)}
```

## Common pitfalls

- Comparing against proprietary TTS systems that rely on preset voices without accounting for their lack of cloning capabilities, leading to unfair audio quality comparisons.
- Ignoring code-switching scenarios, which are critical for evaluating Taiwanese Mandarin TTS robustness in real-world contexts involving entity names and technical jargon.
- Failing to use a diverse range of speakers for voice cloning evaluations, which can overestimate speaker similarity and mask long-tail variability issues.

## Evidence (verbatim from paper)

> For voice cloning evaluations, we evaluate on PER and speaker similarity using samples generated from a diverse range of speakers to gauge the robustness of our system. ... Phonetic accuracy and prosodic appropriateness are the primary evaluation criteria.

## Citation

```bibtex
@misc{hsu2025breezyvoice,
  title={BreezyVoice: Adapting TTS for Taiwanese Mandarin with Enhanced Polyphone Disambiguation -- Challenges and Insights},
  author={Hsu et al. (2025)},
  year={2025},
  note={arXiv:2501.17790}
}
```

- arXiv: 2501.17790