Zero Shot Voice Synthesis Eval

---
name: zero-shot-voice-synthesis-eval
description: Evaluates zero-shot speech synthesis and novel voice generation by measuring how well models can produce intelligible, natural, and speaker-similar audio for unseen speakers using only conditioning embeddings. Use when the user wants to benchmark on English Multi-Accent Dataset (VCTK + Internal), or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2312.14569
  bibtex_key: bilinski2023creating
  confidence: high
---

# zero-shot-voice-synthesis-eval

> Creating New Voices using Normalizing Flows — Bilinski et al. (2023) (arXiv:2312.14569, 2023)

## What this evaluates

Evaluates zero-shot speech synthesis and novel voice generation by measuring how well models can produce intelligible, natural, and speaker-similar audio for unseen speakers using only conditioning embeddings.

## Datasets

- **English Multi-Accent Dataset (VCTK + Internal)** — total ?; splits: train (-1), test (-1)

## Metrics

- `WER` **(primary)** — range: percent
  - Average Word Error Rate between the ground-truth sentence text and the ASR transcription of the generated audio. Lower is better.
- `SECS` — range: [0, 1]
  - Speaker Encoder Cosine Similarity: the average cosine similarity between speaker embeddings extracted from the generated utterance and the target speaker's reference utterance. Higher is better.
- `MUSHRA Naturalness` — range: [0, 100]
  - Subjective rating on a 0-100 scale where listeners rate the perceptual naturalness of the generated audio compared to a reference.
- `MUSHRA Speaker Similarity` — range: [0, 100]
  - Subjective rating on a 0-100 scale where listeners rate how similar the generated speaker sounds to the target reference speaker.

## Input / output format

**Input**: Sentence text, source audio (for VC), speaker embeddings, f0, voicing, phonemes, and accent conditioning.

**Output**: Mel-spectrograms converted to 16 kHz audio samples via a Parallel WaveNet vocoder.

## Scoring recipe

```python
def compute_wer(gold_text, generated_audio):
    asr_transcript = aws_transcribe(generated_audio)
    return word_error_rate(gold_text, asr_transcript)

def compute_secs(generated_audio, target_audio):
    gen_emb = speaker_encoder(generated_audio)
    target_emb = speaker_encoder(target_audio)
    return cosine_similarity(gen_emb, target_emb)

def compute_mushra(generated_audio, reference_audio):
    # Human listeners rate on 0-100 scale
    return human_rating(generated_audio, reference_audio)
```

## Common pitfalls

- Oracle f0 and voicing (vuv) conditioning is used in Flow-VC but is not available in standard TTS inference, creating an unfair comparison if not noted.
- Speaker embeddings may cluster by recording corpus rather than purely by speaker identity, which can artificially inflate or deflate similarity scores.
- ASR model choice significantly impacts WER; US-English models are used for American/Canadian speakers while Australian-English models are used for Australian speakers.

## Evidence (verbatim from paper)

> For the objective evaluation, we extract speaker embeddings from generated utterances using[[22]] and we measure Speaker Encoder Cosine Similarity (SECS), which is defined as the average cosine similarity between extracted speaker embeddings from generated utterances and target embeddings.

## Citation

```bibtex
@misc{bilinski2023creating,
  title={Creating New Voices using Normalizing Flows},
  author={Bilinski et al. (2023)},
  year={2023},
  note={arXiv:2312.14569}
}
```

- arXiv: 2312.14569