Expressive Voice Cloning Eval

---
name: expressive-voice-cloning-eval
description: Evaluates zero-shot and adapted voice cloning models on their ability to preserve speaker identity, transfer expressive style (pitch and rhythm), and generate natural-sounding speech. It measures speaker similarity, style fidelity, and perceptual quality across text-to-speech, imitation, and style transfer tasks. Use when the user wants to benchmark on VCTK, Libri-TTS, or asks about evaluating this task. Reports Speaker Classification Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2102.00151
  bibtex_key: [REDACTED]
  confidence: high
---

# expressive-voice-cloning-eval

> Expressive Neural Voice Cloning — Neekhara et al. (2021) (arXiv:2102.00151, 2021)

## What this evaluates

Evaluates zero-shot and adapted voice cloning models on their ability to preserve speaker identity, transfer expressive style (pitch and rhythm), and generate natural-sounding speech. It measures speaker similarity, style fidelity, and perceptual quality across text-to-speech, imitation, and style transfer tasks.

## Datasets

- **VCTK** — total 108; splits: test (108)
- **Libri-TTS** — total 245; splits: train (245)

## Metrics

- `Speaker Classification Accuracy` **(primary)** — range: [0, 1]
  - Percentage of correctly identified speakers from a 108-class neural classifier trained on VCTK speaker encodings.
- `Speaker Verification Equal Error Rate (SV-EER)` — range: [0, 1]
  - Point where false acceptance and false rejection rates are equal, computed via cosine similarity of speaker encodings between cloned and ground-truth audio.
- `Gross Pitch Error (GPE)` — range: [0, 1]
  - Percentage of frames where pitch is either completely wrong or off by more than 20% compared to the reference.
- `Style-MOS` — range: [1, 5]
  - Mean Opinion Score from crowd-sourced listeners rating style similarity between synthesized and reference audio on a 5-point scale.
- `Naturalness MOS` — range: [1, 5]
  - Mean Opinion Score from listeners rating perceptual naturalness of synthesized audio on a 5-point scale.

## Input / output format

**Input**: Text prompt, target speaker audio samples (for speaker encoding), and style reference audio (for pitch/rhythm/GST embedding).

**Output**: Synthesized mel-spectrogram or audio waveform matching the target speaker's voice and the requested style.

## Scoring recipe

```python
def evaluate_cloning(generated_audios, target_audios, speaker_ids, style_refs):
    speaker_acc = compute_classification_accuracy(generated_audios, speaker_ids)
    sv_eer = compute_eer(cosine_similarity(speaker_encoder(generated_audios), speaker_encoder(target_audios)))
    gpe = compute_gpe(pitch_contour(generated_audios), pitch_contour(style_refs))
    vde = compute_vde(voice_activity(generated_audios), voice_activity(style_refs))
    style_mos = average_ratings(listeners_rate(generated_audios, style_refs, scale=5))
    nat_mos = average_ratings(listeners_rate(generated_audios, scale=5))
    return {'speaker_acc': speaker_acc, 'sv_eer': sv_eer, 'gpe': gpe, 'style_mos': style_mos, 'nat_mos': nat_mos}
```

## Common pitfalls

- Requires a separate single-speaker TTS model (Tacotron 2 + WaveGlow) to generate initial pitch contours for text-only cloning.
- Style reference audio must align with the target text for rhythm extraction; otherwise, a synthetic reference must be generated first.
- Speaker classifier trained on VCTK uses 1151 training speakers, making direct comparison with prior work (2481 speakers) invalid.

## Evidence (verbatim from paper)

> For the above described cloning tasks, we evaluate three aspects of the cloned speech: i) speaker similarity to the target speaker, ii) style similarity to the reference style and iii) speech naturalness. We train a speaker classifier on the VCTK dataset to classify a given utterance as one of the 108 speakers. SV-EER is another objective metric used to evaluate speaker similarity between the cloned audio and the ground-truth reference audio. We use the following error metrics: Gross Pitch Error (GPE), Voicing Decision Error (VDE) and F0 Frame Error (FFE).

## Citation

```bibtex
@misc{neekhara2021expressive,
  title={Expressive Neural Voice Cloning},
  author={Neekhara et al. (2021)},
  year={2021},
  note={arXiv:2102.00151}
}
```

- arXiv: 2102.00151