Vevo Voice Imitation Eval

---
name: vevo-voice-imitation-eval
description: This benchmark evaluates a model's ability to perform zero-shot voice imitation by disentangling linguistic content, speaker timbre, and vocal style (accent/emotion). It probes the model's capacity to generate high-intelligibility speech that accurately transfers the target speaker's identity and stylistic attributes from a reference clip without task-specific fine-tuning. Use when the user wants to benchmark on Vevo Evaluation Set (AB, CV, ACCENT, EMOTION), or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.07243
  bibtex_key: zhang2025vevo
  confidence: high
---

# vevo-voice-imitation-eval

> Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement — Zhang et al. (2025) (arXiv:2502.07243, 2025)

## What this evaluates

This benchmark evaluates a model's ability to perform zero-shot voice imitation by disentangling linguistic content, speaker timbre, and vocal style (accent/emotion). It probes the model's capacity to generate high-intelligibility speech that accurately transfers the target speaker's identity and stylistic attributes from a reference clip without task-specific fine-tuning.

## Datasets

- **Vevo Evaluation Set (AB, CV, ACCENT, EMOTION)** — total 700; splits: test (700)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate calculated using Whisper-large-v3 to transcribe generated audio and compare against ground-truth text. Lower is better.
- `S-SIM` — range: [0, 1]
  - Cosine similarity between speaker embeddings extracted from the generated audio and the reference audio using WavLM TDNN. Higher is better.
- `A-SIM` — range: [0, 1]
  - Cosine similarity between accent embeddings extracted using CommonAccent. Higher is better.
- `E-SIM` — range: [0, 1]
  - Cosine similarity between emotion embeddings extracted using emotion2vec. Higher is better.
- `FPC` — range: [0, 1]
  - F0 correlation between generated and source audio, evaluated only on the EMOTION split.
- `N-MOS` — range: other
  - Mean Opinion Score (1-5) for naturalness rated by human listeners.
- `CMOS` — range: other
  - Comparative MOS (-3 to 3) for naturalness, accentedness, and emotiveness relative to baselines.

## Input / output format

**Input**: Source linguistic content (text or speech) and a reference audio clip containing the target timbre and/or style (accent/emotion).

**Output**: Generated speech waveform matching the source content and the reference's timbre/style.

## Scoring recipe

```python
# WER
gen_transcript = whisper_large_v3(generated_audio)
wer = calculate_wer(gold_text, gen_transcript)

# S-SIM / A-SIM / E-SIM
ref_emb = extractor.encode(reference_audio)
gen_emb = extractor.encode(generated_audio)
sim = cosine_similarity(ref_emb, gen_emb)

# FPC (EMOTION split only)
fpc = pearson_corr(f0(generated_audio), f0(source_audio))
```

## Common pitfalls

- FPC and PS-MOS are only evaluated on the EMOTION split, not the full test set.
- Subjective MOS/CMOS scores depend on listener panels and baseline models hosted on external demo websites, making exact reproduction difficult.
- Vevo-Voice trades intelligibility (WER) for style imitation due to its autoregressive style tokenizer, so lower WER does not indicate overall failure.

## Evidence (verbatim from paper)

> For the objective metrics, we evaluate the intelligibility (WER), speaker similarity (S-SIM), accent similarity (A-SIM), emotion similarity (E-SIM), and F0 correlation (FPC). Specially, we calculate WER based on Whisper-large-v3. For the three similarity metrics – S-SIM, A-SIM, and E-SIM – we calculate the cosine similarity between the embeddings (of speaker, accent, or emotion) of the generated sample and the reference.

## Citation

```bibtex
@misc{zhang2025vevo,
  title={Vevo: Controllable Zero-Shot Voice Imitation with Self-Supervised Disentanglement},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2502.07243}
}
```

- arXiv: 2502.07243