Rvcbench Eval

---
name: rvcbench-eval
description: Evaluates the robustness of modern voice cloning models under realistic deployment conditions, including input variations (accents, text shifts, long context), cross-lingual synthesis, post-processing degradation, and adversarial perturbations. It probes the trade-offs between generation quality, content fidelity, speaker similarity, and deepfake detectability across diverse acoustic and linguistic stressors. Use when the user wants to benchmark on LibriTTS, VCTK, LibriSpeech, RVCBench, or asks about evaluating this task. Reports SIM, WER, MCD.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.00443
  bibtex_key: liao2026rvcbench
  confidence: high
---

# rvcbench-eval

> RVCBench: Benchmarking the Robustness of Voice Cloning Across Modern Audio Generation Models — Liao et al. (2026) (arXiv:2602.00443, 2026)

## What this evaluates

Evaluates the robustness of modern voice cloning models under realistic deployment conditions, including input variations (accents, text shifts, long context), cross-lingual synthesis, post-processing degradation, and adversarial perturbations. It probes the trade-offs between generation quality, content fidelity, speaker similarity, and deepfake detectability across diverse acoustic and linguistic stressors.

## Datasets

- **LibriTTS** — total ?; splits: test (-1)
- **VCTK** — total ?; splits: test (-1)
- **LibriSpeech** — total ?; splits: test (-1)
- **RVCBench** — total 14370; splits: multilingual (-1), audio_shift (-1), text_shift (-1), expression (-1), compression (-1), detectability (-1), anti_protect (-1); repo https://github.com/Nanboy-Ronan/RVCBench

## Metrics

- `SIM` **(primary)** — range: [0, 1]
  - Speaker similarity score computed via cosine similarity of pre-trained speaker embeddings between reference and synthesized audio.
- `WER` **(primary)** — range: percent
  - Word Error Rate measuring transcription accuracy of the synthesized audio against the target text using an ASR system.
- `MCD` **(primary)** — range: other
  - Mel-cepstral Distortion quantifying the spectral difference between reference and synthesized speech frames.
- `MOS` — range: [1, 5]
  - Mean Opinion Score assessing perceptual audio quality and naturalness via human or LLM-as-Judge ratings.
- `EER` — range: percent
  - Equal Error Rate from a deepfake detector (SQ-LLM) where false acceptance and false rejection rates intersect.
- `minDCF` — range: percent
  - Minimum Detection Cost Function evaluating the trade-off between false acceptance and false rejection costs for deepfake detection.
- `ACC` — range: percent
  - Classification accuracy of the deepfake detector in distinguishing ground-truth, benign cloned, and scam cloned speech.

## Input / output format

**Input**: Reference audio clip (speaker voice), target text prompt, and optional perturbation parameters (e.g., noise type, compression codec, adversarial method, language).

**Output**: Synthesized audio waveform matching the target text in the reference speaker's voice.

## Scoring recipe

```python
def compute_metrics(ref_audio, gen_audio, target_text, det_model):
    sim = cosine_similarity(speaker_embed(ref_audio), speaker_embed(gen_audio))
    wer = asr_transcribe(gen_audio).word_error_rate(target_text)
    mcd = mel_cepstral_distortion(ref_audio, gen_audio)
    mos = llm_judge(gen_audio).score()
    det_logits = det_model(gen_audio)
    eer, mindcf = compute_det_curve(det_logits, is_fake_label)
    acc = accuracy(det_logits, is_fake_label)
    return {'SIM': sim, 'WER': wer, 'MCD': mcd, 'MOS': mos, 'EER': eer, 'minDCF': mindcf, 'ACC': acc}
```

## Common pitfalls

- Assuming high generation quality (MOS/SIM) implies strong evasion from deepfake detectors; the paper shows competitive models like FishSpeech remain highly detectable.
- Believing longer reference audio always improves robustness; performance gains plateau after 8–12 seconds and can introduce non-monotonic instability.
- Overlooking that cross-lingual tasks primarily bottleneck on content accuracy (WER) rather than speaker similarity, despite some models maintaining high SIM.

## Evidence (verbatim from paper)

> We assess deepfake detectability against impersonation using EER, minDCF, and ACC, and find detectability varies across VCmodels in Tab.[3]. Higher ACC and lower EER/minDCF denote more deepfake perceptible.

## Citation

```bibtex
@misc{liao2026rvcbench,
  title={RVCBench: Benchmarking the Robustness of Voice Cloning Across Modern Audio Generation Models},
  author={Liao et al. (2026)},
  year={2026},
  note={arXiv:2602.00443}
}
```

- arXiv: 2602.00443