Vcc2018 Eval

---
name: vcc2018-eval
description: Evaluates voice conversion systems on speech naturalness and target speaker similarity using crowdsourced perceptual tests, and assesses linguistic consistency via automatic speech recognition word error rates. It covers both parallel (Hub) and non-parallel (Spoke) conversion tasks. Use when the user wants to benchmark on VCC2018, or asks about evaluating this task. Reports Naturalness (MOS).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1804.04262
  bibtex_key: lorenzotrubavcc2018
  confidence: high
---

# vcc2018-eval

> The Voice Conversion Challenge 2018: Promoting Development of Parallel and Nonparallel Methods — Lorenzo-Trueba et al. (2018) (arXiv:1804.04262, 2018)

## What this evaluates

Evaluates voice conversion systems on speech naturalness and target speaker similarity using crowdsourced perceptual tests, and assesses linguistic consistency via automatic speech recognition word error rates. It covers both parallel (Hub) and non-parallel (Spoke) conversion tasks.

## Datasets

- **VCC2018** — total ?; splits: Hub (-1), Spoke (-1)

## Metrics

- `Naturalness (MOS)` **(primary)** — range: [1, 5]
  - Mean Opinion Score rated by crowdsourced listeners on a standard scale (typically 1-5). Averaged across all speaker pairs and listeners.
- `Similarity` — range: [0, 100] percent
  - Percentage of 'same (sure)' and 'same (not sure)' responses from listeners regarding target speaker identity.
- `Word Error Rate (WER)` — range: [0, 100] percent
  - Percentage of incorrectly transcribed words when the converted speech is processed by a prototype neural-network-based ASR engine, compared against manual ground-truth transcriptions.

## Input / output format

**Input**: Source speech audio files.

**Output**: Converted speech audio waveform.

## Scoring recipe

```python
mos_scores = []
sim_scores = []
for listener in listeners:
    for utterance in utterances:
        rating = listener.rate(utterance) # 1-5 scale
        mos_scores.append(rating)
        if rating in ['same (sure)', 'same (not sure)']:
            sim_scores.append(1.0)
        else:
            sim_scores.append(0.0)
naturalness = mean(mos_scores)
similarity = mean(sim_scores) * 100

wer_scores = []
for system_output in converted_speech:
    asr_transcript = asr_engine.transcribe(system_output)
    wer = calculate_wer(asr_transcript, ground_truth_transcript)
    wer_scores.append(wer)
wer = mean(wer_scores)
```

## Common pitfalls

- MOS and similarity scores are crowdsourced and may vary based on listener demographics and listening environment.
- WER is computed using a specific proprietary ASR prototype (iFlytek), so results may not generalize to other ASR systems.
- The Spoke task uses different linguistic content between source and target, making direct performance comparison with the Hub task non-trivial.

## Evidence (verbatim from paper)

> We carried out a crowdsourced perceptual evaluation... The similarity score is defined as the added percentage of same (not sure) and same (sure) scores for the system. To roughly evaluate the linguistic consistency after voice conversion, the word error rates (WERs) of transcribing the converted speech using an automatic speech recognition (ASR) engine were calculated for all entries and all source-target pairs.

## Citation

```bibtex
@misc{lorenzotrubavcc2018,
  title={The Voice Conversion Challenge 2018: Promoting Development of Parallel and Nonparallel Methods},
  author={Lorenzo-Trueba et al. (2018)},
  year={2018},
  note={arXiv:1804.04262}
}
```

- arXiv: 1804.04262