Vcc2018 Spoke Eval

---
name: vcc2018-spoke-eval
description: Evaluates non-parallel voice conversion by measuring how effectively a system transfers a source speaker's identity to a target speaker while preserving linguistic content. It probes intra-lingual conversion fidelity and cross-lingual adaptation using subjective human ratings. Use when the user wants to benchmark on VCC2018 (Voice Conversion Challenge 2018), VCTK, or asks about evaluating this task. Reports Quality.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1909.06532
  bibtex_key: luong2019bootstrapping
  confidence: high
---

# vcc2018-spoke-eval

> Bootstrapping non-parallel voice conversion from speaker-adaptive text-to-speech — Luong et al. (2019) (arXiv:1909.06532, 2019)

## What this evaluates

Evaluates non-parallel voice conversion by measuring how effectively a system transfers a source speaker's identity to a target speaker while preserving linguistic content. It probes intra-lingual conversion fidelity and cross-lingual adaptation using subjective human ratings.

## Datasets

- **VCC2018 (Voice Conversion Challenge 2018)** — total ?; splits: test (140)
- **VCTK** — total ?; splits: train (-1)

## Metrics

- `Quality` **(primary)** — range: [1, 5]
  - Mean subjective rating score assigned by human listeners to evaluate the overall audio quality of the converted speech.
- `Similarity` — range: [1, 5]
  - Mean subjective rating score assigned by human listeners to evaluate how closely the converted speech matches the target speaker's voice.

## Input / output format

**Input**: Source speaker speech utterances (English or Japanese) provided as audio input.

**Output**: Converted speech waveform matching the target speaker's voice characteristics.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    quality_ratings = []
    similarity_ratings = []
    for pred_audio, target_id in zip(predictions, gold):
        # Human listeners rate on a standard scale (typically 1-5)
        quality_ratings.append(listener.rate_quality(pred_audio))
        similarity_ratings.append(listener.rate_similarity(pred_audio, target_id))
    return {
        'Quality': sum(quality_ratings) / len(quality_ratings),
        'Similarity': sum(similarity_ratings) / len(similarity_ratings)
    }
```

## Common pitfalls

- Relies exclusively on subjective human listening tests rather than objective acoustic metrics like PESQ or STOI.
- Cross-lingual evaluation uses custom in-house bilingual speakers instead of standard benchmark datasets.
- Explicitly ignores the provided source speaker training data from VCC2018, which may limit direct comparison with baselines that use it.

## Evidence (verbatim from paper)

> To validate the proposed method, we reenact the SPOKE task of the Voice Conversion Challenge 2018 (VCC2018). We build the VC system for 4 target speakers (2 males and 2 females) using 81 utterances per person. We then evaluate the SPOKE task using the speech of 4 source speakers (2 males and 2 females); each speaker contribute 35 utterances. Table 1: Detailed subjective results for SPOKE task. (a) Quality (b) Similarity

## Citation

```bibtex
@misc{luong2019bootstrapping,
  title={Bootstrapping non-parallel voice conversion from speaker-adaptive text-to-speech},
  author={Luong et al. (2019)},
  year={2019},
  note={arXiv:1909.06532}
}
```

- arXiv: 1909.06532