Vcc2020 Intra Lingual Vc Eval

---
name: vcc2020-intra-lingual-vc-eval
description: Evaluates a model's ability to convert speech from a source speaker to a target speaker within the same language, leveraging limited parallel data alongside a larger non-parallel corpus. The benchmark probes how well systems can disentangle speaker identity from linguistic content when only a small set of aligned sentences is available for training. Use when the user wants to benchmark on EMIME, or asks about evaluating this task. Reports MOS (Mean Opinion Score).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2008.12527
  bibtex_key: zhao2020vcc2020
  confidence: high
---

# vcc2020-intra-lingual-vc-eval

> Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversion — Zhao et al. (2020) (arXiv:2008.12527, 2020)

## What this evaluates

Evaluates a model's ability to convert speech from a source speaker to a target speaker within the same language, leveraging limited parallel data alongside a larger non-parallel corpus. The benchmark probes how well systems can disentangle speaker identity from linguistic content when only a small set of aligned sentences is available for training.

## Datasets

- **EMIME** — total ?; splits: train (70), test (25)

## Metrics

- `MOS (Mean Opinion Score)` **(primary)** — range: [1, 5]
  - Average of subjective ratings provided by human listeners on a scale (typically 1-5) evaluating naturalness and speaker similarity of the converted audio.

## Input / output format

**Input**: Source speech audio file in English containing the linguistic content to be preserved.

**Output**: Converted speech audio file matching the target speaker's voice characteristics while preserving the source linguistic content.

## Scoring recipe

```python
def compute_mos(predictions, target_speakers, listeners):
    mos_scores = []
    for pred, target in zip(predictions, target_speakers):
        ratings = []
        for listener in listeners:
            nat = listener.score_naturalness(pred)
            sim = listener.score_similarity(pred, target)
            ratings.append((nat + sim) / 2)
        mos_scores.append(sum(ratings) / len(ratings))
    return sum(mos_scores) / len(mos_scores)
```

## Common pitfalls

- Relies entirely on subjective human listening tests rather than objective acoustic metrics.
- Target speakers in cross-lingual task have no source-language recordings, making ground-truth comparison impossible.
- Participants must build both VC and vocoder models, introducing variability in system quality that can skew MOS results.

## Evidence (verbatim from paper)

> Results show significant progress in naturalness and speaker similarity, with intra-lingual systems achieving human-level speaker similarity scores (≥ target speakers) in listening tests, though no system reaches human-level naturalness. Cross-lingual conversion remains difficult, with lower naturalness and similarity, but best systems achieve MOS > 4.0, indicating promising advances in handling language divergence and nonparallel training.

## Citation

```bibtex
@misc{zhao2020vcc2020,
  title={Voice Conversion Challenge 2020: Intra-lingual semi-parallel and cross-lingual voice conversion},
  author={Zhao et al. (2020)},
  year={2020},
  note={arXiv:2008.12527}
}
```

- arXiv: 2008.12527