Svcd Eval

---
name: svcd-eval
description: This protocol evaluates a singing voice conversion model's ability to transform a source singer's voice into a target singer's timbre while preserving musical naturalness and pitch accuracy. It measures both subjective perceptual quality and objective acoustic fidelity using human ratings and signal processing metrics. Use when the user wants to benchmark on VCTK, NUS-48E, or asks about evaluating this task. Reports MOS (Naturalness), MOS (Similarity).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2110.04754
  bibtex_key: wang2021singing
  confidence: high
---

# svcd-eval

> Towards High-fidelity Singing Voice Conversion with Acoustic Reference and Contrastive Predictive Coding — Wang et al. (2021) (arXiv:2110.04754, 2021)

## What this evaluates

This protocol evaluates a singing voice conversion model's ability to transform a source singer's voice into a target singer's timbre while preserving musical naturalness and pitch accuracy. It measures both subjective perceptual quality and objective acoustic fidelity using human ratings and signal processing metrics.

## Datasets

- **VCTK** — total ?; splits: train (-1), val (-1), test (-1)
- **NUS-48E** — total ?; splits: train (-1), val (-1), test (20)

## Metrics

- `MOS (Naturalness)` **(primary)** — range: [1, 5]
  - 1-5 Mean Opinion Score rating the naturalness of the converted singing audio, where 1 is bad and 5 is excellent.
- `MOS (Similarity)` **(primary)** — range: [1, 5]
  - 1-5 Mean Opinion Score rating the timbre-style similarity of the converted audio to the target singer.
- `NCC` — range: [0, 1]
  - Normalized cross-correlation between the pitch contours of the source and converted singing audios to measure pitch matching accuracy.
- `COS-SIM` — range: [0, 1]
  - Cosine similarity between d-vectors of the converted and target audio, extracted using a pretrained speaker verification model (Resemblyzer).

## Input / output format

**Input**: Source singing audio waveform and target singer reference/timbre specification.

**Output**: Converted singing audio waveform.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    mos_naturalness = human_rate(predictions, 'naturalness', scale=1-5)
    mos_similarity = human_rate(predictions, 'timbre_similarity', scale=1-5)
    pitch_src = extract_pitch(gold)
    pitch_pred = extract_pitch(predictions)
    ncc = normalized_cross_correlation(pitch_src, pitch_pred)
    dvec_tgt = extract_dvector(gold)
    dvec_pred = extract_dvector(predictions)
    cos_sim = cosine_similarity(dvec_tgt, dvec_pred)
    return {'mos_naturalness': mos_naturalness, 'mos_similarity': mos_similarity, 'ncc': ncc, 'cos_sim': cos_sim}
```

## Common pitfalls

- MOS scores are highly dependent on the specific panel of 15 music professionals; results are not directly comparable across different papers without identical raters.
- NCC only measures pitch contour alignment, not overall audio quality or timbre fidelity.
- The test set is small (20 segments from 10 singers), making statistical significance of MOS scores sensitive to segment selection.

## Evidence (verbatim from paper)

> For subjective evaluation, we select 1-5 mean opinion score (MOS) (1-bad, 2-poor, 3-fair, 4-good, 5-excellent) for both singing naturalness and timbre-style similarity with the target singer. For objective evaluation, we select normalized cross-correlation (NCC) to measure the pitch matching degree between the source and converted singing audios.

## Citation

```bibtex
@misc{wang2021singing,
  title={Towards High-fidelity Singing Voice Conversion with Acoustic Reference and Contrastive Predictive Coding},
  author={Wang et al. (2021)},
  year={2021},
  note={arXiv:2110.04754}
}
```

- arXiv: 2110.04754