Svcc23 Eval

---
name: svcc23-eval
description: Evaluates singing voice conversion systems on in-domain (singing-to-singing) and cross-domain (speech-to-singing) speaker conversion. It probes the model's ability to preserve target speaker identity and musical prosody while converting source audio to the target voice. Use when the user wants to benchmark on SVCC 2023, or asks about evaluating this task. Reports perceptual quality (subjective evaluation).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.14422
  bibtex_key: huang2023svcc23
  confidence: medium
---

# svcc23-eval

> The Singing Voice Conversion Challenge 2023 — Huang et al. (2023) (arXiv:2306.14422, 2023)

## What this evaluates

Evaluates singing voice conversion systems on in-domain (singing-to-singing) and cross-domain (speech-to-singing) speaker conversion. It probes the model's ability to preserve target speaker identity and musical prosody while converting source audio to the target voice.

## Datasets

- **SVCC 2023** — total ?; splits: train (-1), test (-1); repo https://github.com/lesterphillip/SVCC23_FastSVC

## Metrics

- `perceptual quality (subjective evaluation)` **(primary)** — range: other
  - Human listeners rate converted audio on naturalness and target speaker similarity using subjective listening tests.
- `objective metrics` — range: [0, 1]
  - Automatic computational measures used to assess audio quality and speaker similarity, though noted to correlate weakly with human perception.

## Input / output format

**Input**: Source audio (singing for Task 1, speech for Task 2) and target speaker reference audio (singing for Task 1, speech for Task 2).

**Output**: Converted singing audio matching the target speaker's voice characteristics and the source audio's musical content.

## Scoring recipe

```python
# Pseudo-code for SVCC 2023 evaluation
predictions = model.convert(source_audio, target_ref_audio)
# Subjective evaluation (primary)
naturalness_rating = human_listen_test(predictions, criteria="naturalness")
similarity_rating = human_listen_test(predictions, criteria="speaker_similarity")
# Objective evaluation
obj_score = compute_objective_metrics(predictions, gold_audio)
# Aggregate results across in-domain and cross-domain tasks
final_score = average(naturalness_rating, similarity_rating, obj_score)
```

## Common pitfalls

- Cross-domain SVC is significantly harder than in-domain because models lack target singing style data.
- Objective metrics do not strongly correlate with human perceptual quality, so relying solely on them is misleading.

## Evidence (verbatim from paper)

> Despite top systems achieving human-level naturalness, no model matched target speaker similarity in either task, with cross-domain SVC being significantly harder. Objective metrics failed to correlate strongly with perceptual quality, highlighting the limitations of current evaluation frameworks in capturing subjective fidelity in singing voice conversion.

## Citation

```bibtex
@misc{huang2023svcc23,
  title={The Singing Voice Conversion Challenge 2023},
  author={Huang et al. (2023)},
  year={2023},
  note={arXiv:2306.14422}
}
```

- arXiv: 2306.14422