Samoye Svc Eval

---
name: samoye-svc-eval
description: Evaluates zero-shot singing voice conversion by measuring timbre transfer accuracy and audio quality. It probes the model's ability to disentangle content, pitch, and timbre, and generalize to unseen human and non-human (animal) speakers without fine-tuning. Use when the user wants to benchmark on Custom zero-shot test set, or asks about evaluating this task. Reports MOS-S.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.07728
  bibtex_key: wang2024samoye
  confidence: high
---

# samoye-svc-eval

> SaMoye: Zero-shot Singing Voice Conversion Model Based on Feature Disentanglement and Enhancement — Zihao Wang et al. (2024) (arXiv:2407.07728, 2024)

## What this evaluates

Evaluates zero-shot singing voice conversion by measuring timbre transfer accuracy and audio quality. It probes the model's ability to disentangle content, pitch, and timbre, and generalize to unseen human and non-human (animal) speakers without fine-tuning.

## Datasets

- **Custom zero-shot test set** — total 5; splits: test (5)

## Metrics

- `MOS-S` **(primary)** — range: [1, 5]
  - Mean Opinion Score on Similarity based on a 5-point Likert scale (1=completely different timbre, 5=same timbre).
- `MOS-Q` — range: [1, 5]
  - Mean Opinion Score on Quality based on expert evaluation of audio fidelity.
- `PESQ` — range: [-0.5, 4.5]
  - Perceptual Evaluation of Speech Quality measuring temporal alignment and perceptual filtering.
- `STOI` — range: [0, 1]
  - Short-Time Objective Intelligibility measuring comprehensibility of audio.
- `NISQA` — range: other
  - Non-Intrusive Speech Quality Assessment using a pre-trained deep learning model to predict MOS, noise, coherence, timbre, and loudness.
- `SECS` — range: [0, 1]
  - Speaker Encoder Cosine Similarity computed between CAM++ embeddings of generated and original audio.

## Input / output format

**Input**: Source singing audio waveform and reference audio waveform (target timbre).

**Output**: Converted singing audio waveform.

## Scoring recipe

```python
import numpy as np
from scipy.spatial.distance import cosine

def compute_secs(pred_wav, gold_wav):
    pred_emb = cam_plus_plus_embed(pred_wav)
    gold_emb = cam_plus_plus_embed(gold_wav)
    return 1.0 - cosine(pred_emb, gold_emb)

def compute_mos_s(ratings):
    return np.mean(ratings)
```

## Common pitfalls

- NISQA is pretrained on speech datasets and may misjudge singing audio quality due to the speech-singing gap.
- SECS relies on CAM++, which is trained on human speech, making it unreliable for non-human/animal timbres.
- Subjective MOS scores can be confounded by overall audio quality; high MOS-Q often correlates with high MOS-S regardless of actual timbre fidelity.

## Evidence (verbatim from paper)

> The subjective metrics include:
- Mean Opinion Score on Similarity(MOS-S): MOS-S is based on a 5-score Likert scale, where 5 means the same timbre and 1 for a completely different timbre.  
- Mean Opinion Score on Quality(MOS-Q): MOS is a widely-used audio or video quality evaluation standard based on expert evaluation. The score of MOS is from 1 to 5, where a higher score means higher quality.

## Citation

```bibtex
@misc{wang2024samoye,
  title={SaMoye: Zero-shot Singing Voice Conversion Model Based on Feature Disentanglement and Enhancement},
  author={Zihao Wang et al. (2024)},
  year={2024},
  note={arXiv:2407.07728}
}
```

- arXiv: 2407.07728