Singing Voice Separation Eval

---
name: singing-voice-separation-eval
description: This benchmark evaluates a model's ability to extract singing vocals from mixed audio tracks. It probes the effectiveness of adversarial semi-supervised learning in separating sources without relying on perfectly paired mixture-source training data. Use when the user wants to benchmark on DSD100, iKala, MedleyDB, CCMixter, or asks about evaluating this task. Reports SDR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1711.00048
  bibtex_key: stoller2017adversarial
  confidence: high
---

# singing-voice-separation-eval

> Adversarial Semi-Supervised Audio Source Separation applied to Singing Voice Extraction — Stoller et al. (2017) (arXiv:1711.00048, 2017)

## What this evaluates

This benchmark evaluates a model's ability to extract singing vocals from mixed audio tracks. It probes the effectiveness of adversarial semi-supervised learning in separating sources without relying on perfectly paired mixture-source training data.

## Datasets

- **DSD100** — total ?; splits: train (-1), test (25)
- **iKala** — total ?; splits: unlabeled (-1), validation (-1), test (-1)
- **MedleyDB** — total ?; splits: unlabeled (-1), validation (-1), test (-1)
- **CCMixter** — total ?; splits: unlabeled (-1), validation (-1), test (-1)

## Metrics

- `SDR` **(primary)** — range: other
  - Track-wise normalised Signal-to-Distortion Ratio (SDR). Measures the ratio of energy in the target source to the total energy of distortion, interference, and artifacts in the estimate.
- `SIR` — range: other
  - Track-wise normalised Signal-to-Interference Ratio (SIR). Measures the ratio of energy in the target source to the energy of interfering sources in the estimate.
- `SAR` — range: other
  - Track-wise normalised Signal-to-Artifacts Ratio (SAR). Measures the ratio of energy in the target source to the energy of artifacts introduced by the separation process.

## Input / output format

**Input**: Mono audio downsampled to 8 kHz, converted to a magnitude spectrogram via a 512-point FFT with 50% overlap, and log-normalized. Batched into 64 spectrogram excerpts.

**Output**: Log-normalised magnitude spectrograms for each source, reconstructed into time-domain signals via inverse STFT using the mixture's phase.

## Scoring recipe

```python
def evaluate(predictions, targets):
    sdr_scores, sir_scores, sar_scores = [], [], []
    for pred, tgt in zip(predictions, targets):
        sdr_scores.append(normalised_sdr(pred, tgt))
        sir_scores.append(normalised_sir(pred, tgt))
        sar_scores.append(normalised_sar(pred, tgt))
    return {
        'SDR': sum(sdr_scores) / len(sdr_scores),
        'SIR': sum(sir_scores) / len(sir_scores),
        'SAR': sum(sar_scores) / len(sar_scores)
    }
```

## Common pitfalls

- Using paired mixture-source data for unsupervised training without accounting for dataset bias; the authors explicitly used multi-track stems but treated them as unpaired to isolate the regularization effect.
- Omitting the accompaniment discriminator, which can bias solutions towards favouring vocal quality over accompaniment quality.
- Failing to match the exact 8 kHz mono preprocessing and log-normalization, which significantly alters the spectrogram distributions fed to the model.

## Evidence (verbatim from paper)

> For evaluation, we calculate the track-wise (normalised) SDR, SIR, and SAR metrics*[[25](#bib.bib25 "")]*, with mono estimates and target signals sampled at 8 KHz. Table[2](#S4.F2 "Figure 2 ‣ 4.3.1 Preprocessing ‣ 4.3 Experimental setup ‣ 4 Singing voice separation experiment ‣ Adversarial Semi-Supervised Audio Source Separation applied to Singing Voice Extraction") shows averages over the test set and its subsets containing only tracks from a specific data source.

## Citation

```bibtex
@misc{stoller2017adversarial,
  title={Adversarial Semi-Supervised Audio Source Separation applied to Singing Voice Extraction},
  author={Stoller et al. (2017)},
  year={2017},
  note={arXiv:1711.00048}
}
```

- arXiv: 1711.00048