Sound Demixing Challenge 2023 Eval

---
name: sound-demixing-challenge-2023-eval
description: Evaluates music source separation models on their ability to isolate individual instruments (vocals, bass, drums, other) from mixed audio tracks. It specifically probes robustness to label noise and bleeding artifacts in training data, as well as standard separation performance across different leaderboards. Use when the user wants to benchmark on SDXDB23_LabelNoise, SDXDB23_Bleeding, Standard (MDXDB21), or asks about evaluating this task. Reports SDR (Signal-to-Distortion Ratio).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2308.06979
  bibtex_key: fabbro2023sounddemixing
  confidence: high
---

# sound-demixing-challenge-2023-eval

> The Sound Demixing Challenge 2023 $\unicode{x2013}$ Music Demixing Track — Fabbro et al. (2023) (arXiv:2308.06979, 2023)

## What this evaluates

Evaluates music source separation models on their ability to isolate individual instruments (vocals, bass, drums, other) from mixed audio tracks. It specifically probes robustness to label noise and bleeding artifacts in training data, as well as standard separation performance across different leaderboards.

## Datasets

- **SDXDB23_LabelNoise** — total ?; splits: test (-1)
- **SDXDB23_Bleeding** — total ?; splits: test (-1)
- **Standard (MDXDB21)** — total ?; splits: test (-1)

## Metrics

- `SDR (Signal-to-Distortion Ratio)` **(primary)** — range: dB
  - Computed in decibels (dB) as 10 * log10(||target||^2 / ||prediction - target||^2). The paper reports Global SDR (mean across all instruments) and per-instrument SDR for Bass, Drums, Other, and Vocals.

## Input / output format

**Input**: Mixed audio waveform of a song containing multiple instruments.

**Output**: Four separated audio waveforms corresponding to the target instruments: Bass, Drums, Other, and Vocals.

## Scoring recipe

```python
def compute_sdr(pred, gt):
    target_energy = np.sum(gt ** 2)
    error_energy = np.sum((pred - gt) ** 2)
    if error_energy == 0:
        return float('inf')
    return 10 * np.log10(target_energy / error_energy)

final_score = np.mean([compute_sdr(pred_bass, gt_bass),
                       compute_sdr(pred_drums, gt_drums),
                       compute_sdr(pred_other, gt_other),
                       compute_sdr(pred_vocals, gt_vocals)])
```

## Common pitfalls

- SDR is reported in decibels (dB), not as a linear ratio; higher values indicate better separation quality.
- The challenge uses different leaderboards for different noise conditions (LabelNoise, Bleeding, Standard), so results are not directly comparable across leaderboards without noting the training data corruption type.
- Ensemble methods combine multiple models per instrument, meaning the evaluation must account for per-source weighting rather than a single monolithic model output.

## Evidence (verbatim from paper)

> Table 1: Final LabelNoise leaderboard (models trained only on SDXDB23_LabelNoise; top-5) ... Global SDR (dB) ... We report in Table 4 the performance of our iterative refinement baseline. First of all, we highlight the impact that the errors in the data have on the performance of the model: training on SDXDB23_LabelNoise degrades the average separation quality by 1.42dB, while training on SDXDB23_Bleeding degrades it by 0.83dB.

## Citation

```bibtex
@misc{fabbro2023sounddemixing,
  title={The Sound Demixing Challenge 2023 $\unicode{x2013}$ Music Demixing Track},
  author={Fabbro et al. (2023)},
  year={2023},
  note={arXiv:2308.06979}
}
```

- arXiv: 2308.06979