Slakh2100 Sep Eval

---
name: slakh2100-sep-eval
description: Evaluates the ability of generative models to separate individual musical instrument stems from a mixed audio track. It probes how well the model captures inter-source dependencies and reconstructs clean waveforms for Bass, Drums, Guitar, and Piano. Use when the user wants to benchmark on Slakh2100, or asks about evaluating this task. Reports SI-SDR_i.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2302.02257
  bibtex_key: mariani2023multisource
  confidence: high
---

# slakh2100-sep-eval

> Multi-Source Diffusion Models for Simultaneous Music Generation and Separation — Mariani et al. (2023) (arXiv:2302.02257, 2023)

## What this evaluates

Evaluates the ability of generative models to separate individual musical instrument stems from a mixed audio track. It probes how well the model captures inter-source dependencies and reconstructs clean waveforms for Bass, Drums, Guitar, and Piano.

## Datasets

- **Slakh2100** — total 2100; splits: train (1500), val (375), test (225)

## Metrics

- `SI-SDR_i` **(primary)** — range: other
  - Scale-Invariant Signal-to-Distortion Ratio (improved). It measures the ratio between the target signal and the residual error, normalized by the target's energy to be scale-invariant. Formula: 10 log10(||s_hat^T s||^2 / ||s_hat^T s - s||^2) where s is the reference and s_hat is the estimate.

## Input / output format

**Input**: Mixed audio waveform downsampled to 22kHz, processed in ~12-second context windows.

**Output**: Separated waveform stems for each target instrument (Bass, Drums, Guitar, Piano).

## Scoring recipe

```python
def si_sdr_i(reference, estimate):
    alpha = np.dot(estimate, reference) / np.dot(reference, reference)
    target = alpha * reference
    noise = estimate - target
    sdr = 10 * np.log10(np.sum(target**2) / np.sum(noise**2))
    return sdr
```

## Common pitfalls

- Evaluating only on the four most abundant instruments (Bass, Drums, Guitar, Piano) rather than all 31 classes, which may overestimate generalization to unseen instruments.
- Ignoring the impact of the stochasticity parameter (S_churn) and correction steps (R), which significantly alter sample quality and final SI-SDR_i scores.
- Using context windows of ~12 seconds without specifying boundary handling or padding, which can artificially inflate or deflate separation quality at track edges.

## Evidence (verbatim from paper)

> We use the $	ext{SI-SDR}_{	ext{i}}$ as our evaluation metric (dB – higher is better).

## Citation

```bibtex
@misc{mariani2023multisource,
  title={Multi-Source Diffusion Models for Simultaneous Music Generation and Separation},
  author={Mariani et al. (2023)},
  year={2023},
  note={arXiv:2302.02257}
}
```

- arXiv: 2302.02257