Moisesdb Eval

---
name: moisesdb-eval
description: Evaluates the ability of audio source separation models to isolate individual instrument stems from mixed stereo recordings. It probes fine-grained separation capabilities across a hierarchical taxonomy of up to 11 stems, testing robustness to stem imbalance and un-mastered audio characteristics. Use when the user wants to benchmark on MoisesDB, or asks about evaluating this task. Reports SDR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2307.15913
  bibtex_key: pereira2023moisesdb
  confidence: high
---

# moisesdb-eval

> Moisesdb: A dataset for source separation beyond 4-stems — Pereira et al. (2023) (arXiv:2307.15913, 2023)

## What this evaluates

Evaluates the ability of audio source separation models to isolate individual instrument stems from mixed stereo recordings. It probes fine-grained separation capabilities across a hierarchical taxonomy of up to 11 stems, testing robustness to stem imbalance and un-mastered audio characteristics.

## Datasets

- **MoisesDB** — total 240; splits: test (-1); repo https://github.com/moises-ai/moises-db

## Metrics

- `SDR` **(primary)** — range: dB
  - Signal-to-Distortion Ratio in decibels. Computed as 10 * log10(||s_true||^2 / ||s_true - s_est||^2), where s_true is the ground truth stem and s_est is the estimated stem. Higher values indicate better separation quality.

## Input / output format

**Input**: Stereo mixed audio recording of a song.

**Output**: One or more separated audio stems (waveform files) corresponding to the requested instrument classes (e.g., vocals, drums, bass, piano, guitar, etc.).

## Scoring recipe

```python
def compute_sdr(gt, est):
    num = np.sum(gt ** 2)
    den = np.sum((gt - est) ** 2)
    return 10 * np.log10(num / den) if den > 0 else float('inf')

def evaluate(dataset, preds):
    scores = []
    for track in dataset:
        for stem in track.stems:
            scores.append(compute_sdr(track[stem], preds[track.id][stem]))
    return np.mean(scores)
```

## Common pitfalls

- The dataset contains un-mastered mixes with lower loudness and higher dynamic range, causing distributional shift when evaluating on commercial mastered tracks.
- Stem and track frequencies follow a power-law distribution, making underrepresented stems (e.g., 'wind', 'contrabass') significantly harder to separate and potentially skewing overall averages.
- Not all songs contain all stems; the number of available tracks varies per stem configuration (e.g., N=235 for 4-stem, N=104 for 5-stem), so metrics must be averaged over available tracks rather than assuming a fixed set.

## Evidence (verbatim from paper)

> Table 3: Mean, standard deviation (Std), and median (Mdn) of the SDR in dB for each Model/Method and stem type. The varying number of available tracks is denoted by N. *Overall* indicates performance over all tracks regardless of stem group.

## Citation

```bibtex
@misc{pereira2023moisesdb,
  title={Moisesdb: A dataset for source separation beyond 4-stems},
  author={Pereira et al. (2023)},
  year={2023},
  note={arXiv:2307.15913}
}
```

- arXiv: 2307.15913