Podcastmix Eval

---
name: podcastmix-eval
description: Evaluates the quality of monaural music and speech source separation in podcast audio. It measures both objective signal fidelity using BSS-eval metrics and subjective perceptual quality using standardized listening tests. Use when the user wants to benchmark on PodcastMix, or asks about evaluating this task. Reports SDR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2207.07403
  bibtex_key: schmidt2022podcastmix
  confidence: high
---

# podcastmix-eval

> PodcastMix: A dataset for separating music and speech in podcasts — Schmidt et al. (2022) (arXiv:2207.07403, 2022)

## What this evaluates

Evaluates the quality of monaural music and speech source separation in podcast audio. It measures both objective signal fidelity using BSS-eval metrics and subjective perceptual quality using standardized listening tests.

## Datasets

- **PodcastMix** — total ?; splits: synth test (-1), real with-reference (-1), real no-reference (-1); repo https://github.com/MTG/Podcastmix

## Metrics

- `SDR` **(primary)** — range: dB
  - Source-to-distortion ratio measuring the ratio of target source energy to distortion energy in decibels.
- `SI-SDR` — range: dB
  - Scale-invariant SDR, computed after optimal scaling of the estimate to match the reference, robust to amplitude changes.
- `OVRL` — range: [1, 5]
  - Overall quality mean opinion score from webMUSHRA subjective tests, rated on a 1-5 scale.
- `SIG` — range: [1, 5]
  - Speech quality/distortion mean opinion score from webMUSHRA, rated on a 1-5 scale.
- `BAK` — range: [1, 5]
  - Background intrusiveness mean opinion score from webMUSHRA, rated on a 1-5 scale.

## Input / output format

**Input**: 44.1kHz mono audio waveform of a podcast mixture containing speech and music.

**Output**: Two separated audio waveforms: one for speech and one for music.

## Scoring recipe

```python
def compute_metrics(reference, estimate, subjective_ratings=None):
    sdr = bss_eval_sdr(reference, estimate)
    si_sdr = compute_si_sdr(reference, estimate)
    if subjective_ratings:
        ovrl = mean(r.ovrl for r in subjective_ratings)
        sig = mean(r.sig for r in subjective_ratings)
        bak = mean(r.bak for r in subjective_ratings)
    return {'SDR': sdr, 'SI-SDR': si_sdr, 'OVRL': ovrl, 'SIG': sig, 'BAK': bak}
```

## Common pitfalls

- BSS-eval metrics (SDR, SIR, SAR) require reference stems, so they cannot be computed for the 'PodcastMix-real no-reference' split.
- Models trained on synthetic data often show severe generalization gaps on real podcast audio, leading to artificially low objective and subjective scores.
- Subjective testing splits listeners: naive listeners only rate OVRL, while experienced listeners rate SIG and BAK, making direct comparison across all metrics require careful pooling.

## Evidence (verbatim from paper)

> We primarily rely on BSS_eval metrics: source-to-distortion ratio (SDR), source-to-interference ratio (SIR), and source-to-artifact ratio (SAR). Further, as recommended by previous works, we also report scale-invariant SDR (SI-SDR). Since the above metrics require reference stems, we report those for PodcastMix-synth test and PodcastMix-real with-reference sets.

## Citation

```bibtex
@misc{schmidt2022podcastmix,
  title={PodcastMix: A dataset for separating music and speech in podcasts},
  author={Schmidt et al. (2022)},
  year={2022},
  note={arXiv:2207.07403}
}
```

- arXiv: 2207.07403