Whamr Eval

---
name: whamr-eval
description: Evaluates single-channel speech separation and enhancement (denoising/dereverberation) capabilities under realistic noisy and reverberant conditions using synthetically generated reverberant mixtures. Use when the user wants to benchmark on WHAMR!, or asks about evaluating this task. Reports SI-SDR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1910.10279
  bibtex_key: maciejewski2019whamr
  confidence: high
---

# whamr-eval

> WHAMR!: Noisy and Reverberant Single-Channel Speech Separation — Maciejewski et al. (2019) (arXiv:1910.10279, 2019)

## What this evaluates

Evaluates single-channel speech separation and enhancement (denoising/dereverberation) capabilities under realistic noisy and reverberant conditions using synthetically generated reverberant mixtures.

## Datasets

- **WHAMR!** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `SI-SDR` **(primary)** — range: other
  - Scale-invariant signal-to-distortion ratio. Formula: 10log10(||αs||² / ||αs - ŝ||²), where α = ⟨ŝ, s⟩ / ||s||². It is scale-invariant and used for both training and evaluation.

## Input / output format

**Input**: 4-second audio segments of single-channel mixtures containing speech sources, noise, and/or reverberation.

**Output**: Estimated waveform for each target source (or enhanced source).

## Scoring recipe

```python
def compute_si_sdr(target, estimate):
    alpha = np.dot(estimate, target) / np.dot(target, target)
    scaled_target = alpha * target
    noise = scaled_target - estimate
    s_power = np.dot(scaled_target, scaled_target)
    n_power = np.dot(noise, noise)
    return 10 * np.log10(s_power / n_power)
```

## Common pitfalls

- Outputs from cascaded models may have dynamic range mismatches due to the scale-invariant loss, requiring explicit rescaling before cascading or evaluation.
- SI-SDR is undefined for silent sources, so training on large subsets with sparse speech can cause instability; the authors train on a smaller subset to avoid silent segments.

## Evidence (verbatim from paper)

> We train all networks using permutation invariant training with the scale-invariant signal-to-distortion ratio (SI-SDR, also referred to as SI-SNR) waveform-level training objective. SI-SDR is also the evaluation metric and allows for end-to-end joint training of cascaded enhancement and separation models: SI-SDR = 10log10(||αs||² / ||αs - ŝ||²), α = ⟨ŝ, s⟩ / ||s||².

## Citation

```bibtex
@misc{maciejewski2019whamr,
  title={WHAMR!: Noisy and Reverberant Single-Channel Speech Separation},
  author={Maciejewski et al. (2019)},
  year={2019},
  note={arXiv:1910.10279}
}
```

- arXiv: 1910.10279