Fuss Sound Separation Eval

---
name: fuss-sound-separation-eval
description: Evaluates audio source separation models on mixed reverberant or dry recordings containing 1–4 active sources. It measures reconstruction fidelity and source-counting accuracy across varying mixture complexities. Use when the user wants to benchmark on FUSS, or asks about evaluating this task. Reports SI-SNR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2011.00803
  bibtex_key: wisdom2020fuss
  confidence: high
---

# fuss-sound-separation-eval

> What's All the FUSS About Free Universal Sound Separation Data? — Wisdom et al. (2020) (arXiv:2011.00803, 2020)

## What this evaluates

Evaluates audio source separation models on mixed reverberant or dry recordings containing 1–4 active sources. It measures reconstruction fidelity and source-counting accuracy across varying mixture complexities.

## Datasets

- **FUSS** — total ?; splits: val (-1), test (-1)

## Metrics

- `SI-SNR` **(primary)** — range: dB
  - Scale-invariant signal-to-noise ratio computed via cosine similarity: SI-SNR(y, ŷ) ≈ 10 log10((ρ²(y, ŷ) + ε) / (1 - ρ²(y, ŷ) + ε)), where ρ is the cosine similarity between target y and estimate ŷ, and ε=10⁻⁸. Estimates are first aligned to references via permutation matching, and pairs with inactive references or estimates >20 dB below the quietest active reference are discarded.

## Input / output format

**Input**: Mixed audio waveforms (dry or reverberant) containing 1 to 4 active sound sources.

**Output**: Separated audio waveforms corresponding to each estimated source.

## Scoring recipe

```python
def compute_sisnr(y, y_hat, eps=1e-8):
    # 1. Align estimates to references by maximizing SI-SNR over permutations
    # 2. Discard pairs where y is all-zeros or power(y_hat) < power(quietest_nonzero_ref) - 20dB
    rho = np.dot(y, y_hat) / (np.linalg.norm(y) * np.linalg.norm(y_hat) + eps)
    rho_sq = rho ** 2
    sisnr = 10 * np.log10((rho_sq + eps) / (1 - rho_sq + eps))
    return sisnr
```

## Common pitfalls

- Using the original scale-based SI-SNR formula instead of the cosine-similarity variant yields overly optimistic scores, particularly for under-separation.
- Failing to align estimates to references via permutation matching before computing scores.
- Not filtering out estimate-reference pairs where the reference is inactive or the estimate power is >20 dB below the quietest active reference.

## Evidence (verbatim from paper)

> To evaluate performance, we use scale-invariant signal-to-noise ratio (SI-SNR)... To correct for this, we use an alternate formulation, SI-SNR(y,ŷ) = 10 log10(ρ²(y,ŷ)/(1-ρ²(y,ŷ))) ≈ 10 log10((ρ²(y,ŷ)+ε)/(1-ρ²(y,ŷ)+ε)), where ρ(y,ŷ) = yᵀŷ/(||y||||ŷ||) ≈ yᵀŷ/(||y||||ŷ||+ε) is the cosine similarity between y and ŷ, with stabilizer ε. In our experiments we use ε=10⁻⁸.

## Citation

```bibtex
@misc{wisdom2020fuss,
  title={What's All the FUSS About Free Universal Sound Separation Data?},
  author={Wisdom et al. (2020)},
  year={2020},
  note={arXiv:2011.00803}
}
```

- arXiv: 2011.00803