Voicebank Demand Sep Eval

---
name: voicebank-demand-sep-eval
description: Evaluates zero-shot language-queried speech enhancement by isolating clean speech from noisy backgrounds. The benchmark tests the model's ability to enhance speech using a fixed text query. Use when the user wants to benchmark on Voicebank-DEMAND, or asks about evaluating this task. Reports PESQ.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2308.05037
  bibtex_key: liu2023separate
  confidence: high
---

# voicebank-demand-sep-eval

> Separate Anything You Describe — Liu et al. (2023) (arXiv:2308.05037, 2023)

## What this evaluates

Evaluates zero-shot language-queried speech enhancement by isolating clean speech from noisy backgrounds. The benchmark tests the model's ability to enhance speech using a fixed text query.

## Datasets

- **Voicebank-DEMAND** — total 824; splits: test (824)

## Metrics

- `PESQ` **(primary)** — range: score
  - Perceptual Evaluation of Speech Quality, a standardized objective measure of speech quality.
- `CSIG` — range: score
  - MOS predictor of signal distortion, part of the P.835 standard.
- `CBAK` — range: score
  - MOS predictor of background-noise intrusiveness, part of the P.835 standard.
- `COVL` — range: score
  - MOS predictor of overall signal quality, part of the P.835 standard.
- `SSNR` — range: dB
  - Segmental signal-to-noise ratio, measures SNR over short time frames.

## Input / output format

**Input**: A noisy speech utterance (mixed at 15, 10, 5, or 0 dB SNR) paired with the fixed text query 'Speech'.

**Output**: Enhanced/cleaned speech waveform corresponding to the target utterance.

## Scoring recipe

```python
def score(pred, gold):
  pesq = compute_pesq(pred, gold)
  csig = compute_csig(pred, gold)
  cbak = compute_cbak(pred, gold)
  covl = compute_covl(pred, gold)
  ssnr = compute_ssnr(pred, gold)
  return pesq, csig, cbak, covl, ssnr
```

## Common pitfalls

- All audio clips are resampled to 16 kHz for fair comparison with prior speech enhancement systems.
- The text query is fixed to 'Speech' rather than being a natural language description of the speaker or content.

## Evidence (verbatim from paper)

> The test set of the Voicebank-DEMAND dataset includes a total of 824 utterances, which is used to evaluate the zero-shot performance of our model on speech enhancement. ... We use 'Speech' as the input text query to perform speech enhancement. ... we apply the perceptual evaluation of speech quality (PESQ), mean opinion score (MOS) predictor of signal distortion (CSIG), MOS predictor of background-noise intrusiveness (CBAK), MOS predictor of overall signal quality (COVL) and segmental signal-to-ratio noise (SSNR) for evaluation.

## Citation

```bibtex
@misc{liu2023separate,
  title={Separate Anything You Describe},
  author={Liu et al. (2023)},
  year={2023},
  note={arXiv:2308.05037}
}
```

- arXiv: 2308.05037