Dns Challenge Eval

---
name: dns-challenge-eval
description: Evaluates the perceptual quality and intelligibility of deep noise suppression models under real-world, non-stationary noise conditions. It specifically probes whether models generalize from synthetic training data to real-world acoustic environments. Use when the user wants to benchmark on DNS Challenge Dataset, or asks about evaluating this task. Reports ITU-T P.808.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2005.13981
  bibtex_key: reddy2020dnschallenge
  confidence: medium
---

# dns-challenge-eval

> The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Results — Reddy et al. (2020) (arXiv:2005.13981, 2020)

## What this evaluates

Evaluates the perceptual quality and intelligibility of deep noise suppression models under real-world, non-stationary noise conditions. It specifically probes whether models generalize from synthetic training data to real-world acoustic environments.

## Datasets

- **DNS Challenge Dataset** — total ?; splits: test (-1); repo https://github.com/microsoft/DNS-Challenge

## Metrics

- `ITU-T P.808` **(primary)** — range: other
  - Crowd-sourced subjective Mean Opinion Score derived from the ITU-T P.808 framework, measuring perceived speech quality after enhancement.
- `PESQ` — range: [0, 1]
  - Perceptual Evaluation of Speech Quality, a standard objective metric for speech enhancement that correlates with human listening tests.
- `STOI` — range: [0, 1]
  - Short-Time Objective Intelligibility, a metric that predicts speech intelligibility based on temporal envelope correlation.

## Input / output format

**Input**: Paired clean speech and noisy speech recordings (real-world non-stationary noise conditions).

**Output**: Enhanced speech waveform.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    # predictions and gold are aligned speech waveforms
    pesq_scores = [pesq(gold[i], predictions[i]) for i in range(len(gold))]
    stoi_scores = [stoi(gold[i], predictions[i]) for i in range(len(gold))]
    # Subjective MOS is obtained via crowd-sourced ITU-T P.808 framework
    mos_scores = get_subjective_mos(predictions)
    return {
        'PESQ': np.mean(pesq_scores),
        'STOI': np.mean(stoi_scores),
        'ITU-T P.808': np.mean(mos_scores)
    }
```

## Common pitfalls

- Models trained exclusively on synthetic noise often degrade significantly when evaluated on real-world recordings, making synthetic-only testing misleading.
- Subjective evaluation requires strict adherence to the ITU-T P.808 framework and careful crowd-sourcing calibration to ensure perceptual scores align with human listeners.

## Evidence (verbatim from paper)

> The INTERSPEECH 2020 Deep Noise Suppression Challenge introduces a large-scale, real-world representative dataset of clean speech and noise recordings derived from Librivox and Audioset/Freesound, with a focus on high-quality, non-stationary noise conditions. It presents an open-source, scalable online subjective evaluation framework based on ITU-T P.808 to enable reliable, crowd-sourced perceptual testing, addressing the gap between synthetic test performance and real-world deployment.

## Citation

```bibtex
@misc{reddy2020dnschallenge,
  title={The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Results},
  author={Reddy et al. (2020)},
  year={2020},
  note={arXiv:2005.13981}
}
```

- arXiv: 2005.13981