Discongan Se Eval

---
name: discongan-se-eval
description: Evaluates speech enhancement models in extremely low SNR conditions by measuring noise suppression, speech quality preservation, and intelligibility using both objective metrics and subjective listening tests. Use when the user wants to benchmark on Low-SNR Dataset, VB-DMD Dataset, DNS Non-Reverb Test Dataset, DNS Real Recordings, or asks about evaluating this task. Reports PESQ.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.20859
  bibtex_key: shetu2025discongan
  confidence: high
---

# discongan-se-eval

> Leveraging Discriminative Latent Representations for Conditioning GAN-Based Speech Enhancement — Shrishti Saha Shetu, E.A.P. Habets, Andreas Brendel (2025) (arXiv:2508.20859, 2025)

## What this evaluates

Evaluates speech enhancement models in extremely low SNR conditions by measuring noise suppression, speech quality preservation, and intelligibility using both objective metrics and subjective listening tests.

## Datasets

- **Low-SNR Dataset** — total 1200; splits: test (1200); repo https://github.com/fhgainr/Low-SNR-Eval-Data
- **VB-DMD Dataset** — total ?; splits: test (-1)
- **DNS Non-Reverb Test Dataset** — total ?; splits: test (-1)
- **DNS Real Recordings** — total 300; splits: test (300)

## Metrics

- `PESQ` **(primary)** — range: [1, 4.5]
  - Perceptual Evaluation of Speech Quality; scores range from 1 (poor) to 4.5 (excellent) based on psychoacoustic modeling of human hearing.
- `SI-SDR` — range: dB
  - Scale-Invariant Signal-to-Distortion Ratio; measures overall quality independent of amplitude scaling, calculated as the ratio of target signal energy to distortion energy.
- `FwSegSNR` — range: dB
  - Frequency-weighted Segmental SNR; measures segmental SNR with perceptual weighting in the frequency domain.
- `WER/CER` — range: percent
  - Word Error Rate and Character Error Rate computed using Whisper ASR on enhanced vs. clean speech signals.
- `WebMUSHRA MOS` — range: [0, 100]
  - Subjective overall quality rating on a 0–100 scale via multi-stimuli listening test with 14 participants.

## Input / output format

**Input**: Noisy speech audio segments (typically 10 seconds long) sampled at 16 kHz.

**Output**: Enhanced speech audio (16 kHz), evaluated against clean references or via non-intrusive predictors.

## Scoring recipe

```python
def evaluate(pred, ref):
    pesq_val = pesq(ref, pred, fs=16000)
    si_sdr_val = scale_invariant_sdr(ref, pred)
    fwseg_val = freq_weighted_seg_snr(ref, pred)
    wer_val = whisper_wer(ref, pred)
    return {'PESQ': pesq_val, 'SI-SDR': si_sdr_val, 'FwSegSNR': fwseg_val, 'WER': wer_val}
```

## Common pitfalls

- Real recordings (DNS Real) lack clean references, so intrusive metrics like PESQ and SI-SDR cannot be computed; only non-intrusive metrics apply.
- WER/CER are computed using Whisper ASR on clean references rather than ground-truth transcripts, which may introduce transcription bias.
- The low-SNR test set uses non-uniform SNR intervals (e.g., [-15,-12], [-11,-8] dB), requiring careful grouping when reporting results.

## Evidence (verbatim from paper)

> To evaluate the proposed method and compare it with baseline methods, we used widely adopted intrusive and non-intrusive objective metrics. When clean references were available, we used several intrusive (full-reference) metrics: The perceptual evaluation of speech quality (PESQ) provides scores from 1 (poor) to 4.5 (excellent), including narrowband and wideband variants. The frequency-weighted segmental SNR (FwSegSNR) measures segmental SNR with perceptual weighting in the frequency domain. The scale-invariant signal-to-distortion ratio (SI-SDR) calculates overall quality independent of the amplitude scaling.

## Citation

```bibtex
@misc{shetu2025discongan,
  title={Leveraging Discriminative Latent Representations for Conditioning GAN-Based Speech Enhancement},
  author={Shrishti Saha Shetu, E.A.P. Habets, Andreas Brendel (2025)},
  year={2025},
  note={arXiv:2508.20859}
}
```

- arXiv: 2508.20859