Ersb Eval

---
name: ersb-eval
description: This benchmark evaluates the environmental resilience of discrete speech codecs by measuring how reconstruction quality and downstream task performance degrade under varying signal-to-noise ratios, loudness levels, and real-world acoustic conditions. It probes both signal fidelity and semantic/intelligibility consistency after codec compression and subsequent speech enhancement or recognition. Use when the user wants to benchmark on Environment-Resilient Speech Codec Benchmark (ERSB), or asks about evaluating this task. Reports PESQ, STOI.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.22515
  bibtex_key: wang2025towards
  confidence: high
---

# ersb-eval

> Towards General Discrete Speech Codec for Complex Acoustic Environments: A Study of Reconstruction and Downstream Task Consistency — Wang et al. (2025) (arXiv:2505.22515, 2025)

## What this evaluates

This benchmark evaluates the environmental resilience of discrete speech codecs by measuring how reconstruction quality and downstream task performance degrade under varying signal-to-noise ratios, loudness levels, and real-world acoustic conditions. It probes both signal fidelity and semantic/intelligibility consistency after codec compression and subsequent speech enhancement or recognition.

## Datasets

- **Environment-Resilient Speech Codec Benchmark (ERSB)** — total ?; splits: simulated (-1), real-world (-1)

## Metrics

- `PESQ` **(primary)** — range: other
  - Perceptual Evaluation of Speech Quality score computed between reference and reconstructed audio using the VERSA toolkit.
- `STOI` **(primary)** — range: [0, 1]
  - Short-Time Objective Intelligibility score measuring speech intelligibility between reference and reconstructed audio.
- `SI-SDR` — range: dB
  - Scale-Invariant Signal-to-Distortion Ratio in dB, measuring enhancement performance before and after codec reconstruction.
- `WER` — range: percent
  - Word Error Rate in percent, measuring ASR transcription accuracy on enhanced speech after codec reconstruction.

## Input / output format

**Input**: Clean or noisy speech audio (simulated via DNS5/CHiME1/RIR mixing or real-world recordings from DNS1/CHiME4).

**Output**: Reconstructed speech audio after codec compression/decompression. Downstream outputs include enhanced audio (via SepFormer) and transcriptions (via Whisper Large-v3).

## Scoring recipe

```python
# Reconstruction metrics
pesq = compute_pesq(ref_audio, codec_out)
stoi = compute_stoi(ref_audio, codec_out)

# Downstream consistency metrics
enh_codec = run_se(codec_out)  # SepFormer
enh_ref = run_se(ref_audio)
delta_si_sdr = compute_si_sdr(enh_codec, ref_audio) - compute_si_sdr(enh_ref, ref_audio)

trans_codec = run_asr(enh_codec)  # Whisper
trans_ref = run_asr(enh_ref)
delta_wer = compute_wer(trans_codec, ref_text) - compute_wer(trans_ref, ref_text)
```

## Common pitfalls

- ΔSI-SDR and ΔWER measure relative degradation rather than absolute performance; negative ΔSI-SDR or positive ΔWER indicates codec-induced harm.
- Simulated SNR/loudness variations do not capture real-world noise distributions; real-world tests require TSI-SDR binning via TorchAudio-Squim.
- Codecs are evaluated at fixed bitrates; comparing across different bitrate configurations without normalization can skew results.

## Evidence (verbatim from paper)

> For the evaluation of signal metrics, we select the VERSA toolkit [22] to measure the PESQ, STOI, and SI-SDR metrics. ... Table 3: ΔTSI-SDR (dB) of SE task and ΔWER (%) of SE+ASR task on the TSI-SDR-divided CHiME4.

## Citation

```bibtex
@misc{wang2025towards,
  title={Towards General Discrete Speech Codec for Complex Acoustic Environments: A Study of Reconstruction and Downstream Task Consistency},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2505.22515}
}
```

- arXiv: 2505.22515