Ears Wham Eval

---
name: ears-wham-eval
description: Evaluates speech enhancement models by measuring how well they recover clean speech from noisy mixtures across a wide range of signal-to-noise ratios and speaker demographics. The benchmark covers both controlled training/validation splits and a blind test set with unseen speakers and noise. Use when the user wants to benchmark on EARS-WHAM, or asks about evaluating this task. Reports SI-SDR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.06185
  bibtex_key: richter2024ears
  confidence: high
---

# ears-wham-eval

> EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberation — Richter et al. (2024) (arXiv:2406.06185, 2024)

## What this evaluates

Evaluates speech enhancement models by measuring how well they recover clean speech from noisy mixtures across a wide range of signal-to-noise ratios and speaker demographics. The benchmark covers both controlled training/validation splits and a blind test set with unseen speakers and noise.

## Datasets

- **EARS-WHAM** — total ?; splits: train (32485), val (632), test (886), blind_test (743)

## Metrics

- `SI-SDR` **(primary)** — range: dB
  - Scale-Invariant Signal-to-Distortion Ratio. Measures the ratio between the target signal energy and the residual noise/error energy, invariant to scaling.
- `POLQA` — range: [1, 5]
  - Perceptual Objective Listening Quality Analysis. ITU standard for predicting perceived speech quality based on psychoacoustic models.
- `PESQ` — range: [-0.5, 4.5]
  - Perceptual Evaluation of Speech Quality. ITU standard for predicting speech quality by comparing enhanced and reference signals.
- `ESTOI` — range: [0, 1]
  - Extended Short-Time Objective Intelligibility. Predicts speech intelligibility based on temporal envelope correlation in short-time segments.
- `SIGMOS` — range: [1, 5]
  - Speech Quality MOS. Deep learning-based predictor of mean opinion score for speech quality.
- `DNSMOS` — range: [1, 5]
  - Deep Noise Suppression MOS. Deep learning-based predictor of mean opinion score for speech quality and noise suppression.
- `WER` — range: percent
  - Word Error Rate. Percentage of words incorrectly recognized compared to the reference transcript.

## Input / output format

**Input**: Noisy audio file (and clean reference audio for intrusive metrics).

**Output**: Enhanced audio file.

## Scoring recipe

```python
def compute_metrics(predictions, references):
    scores = {}
    scores['SI-SDR'] = compute_si_sdr(predictions, references)  # dB
    scores['POLQA'] = compute_polqa(predictions, references)    # 1-5
    scores['PESQ'] = compute_pesq(predictions, references)      # -0.5 to 4.5
    scores['ESTOI'] = compute_estoi(predictions, references)    # 0-1
    scores['SIGMOS'] = compute_sigmos(predictions)              # 1-5
    scores['DNSMOS'] = compute_dnsmos(predictions)              # 1-5
    scores['WER'] = compute_wer(predictions, references)        # percent
    return scores
```

## Common pitfalls

- SNR is computed using LKFS loudness (ITU-R BS.1770) excluding silent regions, not raw RMS.
- Blind test set excludes ground truth and requires submission to an online evaluation server.
- Metrics are computed at specific maximum frequencies (e.g., SI-SDR at 24 kHz, WER at 8 kHz).

## Evidence (verbatim from paper)

> Table 4: Results on EARS-WHAM. Column groups are the method name, intrusive metrics, non-intrusive metrics, and WER. Below each metric is the maximum frequency taken into account for the assessment.

## Citation

```bibtex
@misc{richter2024ears,
  title={EARS: An Anechoic Fullband Speech Dataset Benchmarked for Speech Enhancement and Dereverberation},
  author={Richter et al. (2024)},
  year={2024},
  note={arXiv:2406.06185}
}
```

- arXiv: 2406.06185