Robust Asr Eval

---
name: robust-asr-eval
description: Evaluates the robustness of monaural automatic speech recognition systems under noisy and reverberant conditions. It measures how well a decoupled frontend speech enhancement module improves the word error rate of a backend ASR model trained exclusively on clean speech. Use when the user wants to benchmark on WSJ0 SI-84, CHiME-2, LibriSpeech, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.06387
  bibtex_key: yang2024decoupling
  confidence: high
---

# robust-asr-eval

> Towards Decoupling Frontend Enhancement and Backend Recognition in Monaural Robust ASR — Yang et al. (2024) (arXiv:2403.06387, 2024)

## What this evaluates

Evaluates the robustness of monaural automatic speech recognition systems under noisy and reverberant conditions. It measures how well a decoupled frontend speech enhancement module improves the word error rate of a backend ASR model trained exclusively on clean speech.

## Datasets

- **WSJ0 SI-84** — total 8674; splits: train (7138), val (1206), test (330)
- **CHiME-2** — total ?; splits: train (7138), val (409), test (1980)
- **LibriSpeech** — total ?; splits: train (-1), dev (-1), test (-1)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate calculated as the percentage of incorrectly recognized words (substitutions, deletions, insertions) relative to the reference transcript length.
- `STOI` — range: [0, 1]
  - Short-Time Objective Intelligibility, a correlation-based metric that predicts speech intelligibility on a scale from 0 to 1.
- `PESQ` — range: [-0.5, 4.5]
  - Perceptual Evaluation of Speech Quality, a standardized ITU-T metric for assessing speech quality, typically ranging from -0.5 to 4.5.

## Input / output format

**Input**: 16 kHz monaural speech waveform. For SE, noisy/reverberant speech frames (16 ms window, 2 ms shift, RMS normalized). For ASR, log-Mel spectrograms derived from STFT of enhanced or raw speech.

**Output**: For SE: enhanced speech waveform. For ASR: recognized text transcript.

## Scoring recipe

```python
def compute_metrics(clean_ref, enhanced_pred, asr_hypothesis, asr_reference):
    stoi = compute_stoi(clean_ref, enhanced_pred)
    pesq = compute_pesq(clean_ref, enhanced_pred, fs=16000)
    wer = (edit_distance(asr_hypothesis, asr_reference) / len(asr_reference.split())) * 100
    return {'STOI': stoi, 'PESQ': pesq, 'WER': wer}
```

## Common pitfalls

- Frontend and backend are trained independently; joint training is explicitly avoided to prevent distortion-induced mismatch.
- Reverberation test conditions use direct-path speech alignment to ensure fair WER comparison across different RIRs.
- Noise and reverberation are generated dynamically during training, so models are evaluated on unseen RIRs and noise types.

## Evidence (verbatim from paper)

> The SE performance is evaluated using standard STOI and perceptual evaluation of speech quality (PESQ) metrics. STOI ranges typically between [0, 1] and indicates speech intelligibility, usually in percentage. PESQ ranges between [-0.5, 4.5] and a higher score denotes higher speech quality. TABLE I: ASR (%WER) Results of the Proposed DN-ARN System and Comparison Systems on WSJ.

## Citation

```bibtex
@misc{yang2024decoupling,
  title={Towards Decoupling Frontend Enhancement and Backend Recognition in Monaural Robust ASR},
  author={Yang et al. (2024)},
  year={2024},
  note={arXiv:2403.06387}
}
```

- arXiv: 2403.06387