Chime3 Se Eval

---
name: chime3-se-eval
description: This benchmark evaluates speech enhancement models by measuring how well they clean noisy speech features before they are processed by a downstream automatic speech recognition (ASR) system. It probes the model's ability to preserve speech structure and reduce noise in challenging real-world far-field conditions. Use when the user wants to benchmark on CHiME-3, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1809.02253
  bibtex_key: meng2018cycleconsistent
  confidence: high
---

# chime3-se-eval

> Cycle-Consistent Speech Enhancement — Meng et al. (2018) (arXiv:1809.02253, 2018)

## What this evaluates

This benchmark evaluates speech enhancement models by measuring how well they clean noisy speech features before they are processed by a downstream automatic speech recognition (ASR) system. It probes the model's ability to preserve speech structure and reduce noise in challenging real-world far-field conditions.

## Datasets

- **CHiME-3** — total ?; splits: train (9137), test (-1)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate (WER) computed by decoding enhanced speech features using a standard WSJ 5K word 3-gram language model and a clean DNN acoustic model.
- `RWERR` — range: percent
  - Relative WER Reduction calculated as (WER_baseline - WER_method) / WER_baseline * 100%.

## Input / output format

**Input**: 87-dimensional vectors formed by appending 1st and 2nd order delta features to 29-dimensional log Mel filterbank (LFB) features, globally mean and variance normalized.

**Output**: 29-dimensional enhanced log Mel filterbank (LFB) features.

## Scoring recipe

```python
enhanced_features = model_F(noisy_features)
transcript = asr_decoder(
    enhanced_features, 
    lm="WSJ_5K_3gram", 
    am="clean_DNN"
)
wer = compute_wer(transcript, reference_transcript)
rwerr = (wer_baseline - wer) / wer_baseline * 100
```

## Common pitfalls

- The evaluation metric is downstream ASR Word Error Rate (WER), not direct speech quality metrics like PESQ or STOI.
- The test set specifically uses only the 5th microphone channel from the CHiME-3 development set, not the full dataset.
- In Section 4.3, the acoustic model is re-trained on enhanced training data, which changes the evaluation protocol compared to the fixed acoustic model in Sections 4.1 and 4.2.

## Evidence (verbatim from paper)

> The real far-field noisy speech from the 5th microphone channel in CHiME-3 development data set is used for testing. We pre-train a clean DNN acoustic model as in Section 3.2 of [[31]] using 9137 clean training utterances in CHiME-3 to evaluate the ASR word error rate (WER) performance of the test features enhanced by F. The acoustic model is further re-trained with enhanced feature for better WERs. A standard WSJ 5K word 3-gram language model is used for decoding.

## Citation

```bibtex
@misc{meng2018cycleconsistent,
  title={Cycle-Consistent Speech Enhancement},
  author={Meng et al. (2018)},
  year={2018},
  note={arXiv:1809.02253}
}
```

- arXiv: 1809.02253