Chime4 Adapter Eval

---
name: chime4-adapter-eval
description: Evaluates the robustness of Automatic Speech Recognition (ASR) models under various real-world noise conditions (bus, cafe, pedestrian, street junction) using adapter-based fine-tuning and speech enhancement front-ends. Use when the user wants to benchmark on CHiME-4, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.18275
  bibtex_key: shi2024explorationadapter
  confidence: high
---

# chime4-adapter-eval

> Exploration of Adapter for Noise Robust Automatic Speech Recognition — Hao Shi et al. (2024) (arXiv:2402.18275, 2024)

## What this evaluates

Evaluates the robustness of Automatic Speech Recognition (ASR) models under various real-world noise conditions (bus, cafe, pedestrian, street junction) using adapter-based fine-tuning and speech enhancement front-ends.

## Datasets

- **CHiME-4** — total ?; splits: train (-1), dev (-1), test (-1); repo https://spandh.dcs.shef.ac.uk/chime_challenge/CHiME4/index.html

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate (WER) = (Substitutions + Deletions + Insertions) / Total Reference Words × 100%. Standard ASR evaluation metric reported as a percentage.

## Input / output format

**Input**: 16 kHz audio recordings processed into 80-dimension LFMB features, with corresponding ground-truth transcripts.

**Output**: Predicted text transcription.

## Scoring recipe

```python
def compute_wer(predictions, references):
    total_words = sum(len(ref.split()) for ref in references)
    errors = 0
    for pred, ref in zip(predictions, references):
        errors += edit_distance(pred.split(), ref.split())
    return (errors / total_words) * 100 if total_words > 0 else 0.0
```

## Common pitfalls

- Confusing the development set (channels 1-6) with the evaluation set (channel 5) for final reporting.
- Using simulated noisy data for the final metric calculation instead of the real noisy test sets.
- Failing to average WER across the four distinct noise conditions (bus, str, ped, caf) before reporting the overall score.

## Evidence (verbatim from paper)

> Table 1: The performance of baseline pretrained ASR model. ... The Channel 5 real noisy data from development and evaluation sets were used for testing.

## Citation

```bibtex
@misc{shi2024explorationadapter,
  title={Exploration of Adapter for Noise Robust Automatic Speech Recognition},
  author={Hao Shi et al. (2024)},
  year={2024},
  note={arXiv:2402.18275}
}
```

- arXiv: 2402.18275