Chime4 Ami Asr Eval

---
name: chime4-ami-asr-eval
description: Evaluates automatic speech recognition systems on real-world meeting and close-talking microphone speech. It measures how well acoustic models trained on raw waveforms generalize to challenging multi-microphone environments compared to traditional feature-based baselines. Use when the user wants to benchmark on CHiME4, AMI, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1906.11047
  bibtex_key: vonplaten2019multi
  confidence: high
---

# chime4-ami-asr-eval

> Multi-Span Acoustic Modelling using Raw Waveform Signals — Patrick von Platen, Chao Zhang, Philip Woodland (arXiv:1906.11047, 2019)

## What this evaluates

Evaluates automatic speech recognition systems on real-world meeting and close-talking microphone speech. It measures how well acoustic models trained on raw waveforms generalize to challenging multi-microphone environments compared to traditional feature-based baselines.

## Datasets

- **CHiME4** — total ?; splits: train (-1), dev (-1)
- **AMI** — total ?; splits: train (-1), dev (-1), eval (-1)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate: the minimum number of insertions, deletions, and substitutions of words required to transform the predicted transcript into the reference transcript, divided by the total number of words in the reference.

## Input / output format

**Input**: Raw waveform audio signals, shifted by 10ms (160 samples) per frame. Input is normalized to zero mean and unit variance (globally for CHiME4, per utterance and per meeting for AMI).

**Output**: Softmax probabilities over clustered triphone-states, decoded via Viterbi algorithm into word sequences using a language model.

## Scoring recipe

```python
def compute_wer(predictions, references):
    total_errors = 0
    total_words = 0
    for pred, ref in zip(predictions, references):
        edit_dist = levenshtein_distance(pred.split(), ref.split())
        total_errors += edit_dist
        total_words += len(ref.split())
    return (total_errors / total_words) * 100 if total_words > 0 else 0.0
```

## Common pitfalls

- Microphone failure detection algorithm must be applied to select the correct channel for the CHiME4 dev set.
- Normalization differs between datasets: CHiME4 uses global normalization, while AMI uses per-utterance zero mean and per-meeting unit variance.
- Alignment is done at 10ms intervals to clustered triphone-states, not directly to words.

## Evidence (verbatim from paper)

> Experimental results on CHiME4 and AMI datasets show a 5% relative WER reduction over FBANK-based models, with evidence that smaller kernels (50 params) outperform larger ones (400 params), and that learned filters diverge significantly from traditional log-Mel filters, indicating effective feature learning beyond handcrafted designs.

## Citation

```bibtex
@misc{vonplaten2019multi,
  title={Multi-Span Acoustic Modelling using Raw Waveform Signals},
  author={Patrick von Platen, Chao Zhang, Philip Woodland},
  year={2019},
  note={arXiv:1906.11047}
}
```

- arXiv: 1906.11047