Rats Asr Wer Eval

---
name: rats-asr-wer-eval
description: This evaluation probes the robustness of automatic speech recognition (ASR) systems when trained on extremely limited in-domain noisy data. It measures how well a model can generalize to real-world noisy conditions by leveraging synthetic noisy data generated via a GAN, compared to traditional data augmentation and fine-tuning baselines. Use when the user wants to benchmark on RATS (Channel A), or asks about evaluating this task. Reports WER (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2203.15321
  bibtex_key: chen2022noiserobust
  confidence: high
---

# rats-asr-wer-eval

> Noise-robust Speech Recognition with 10 Minutes Unparalleled In-domain Data — Chen Chen et al. (2022) (arXiv:2203.15321, 2022)

## What this evaluates

This evaluation probes the robustness of automatic speech recognition (ASR) systems when trained on extremely limited in-domain noisy data. It measures how well a model can generalize to real-world noisy conditions by leveraging synthetic noisy data generated via a GAN, compared to traditional data augmentation and fine-tuning baselines.

## Datasets

- **RATS (Channel A)** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `WER (%)` **(primary)** — range: percent
  - Word Error Rate calculated as (Substitutions + Deletions + Insertions) / Total Reference Words, expressed as a percentage. Lower values indicate better transcription accuracy.

## Input / output format

**Input**: 80-dimensional log-mel spectrogram features extracted from audio utterances.

**Output**: Sequence of byte-pair-encoding (BPE) tokens from a vocabulary of size 994.

## Scoring recipe

```python
def calculate_wer(predictions, references):
    total_errors = 0
    total_words = 0
    for pred, ref in zip(predictions, references):
        # Align using Levenshtein distance
        dist = levenshtein_distance(pred, ref)
        total_errors += dist
        total_words += len(ref)
    wer = (total_errors / total_words) * 100 if total_words > 0 else 0.0
    return wer
```

## Common pitfalls

- Failing to apply speed perturbation (×0.9, ×1.0, ×1.1) during training, which the paper notes significantly lowers WER across all baselines.
- Misinterpreting data requirements: the proposed method uses 10 minutes of unlabeled noisy data for GAN training, whereas baselines like Finetune use labeled data and Mixup uses unlabeled noisy data from the full 44.3-hour set.
- Comparing single-path and dual-path ASR results without noting that the dual-path system inherently yields lower WER due to its architecture, not just data generation.

## Evidence (verbatim from paper)

> Table [3](#S4.T3 "Table 3 ‣ 4.4.3 Benchmark against other competitive methods ‣ 4.4 Results ‣ 4 Experiments and results ‣ Noise-robust Speech Recognition with 10 Minutes Unparalleled In-domain Data") summarizes the comparison between the proposed Simu-GAN and other competitive techniques in terms of the WER (%). We observe that the proposed Simu-GAN obtained the best performance. Comparing with the “Mixup” methods, the proposed Simu-GAN achieves the 7.3% absolute WER improvements.

## Citation

```bibtex
@misc{chen2022noiserobust,
  title={Noise-robust Speech Recognition with 10 Minutes Unparalleled In-domain Data},
  author={Chen Chen et al. (2022)},
  year={2022},
  note={arXiv:2203.15321}
}
```

- arXiv: 2203.15321