Gigaspeech2 Asr Eval

---
name: gigaspeech2-asr-eval
description: Evaluates automatic speech recognition (ASR) models on low-resource languages (Thai, Indonesian, Vietnamese) to measure transcription accuracy against reference texts. It probes the model's ability to handle domain-shifted audio and varying linguistic structures using character-level or word-level error metrics. Use when the user wants to benchmark on GigaSpeech 2, Common Voice 17.0, FLEURS, or asks about evaluating this task. Reports CER/WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.11546
  bibtex_key: yang2024gigaspeech2
  confidence: high
---

# gigaspeech2-asr-eval

> GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages with Automated Crawling, Transcription and Refinement — Yang et al. (2024) (arXiv:2406.11546, 2024)

## What this evaluates

Evaluates automatic speech recognition (ASR) models on low-resource languages (Thai, Indonesian, Vietnamese) to measure transcription accuracy against reference texts. It probes the model's ability to handle domain-shifted audio and varying linguistic structures using character-level or word-level error metrics.

## Datasets

- **GigaSpeech 2** — total ?; splits: DEV (-1), TEST (-1)
- **Common Voice 17.0** — total ?; splits: TEST (-1)
- **FLEURS** — total ?; splits: TEST (-1)

## Metrics

- `CER/WER` **(primary)** — range: percent
  - Standard Levenshtein edit distance normalized by the reference length. Character-level edits are counted for Thai (CER); word-level edits are counted for Indonesian and Vietnamese (WER).

## Input / output format

**Input**: Raw audio recordings of speech in Thai, Indonesian, or Vietnamese.

**Output**: Transcribed text string corresponding to the input audio.

## Scoring recipe

```python
def compute_cer_wer(predictions, references, word_level=False):
    total_errors = 0
    total_ref_len = 0
    for p, r in zip(predictions, references):
        tokens_p = p.split() if word_level else list(p)
        tokens_r = r.split() if word_level else list(r)
        total_errors += levenshtein_distance(tokens_p, tokens_r)
        total_ref_len += len(tokens_r)
    return (total_errors / total_ref_len) * 100
```

## Common pitfalls

- CER is used for Thai while WER is used for Indonesian and Vietnamese; mixing these up will invalidate cross-lingual comparisons.
- Performance drops on Common Voice and FLEURS are attributed to domain mismatch rather than model failure, so cross-dataset comparisons should account for audio distribution shifts.
- Comparing models with vastly different parameter counts (e.g., 151.9M vs 1.5B) without noting compute/size differences can be misleading.

## Evidence (verbatim from paper)

> The ASR performance is evaluated regarding character error rate (CER) or word error rate (WER) on three distinct test sets from GigaSpeech 2, Common Voice 17.0, and FLEURS.

## Citation

```bibtex
@misc{yang2024gigaspeech2,
  title={GigaSpeech 2: An Evolving, Large-Scale and Multi-domain ASR Corpus for Low-Resource Languages with Automated Crawling, Transcription and Refinement},
  author={Yang et al. (2024)},
  year={2024},
  note={arXiv:2406.11546}
}
```

- arXiv: 2406.11546