Yodas Speech Recognition Eval

---
name: yodas-speech-recognition-eval
description: Evaluates monolingual automatic speech recognition performance across multiple languages using a large-scale, YouTube-collected audio dataset. It probes the model's ability to accurately transcribe spoken language from noisy, real-world video subtitles after alignment filtering. Use when the user wants to benchmark on YODAS, or asks about evaluating this task. Reports CER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.00899
  bibtex_key: li2024yodas
  confidence: high
---

# yodas-speech-recognition-eval

> YODAS: Youtube-Oriented Dataset for Audio and Speech — Li et al. (2024) (arXiv:2406.00899, 2024)

## What this evaluates

Evaluates monolingual automatic speech recognition performance across multiple languages using a large-scale, YouTube-collected audio dataset. It probes the model's ability to accurately transcribe spoken language from noisy, real-world video subtitles after alignment filtering.

## Datasets

- **YODAS** — total ?; splits: train (1000000), test (1000)

## Metrics

- `CER` **(primary)** — range: percent
  - Character Error Rate, calculated as the minimum number of character edits (insertions, deletions, substitutions) required to transform the predicted transcript into the reference transcript, divided by the number of characters in the reference transcript.

## Input / output format

**Input**: Raw audio waveform paired with a reference transcript (subtitle) for a specific language.

**Output**: Greedy decoded character sequence (transcript) from the model.

## Scoring recipe

```python
def compute_cer(predictions, references):
    total_errors = 0
    total_ref_chars = 0
    for pred, ref in zip(predictions, references):
        total_errors += edit_distance(pred, ref)
        total_ref_chars += len(ref)
    return (total_errors / total_ref_chars) * 100
```

## Common pitfalls

- Failing to apply the CTC alignment score threshold (>2.0) before training/evaluation, which includes heavily misaligned audio-text pairs and inflates error rates.
- Using automatic subtitles instead of manual ones without accounting for the significantly higher deletion error rate (26.6% vs 7.0%) in the automatic subset.
- Applying a uniform BPE vocabulary size across all languages, which fails to account for the higher character complexity of CJK scripts and artificially worsens CER.

## Evidence (verbatim from paper)

> Table.[5] displays the testing outcomes for the top-15 languages, measured by the Character Error Rate (CER). The respective CER for each language spans from 6 to 15. The best performance is recorded for Hungarian, with a CER of 6.2, while Japanese exhibits the least performance with a CER of 14.7.

## Citation

```bibtex
@misc{li2024yodas,
  title={YODAS: Youtube-Oriented Dataset for Audio and Speech},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2406.00899}
}
```

- arXiv: 2406.00899