Open Asr Leaderboard Eval

---
name: open-asr-leaderboard-eval
description: Evaluates multilingual speech recognition models on transcription accuracy and inference efficiency across short-form, long-form, and diverse language settings. It standardizes text normalization and reports both word error rate and inverse real-time factor to enable fair accuracy–efficiency comparisons. Use when the user wants to benchmark on Short-form English, Multilingual, Long-form, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.06961
  bibtex_key: srivastav2025openasr
  confidence: high
---

# open-asr-leaderboard-eval

> Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual Speech Recognition Evaluation — Srivastav et al. (2025) (arXiv:2510.06961, 2025)

## What this evaluates

Evaluates multilingual speech recognition models on transcription accuracy and inference efficiency across short-form, long-form, and diverse language settings. It standardizes text normalization and reports both word error rate and inverse real-time factor to enable fair accuracy–efficiency comparisons.

## Datasets

- **Short-form English** — total ?; splits: test (-1)
- **Multilingual** — total ?; splits: test (-1)
- **Long-form** — total ?; splits: test (-1)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate: (Substitutions + Deletions + Insertions) / Total Reference Words. Standard ASR metric measuring transcription accuracy.
- `RTFx` — range: other
  - Inverse Real-Time Factor: Audio Duration / Inference Time. Measures throughput efficiency (words processed per real-time second).

## Input / output format

**Input**: Raw audio files (short-form, long-form, or multilingual recordings).

**Output**: Transcribed text string.

## Scoring recipe

```python
def compute_wer(predictions, references):
    # Standard WER calculation over tokenized text
    return wer_score(predictions, references) * 100

def compute_rtf(audio_durations, inference_times):
    # RTFx = 1 / RT = audio_duration / inference_time
    return audio_durations / inference_times
```

## Common pitfalls

- Closed-source models cannot be fairly evaluated for RTFx due to upload latency and lack of GPU control.
- Improving English WER often degrades multilingual coverage due to training data specialization.
- SSL encoders paired with CTC decoders significantly underperform compared to Conformer+LLM architectures.

## Evidence (verbatim from paper)

> While the latter can achieve superior RTFx, this comes at the cost of accuracy: e.g., the best CTC-based model (NVIDIA Parakeet CTC 1.1B) ranks only 23rd in terms of WER.

## Citation

```bibtex
@misc{srivastav2025openasr,
  title={Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual Speech Recognition Evaluation},
  author={Srivastav et al. (2025)},
  year={2025},
  note={arXiv:2510.06961}
}
```

- arXiv: 2510.06961