Wenetspeech Eval

---
name: wenetspeech-eval
description: Evaluates Mandarin speech recognition systems across diverse, real-world domains (internet, meetings) and mixed Mandarin-English content. It benchmarks the robustness of ASR models against noisy, production-level audio and varying data scales. Use when the user wants to benchmark on WenetSpeech, or asks about evaluating this task. Reports MER%.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2110.03370
  bibtex_key: zhang2021wenetspeech
  confidence: high
---

# wenetspeech-eval

> WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognition — Zhang et al. (2021) (arXiv:2110.03370, 2021)

## What this evaluates

Evaluates Mandarin speech recognition systems across diverse, real-world domains (internet, meetings) and mixed Mandarin-English content. It benchmarks the robustness of ASR models against noisy, production-level audio and varying data scales.

## Datasets

- **WenetSpeech** — total 22400; splits: train_S (-1), train_M (-1), train_L (-1), dev (-1), test_net (-1), test_meeting (-1); repo https://github.com/wenet-e2e/WenetSpeech

## Metrics

- `MER%` **(primary)** — range: percent
  - Mixture Error Rate (MER) is calculated as the edit distance between predicted and ground truth tokens, where tokens are defined as Mandarin characters and English words, divided by the length of the ground truth sequence, expressed as a percentage.

## Input / output format

**Input**: Audio recordings (typically processed into FBank features, optionally with i-vectors) paired with ground truth transcriptions containing mixed Mandarin characters and English words.

**Output**: Predicted text transcription.

## Scoring recipe

```python
def compute_mer(predictions, references):
    # Tokenize both into Mandarin characters and English words
    pred_tokens = tokenize_mandarin_english(predictions)
    ref_tokens = tokenize_mandarin_english(references)
    # Compute Levenshtein edit distance
    edits = levenshtein_distance(pred_tokens, ref_tokens)
    # Calculate error rate as percentage
    mer = (edits / len(ref_tokens)) * 100
    return mer
```

## Common pitfalls

- MER treats Mandarin characters and English words as uniform tokens in edit distance, differing from standard Chinese Character Error Rate (CER) or Word Error Rate (WER).
- The paper explicitly states these are baseline results for toolkits (Kaldi, ESPnet, WeNet) and do not reflect state-of-the-art performance.
- Different toolkits use varying feature extraction parameters (e.g., FBank window/shift, i-vectors) and decoding strategies, so results are toolkit-specific rather than purely model-specific.

## Evidence (verbatim from paper)

> In Table 5, we report the experimental results in Mixture Error Rate (MER)[[40]], which considers Mandarin characters and English words as the tokens in the edit distance calculation, on three designed test sets and one well-known, publicly available test set (i.e. AIShell-1[[41]] test) with Kaldi, ESPNet and WeNet toolkits respectively.

## Citation

```bibtex
@misc{zhang2021wenetspeech,
  title={WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognition},
  author={Zhang et al. (2021)},
  year={2021},
  note={arXiv:2110.03370}
}
```

- arXiv: 2110.03370