Paddlespeech Eval

---
name: paddlespeech-eval
description: Evaluates a speech processing toolkit across five core tasks: environmental sound classification, automatic speech recognition, punctuation restoration, speech translation, and text-to-speech synthesis. It probes the model's ability to handle diverse audio and text inputs, perform sequence labeling, and generate high-quality synthetic speech. Use when the user wants to benchmark on ESC-50, Librispeech, Aishell-1, IWSLT2012-zh, MuST-C, CSMSC, or asks about evaluating this task. Reports 5-fold cross validation accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2205.12007
  bibtex_key: zhang2022paddlespeech
  confidence: high
---

# paddlespeech-eval

> PaddleSpeech: An Easy-to-Use All-in-One Speech Toolkit — Zhang et al. (2022) (arXiv:2205.12007, 2022)

## What this evaluates

Evaluates a speech processing toolkit across five core tasks: environmental sound classification, automatic speech recognition, punctuation restoration, speech translation, and text-to-speech synthesis. It probes the model's ability to handle diverse audio and text inputs, perform sequence labeling, and generate high-quality synthetic speech.

## Datasets

- **ESC-50** — total 2000; splits: 5-fold cross-validation (-1)
- **Librispeech** — total ?; splits: train (-1), val (-1), test-clean (-1), test-other (-1); repo http://www.openslr.org/12/
- **Aishell-1** — total ?; splits: train (340), val (40), test (20); repo http://www.aishelltech.com/kysjcp
- **IWSLT2012-zh** — total 150000; splits: train (147000), val (2000), test (1000); repo https://hltc.cs.ust.hk/iwslt/
- **MuST-C** — total ?; splits: tst-COMMON (-1)
- **CSMSC** — total ?; splits: train (-1); repo https://www.data-baker.com/open_source.html

## Metrics

- `5-fold cross validation accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly classified audio samples out of the total, averaged over 5 random data splits.
- `WER` — range: [0, 1]
  - Word Error Rate: normalized edit distance between predicted and reference word sequences.
- `CER` — range: [0, 1]
  - Character Error Rate: normalized edit distance between predicted and reference character sequences.
- `F1-score` — range: [0, 1]
  - Harmonic mean of precision and recall for sequence labeling tasks (comma, period, question mark, empty).
- `BLEU` — range: [0, 100]
  - Detokenized case-sensitive BLEU score using SacreBLEU, measuring n-gram overlap between translated and reference text.
- `MOS` — range: [1, 5]
  - Mean Opinion Score: average rating from human listeners on a 1-5 scale for naturalness, collected via crowdMOS toolkit.

## Input / output format

**Input**: Audio waveform files (resampled to 32kHz, monophonic) or text sequences, depending on the specific task.

**Output**: Predicted class labels, transcribed text, restored punctuation tags, translated text, or synthesized audio waveforms.

## Scoring recipe

```python
def compute_metrics(predictions, gold, task):
    if task == 'sound_classification':
        return sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    elif task == 'asr':
        return edit_distance(predictions, gold) / len(gold)  # WER/CER
    elif task == 'punctuation':
        return f1_score(predictions, gold)
    elif task == 'translation':
        return sacrebleu.compute(predictions, [gold])
    elif task == 'tts':
        return human_mos_ratings(predictions)  # 1-5 scale
    return metric_value
```

## Common pitfalls

- Librispeech results are reported separately for test-clean and test-other splits; mixing them inflates performance.
- TTS evaluation relies on subjective human MOS scores rather than objective acoustic metrics, making cross-study comparison difficult.
- Punctuation restoration simplifies the target set to only comma, period, and question mark, ignoring other punctuation types.

## Evidence (verbatim from paper)

> We report word error rate (WER) and character error rate (CER) for Librispeech (English) and Aishell (Mandarin) speech recognition, respectively.

## Citation

```bibtex
@misc{zhang2022paddlespeech,
  title={PaddleSpeech: An Easy-to-Use All-in-One Speech Toolkit},
  author={Zhang et al. (2022)},
  year={2022},
  note={arXiv:2205.12007}
}
```

- arXiv: 2205.12007