Usc Asr Eval

---
name: usc-asr-eval
description: Evaluates automatic speech recognition (ASR) systems on Uzbek language audio by measuring character and word error rates against manually transcribed ground truth. It probes the model's ability to accurately transcribe low-resource speech data without relying on external linguistic resources or pronunciation dictionaries. Use when the user wants to benchmark on USC, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2107.14419
  bibtex_key: musaev2021usc
  confidence: high
---

# usc-asr-eval

> USC: An Open-Source Uzbek Speech Corpus and Initial Speech Recognition Experiments — Musaev et al. (2021) (arXiv:2107.14419, 2021)

## What this evaluates

Evaluates automatic speech recognition (ASR) systems on Uzbek language audio by measuring character and word error rates against manually transcribed ground truth. It probes the model's ability to accurately transcribe low-resource speech data without relying on external linguistic resources or pronunciation dictionaries.

## Datasets

- **USC** — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/IS2AI/Uzbek_ASR

## Metrics

- `WER` **(primary)** — range: percent
  - Standard edit distance metric: (Substitutions + Deletions + Insertions) / Total words in reference. Expressed as a percentage.
- `CER` — range: percent
  - Standard edit distance metric: (Substitutions + Deletions + Insertions) / Total characters in reference. Expressed as a percentage.

## Input / output format

**Input**: Audio recordings processed into acoustic features (MFCCs for DNN-HMM; 80-dim filterbank + pitch for E2E), paired with ground truth transcriptions.

**Output**: Predicted text sequence (character-level for E2E models, word-level for DNN-HMM).

## Scoring recipe

```python
def compute_wer_cer(predictions, references):
    total_errors = 0
    total_units = 0
    for pred, ref in zip(predictions, references):
        dist = levenshtein_distance(pred, ref)
        total_errors += dist
        total_units += len(ref)
    return (total_errors / total_units) * 100
```

## Common pitfalls

- Models are evaluated without external linguistic resources (lexicons, pronunciation models), which may inflate error rates compared to resource-rich language benchmarks.
- E2E models output character-level sequences, so WER requires post-hoc word segmentation or graphemic mapping, introducing tokenization ambiguities.
- Performance varies significantly based on language model integration and data augmentation (speed/spectral), making direct model comparisons sensitive to pipeline choices.

## Evidence (verbatim from paper)

> We conducted speech recognition experiments to demonstrate the reliability of the USC dataset. We built both DNN-HMM and E2E speech recognition models using our dataset (see Section[3]) and evaluated them using the character error rate (CER) and word error rate (WER) metrics. We did not use any external data and other available linguistic resources such as lexicon, pronunciation models, and vocabulary.

## Citation

```bibtex
@misc{musaev2021usc,
  title={USC: An Open-Source Uzbek Speech Corpus and Initial Speech Recognition Experiments},
  author={Musaev et al. (2021)},
  year={2021},
  note={arXiv:2107.14419}
}
```

- arXiv: 2107.14419