Speech Rep Eval

---
name: speech-rep-eval
description: Evaluates the quality of compressed semantic speech representations across automatic speech recognition, speech-to-text translation, and voice conversion tasks. It probes how well adaptive entropy-based token aggregation preserves linguistic and acoustic information under varying compression ratios. Use when the user wants to benchmark on LibriSpeech, CVSS-C, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.00503
  bibtex_key: zuo2025entropy
  confidence: high
---

# speech-rep-eval

> Entropy-based Coarse and Compressed Semantic Speech Representation Learning — Zuo et al. (2025) (arXiv:2509.00503, 2025)

## What this evaluates

Evaluates the quality of compressed semantic speech representations across automatic speech recognition, speech-to-text translation, and voice conversion tasks. It probes how well adaptive entropy-based token aggregation preserves linguistic and acoustic information under varying compression ratios.

## Datasets

- **LibriSpeech** — total ?; splits: test-clean (-1)
- **CVSS-C** — total ?; splits: test (-1)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate: normalized edit distance between predicted and reference word sequences, expressed as a percentage.
- `CER` — range: percent
  - Character Error Rate: normalized edit distance between predicted and reference character sequences, expressed as a percentage.
- `BLEU` — range: [0, 100]
  - Bilingual Evaluation Understudy: geometric mean of modified n-gram precisions with a brevity penalty to penalize overly short translations.
- `Q-MOS` — range: [1, 5]
  - Quality Mean Opinion Score: subjective rating of naturalness and clarity on a 1-5 scale collected via human annotators.
- `S-MOS` — range: [1, 5]
  - Speaker Similarity Mean Opinion Score: subjective rating of speaker identity preservation on a 1-5 scale collected via human annotators.
- `UTMOS` — range: other
  - Automatic Mean Opinion Score prediction using a pretrained UTMOS model to estimate speech quality without human annotation.

## Input / output format

**Input**: Raw speech audio or extracted HuBERT semantic units (k=500 clusters), optionally compressed via entropy-based token aggregation at specified Hz rates.

**Output**: For ASR/ST: autoregressively generated text tokens. For VC: synthesized mel-spectrogram.

## Scoring recipe

```python
def compute_metrics(predictions, references, audio_inputs=None):
    wer = 100 * edit_distance(predictions.words, references.words) / max(len(references.words), 1)
    cer = 100 * edit_distance(predictions.chars, references.chars) / max(len(references.chars), 1)
    bleu = compute_bleu(references.text, predictions.text)
    q_mos, s_mos = human_mos_ratings(predictions.audio) if audio_inputs else None
    utmos = utmos_model.predict(predictions.audio)
    latency = avg_decode_time(audio_inputs, predictions)
    return {'WER': wer, 'CER': cer, 'BLEU': bleu, 'Q-MOS': q_mos, 'S-MOS': s_mos, 'UTMOS': utmos, 'Latency': latency}
```

## Common pitfalls

- Inference latency is hardware-dependent (measured on a single NVIDIA V100), so cross-device comparisons require normalization.
- VC quality relies on both subjective human ratings (Q-MOS/S-MOS) and automatic UTMOS; omitting either gives an incomplete quality assessment.
- Token rate thresholds (7, 15, 24 Hz) directly impact compression; results must be reported alongside the exact entropy threshold used.

## Evidence (verbatim from paper)

> We evaluate ASR and voice conversion using both objective and subjective metrics. ASR performance is assessed by Word Error Rate (WER) and Character Error Rate (CER), while voice conversion quality is evaluated via Q-MOS (naturalness and clarity) and S-MOS (speaker similarity) through human ratings on Amazon Mechanical Turk, complemented by automatic predictions using UTMOS. For speech-to-text translation, we report BLEU scores computed between the generated and reference text.

## Citation

```bibtex
@misc{zuo2025entropy,
  title={Entropy-based Coarse and Compressed Semantic Speech Representation Learning},
  author={Zuo et al. (2025)},
  year={2025},
  note={arXiv:2509.00503}
}
```

- arXiv: 2509.00503