Cml Tts Eval

---
name: cml-tts-eval
description: Evaluates the quality of text-to-speech models trained on the CML-TTS dataset across seven low-resource languages. It probes speaker similarity preservation and text fidelity in synthesized audio under both seen and unseen speaker (zero-shot) conditions. Use when the user wants to benchmark on CML-TTS, or asks about evaluating this task. Reports SECS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.10097
  bibtex_key: oliveira2023cmltts
  confidence: high
---

# cml-tts-eval

> CML-TTS A Multilingual Dataset for Speech Synthesis in Low-Resource Languages — Oliveira et al. (2023) (arXiv:2306.10097, 2023)

## What this evaluates

Evaluates the quality of text-to-speech models trained on the CML-TTS dataset across seven low-resource languages. It probes speaker similarity preservation and text fidelity in synthesized audio under both seen and unseen speaker (zero-shot) conditions.

## Datasets

- **CML-TTS** — total ?; splits: train (-1), dev (-1), test (-1)

## Metrics

- `SECS` **(primary)** — range: [-1, 1]
  - Speaker Encoder Cosine Similarity. Computes cosine similarity between speaker embeddings extracted from generated audio and ground truth audio using the Resemblyzer model. Values closer to 1 indicate higher speaker similarity, while values closer to -1 indicate low similarity.
- `WER` — range: [0, 1]
  - Word Error Rate. Calculated by transcribing synthesized audio with Wav2Vec 2.0 XLSR Large and computing the Levenshtein distance at the word level against the ground truth text.
- `CER` — range: [0, 1]
  - Character Error Rate. Calculated as CER = (S + D + I) / N, where S, D, and I are the number of substitutions, deletions, and insertions, and N is the total number of characters in the ground truth text.

## Input / output format

**Input**: 1,000 text sentences per language and speaker reference audio/embeddings (randomly extracted from train, dev, or test sets).

**Output**: Synthesized audio files for each sentence.

## Scoring recipe

```python
def compute_metrics(predictions_audio, gold_text, gold_audio):
    # SECS
    pred_emb = resemblyzer.encode(predictions_audio)
    gold_emb = resemblyzer.encode(gold_audio)
    secs = cosine_similarity(pred_emb, gold_emb)
    
    # WER & CER
    pred_text = wav2vec2_transcribe(predictions_audio)
    wer = levenshtein_distance_words(gold_text, pred_text) / len(gold_text.split())
    cer = levenshtein_distance_chars(gold_text, pred_text) / len(gold_text)
    
    return secs, wer, cer
```

## Common pitfalls

- Portuguese WER/CER results are artificially inflated due to historical orthographic changes in the LibriVox source books, not model deficiency.
- No language model is applied during ASR transcription, which can skew error rates for languages with complex spelling or morphology.
- SECS values are tightly coupled to the specific speaker encoder architecture (Resemblyzer/VoxCeleb); results are not directly comparable across different encoder implementations.

## Evidence (verbatim from paper)

> To evaluate the similarity between the synthesized speech and the ground truth, we calculate the Speaker Encoder Cosine Similarity (SECS) between the embeddings extracted from the generated audios and from the ground truth audios. In SECS, the closer to 1, the greater the similarity, while the closer to -1 indicates low speaker similarity. Using the transcripts and the ground truth text, we calculated the Word Error Rate (WER) and Character Error Rate (CER) metrics. CER metric is calculated according to the equation $CER\=\frac{S+D+I}{N}$ where $S$ is the number of substitutions, $D$ deletions, $I$ insertions, and $N$ is the total characters of the ground truth text.

## Citation

```bibtex
@misc{oliveira2023cmltts,
  title={CML-TTS A Multilingual Dataset for Speech Synthesis in Low-Resource Languages},
  author={Oliveira et al. (2023)},
  year={2023},
  note={arXiv:2306.10097}
}
```

- arXiv: 2306.10097