Hebrew Tts Eval

---
name: hebrew-tts-eval
description: Evaluates the quality of a diacritic-free Hebrew text-to-speech system by measuring content accuracy, speech naturalness, and speaker similarity against baseline models and different text tokenization strategies. Use when the user wants to benchmark on Hebrew TTS Test Set, or asks about evaluating this task. Reports Content Preservation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2407.12206
  bibtex_key: roth2024diacriticfree
  confidence: high
---

# hebrew-tts-eval

> A Language Modeling Approach to Diacritic-Free Hebrew TTS — Roth et al. (2024) (arXiv:2407.12206, 2024)

## What this evaluates

Evaluates the quality of a diacritic-free Hebrew text-to-speech system by measuring content accuracy, speech naturalness, and speaker similarity against baseline models and different text tokenization strategies.

## Datasets

- **Hebrew TTS Test Set** — total ?; splits: test (-1)

## Metrics

- `WER` — range: percent
  - Word Error Rate: the percentage of words incorrectly recognized in the generated speech compared to a reference transcription.
- `CER` — range: percent
  - Character Error Rate: the percentage of characters incorrectly recognized in the generated speech compared to a reference transcription.
- `Speaker Similarity` — range: [0, 1]
  - Automatic metric measuring the cosine similarity between speaker embeddings extracted from the reference and generated audio.
- `Content Preservation` **(primary)** — range: other
  - Human study metric where listeners rate how well the generated speech preserves the input text content on a Likert-like scale.
- `Naturalness` — range: other
  - Human study metric where listeners rate the perceptual naturalness of the generated speech.

## Input / output format

**Input**: Non-diacritic Hebrew text, processed via word-piece or character-level tokenization.

**Output**: Synthesized speech audio waveform.

## Scoring recipe

```python
# Automatic metrics
wer = compute_wer(reference_text, generated_audio)
cer = compute_cer(reference_text, generated_audio)
speaker_sim = cosine_similarity(ref_speaker_emb, gen_speaker_emb)

# Human study
content_scores = [human_rating(gen_audio) for gen_audio in generated_samples]
content_preservation = mean(content_scores)
naturalness = mean([human_rating_naturalness(gen_audio) for gen_audio in generated_samples])
```

## Common pitfalls

- Automatic WER and CER metrics are computed via transcriptions that ignore pronunciation differences, so homographs with different pronunciations are scored identically.
- Human content preservation scores show larger performance gaps between models than automatic WER/CER metrics, indicating automatic metrics may underestimate content errors.
- Speaker similarity and naturalness are subjective or embedding-based and may not correlate perfectly with perceived quality.

## Evidence (verbatim from paper)

> Interestingly, when considering WER, CER, and Speaker similarity, the Overflow method provides comparable performance to ours while being superior to the MMS model. The main difference between the methods is reflected in the naturalness of the generated speech. Moreover, it is worth mentioning that although the WER and CER are comparable across all methods (with MMS achieving worse WER and Overflow achieving worse CER), these are based on automatic transcriptions that do not take into account the pronunciation, meaning two different words can be transcribed to the same sequence characters while reflecting completely different pronunciation. However, when investigating the content metric under the human study we observe larger differences.

## Citation

```bibtex
@misc{roth2024diacriticfree,
  title={A Language Modeling Approach to Diacritic-Free Hebrew TTS},
  author={Roth et al. (2024)},
  year={2024},
  note={arXiv:2407.12206}
}
```

- arXiv: 2407.12206