Llama Vits Tts Eval

---
name: llama-vits-tts-eval
description: Evaluates the naturalness, intelligibility, and emotional expressiveness of a non-autoregressive TTS model enhanced with LLM-derived semantic embeddings. Probes how well semantic tokens from Llama2 versus BERT improve acoustic quality and emotion similarity compared to baselines. Use when the user wants to benchmark on LJSpeech, 1-hour LJSpeech, EmoV_DB_bea_sem, or asks about evaluating this task. Reports ESMOS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.06714
  bibtex_key: [REDACTED]
  confidence: high
---

# llama-vits-tts-eval

> Llama-VITS: Enhancing TTS Synthesis with Semantic Awareness — Feng et al. (2024) (arXiv:2404.06714, 2024)

## What this evaluates

Evaluates the naturalness, intelligibility, and emotional expressiveness of a non-autoregressive TTS model enhanced with LLM-derived semantic embeddings. Probes how well semantic tokens from Llama2 versus BERT improve acoustic quality and emotion similarity compared to baselines.

## Datasets

- **LJSpeech** — total ?; splits: train (-1), test (-1)
- **1-hour LJSpeech** — total ?; splits: train (-1), test (-1)
- **EmoV_DB_bea_sem** — total ?; splits: train (-1), test (51)

## Metrics

- `ESMOS` **(primary)** — range: [1, 5]
  - Emotion Similarity Mean Opinion Score. Participants rate emotion similarity to ground truth on a 5-point scale: 1 (Bad Match) to 5 (Excellent Match).
- `UTMOS` — range: [1, 5]
  - UTokyo-SaruLab Mean Opinion Score. A neural MOS prediction network trained on Blizzard and Voice Conversion Challenge data.
- `MCD` — range: dB
  - Mel-Cepstral Distortion. Measures spectral distortion between synthesized and reference speech in decibels.
- `CER` — range: percent
  - Character Error Rate. Computed using Whisper-large to transcribe synthesized audio and compare against ground truth text.
- `WER` — range: percent
  - Word Error Rate. Computed using Whisper-large to transcribe synthesized audio and compare against ground truth text.

## Input / output format

**Input**: Text transcript or phoneme sequence, optionally augmented with global or sequential semantic embeddings extracted from Llama2 or BERT models.

**Output**: Synthesized speech waveform.

## Scoring recipe

```python
# ESMOS (Subjective)
esmos_scores = []
for sample in test_samples:
    score = mturk_participant_rating(sample.synthesized_audio, sample.gt_audio, scale=1-5)
    esmos_scores.append(score)
esmos = mean(esmos_scores)

# UTMOS (Objective)
utmos_scores = [speechmos_model.predict(audio) for audio in test_audios]
utmos = mean(utmos_scores)

# MCD (Objective)
mcd_values = [mel_cepstral_distortion(ref, synth) for ref, synth in zip(ref_audios, synth_audios)]
mcd = mean(mcd_values)

# CER/WER (Objective via Whisper-large)
transcriptions = whisper_large.transcribe(test_audios)
cer = character_error_rate(transcriptions, ground_truth_texts)
wer = word_error_rate(transcriptions, ground_truth_texts)
```

## Common pitfalls

- ESMOS instructions explicitly tell participants to ignore voice quality, style, and audio quality, focusing solely on emotiveness; failing to enforce this skews results.
- Training protocols differ by dataset size: LJSpeech variants train from scratch for 100k steps, while EmoV_DB_bea_sem fine-tunes a 100k-step LJSpeech checkpoint for 150k steps, making step-count comparisons invalid across datasets.
- Objective metrics rely on specific external toolchains (SpeechMOS for UTMOS, ESPnet for MCD, Whisper-large for CER/WER) rather than native model outputs, requiring strict version control for reproducibility.

## Evidence (verbatim from paper)

> In subjective evaluation, we conduct Emotion Similarity Mean Opinion Score (ESMOS) experiments to evaluate emotion similarity for EmoV_DB_bea_sem. ... In objective evaluation, we utilize UTokyo-SaruLab Mean Opinion Score (UTMOS), Mel-Cepstral Distortion (MCD), and speech recognition performance measured by Character Error Rate (CER) and Word Error Rate (WER).

## Citation

```bibtex
@misc{feng2024llamavits,
  title={Llama-VITS: Enhancing TTS Synthesis with Semantic Awareness},
  author={Feng et al. (2024)},
  year={2024},
  note={arXiv:2404.06714}
}
```

- arXiv: 2404.06714