Grad Tts Eval

---
name: grad-tts-eval
description: Evaluates text-to-speech synthesis quality, inference efficiency, and probabilistic modeling accuracy of a diffusion-based model. It probes the trade-off between synthesis fidelity and computational cost by varying reverse diffusion steps, and measures human-perceived audio quality against strong baselines. Use when the user wants to benchmark on LJSpeech, or asks about evaluating this task. Reports MOS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2105.06337
  bibtex_key: [REDACTED]
  confidence: high
---

# grad-tts-eval

> Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech — Popov et al. (2021) (arXiv:2105.06337, 2021)

## What this evaluates

Evaluates text-to-speech synthesis quality, inference efficiency, and probabilistic modeling accuracy of a diffusion-based model. It probes the trade-off between synthesis fidelity and computational cost by varying reverse diffusion steps, and measures human-perceived audio quality against strong baselines.

## Datasets

- **LJSpeech** — total ?; splits: train (-1), test (500)

## Metrics

- `MOS` **(primary)** — range: [1, 5]
  - Average of 10 human ratings per audio sample on a 9-point Likert scale (1.0 to 5.0 in 0.5 increments).
- `Log-likelihood` — range: other
  - Average data log-likelihood estimated over 50 test sentences using the instantaneous change of variables formula and Hutchinson’s trace estimator.
- `RTF` — range: other
  - Real-Time Factor: ratio of total inference time to total generated audio duration.

## Input / output format

**Input**: Phonemized text sequence

**Output**: Synthesized audio waveform

## Scoring recipe

```python
def compute_mos(predictions, gold):
    ratings = [human_listen_test(audio) for audio in predictions]
    return mean(ratings)

def compute_rtf(predictions, gold):
    total_time = sum(inference_time(audio) for audio in predictions)
    total_dur = sum(audio_duration(audio) for audio in predictions)
    return total_time / total_dur

def compute_log_likelihood(predictions, gold):
    lls = [compute_log_likelihood_score(text) for text in gold[:50]]
    return mean(lls)
```

## Common pitfalls

- Diffusion loss convergence does not guarantee high synthesis quality; inaccurate gradient predictions for a small subset of time steps can severely degrade output despite low loss.
- Baseline comparisons may be skewed if unofficial implementations are used (e.g., FastSpeech in this work).
- Inference speed vs. quality trade-off is highly sensitive to the number of reverse diffusion steps; N=10 achieves near-optimal MOS, making higher N values computationally wasteful.

## Evidence (verbatim from paper)

> For Mean Opinion Score (MOS) estimation we synthesized 40 sentences from the test set with each model. The assessors were asked to estimate the quality of synthesized speech on a nine-point Likert scale, the lowest and the highest scores being 1 point (“Bad”) and 5 points (“Excellent”) with a step of 0.5 point. To ensure the reliability of the obtained results, only Master assessors were assigned to complete the listening test. Each audio was evaluated by 10 assessors.

## Citation

```bibtex
@misc{popov2021gradtts,
  title={Grad-TTS: A Diffusion Probabilistic Model for Text-to-Speech},
  author={Popov et al. (2021)},
  year={2021},
  note={arXiv:2105.06337}
}
```

- arXiv: 2105.06337