Tts Nl Guidance Eval

---
name: tts-nl-guidance-eval
description: Evaluates a text-to-speech model's ability to generate audio that matches natural language descriptions of speaker attributes (gender, accent, pitch, speaking rate, recording quality) and overall audio fidelity. It measures both objective acoustic metrics and subjective human ratings of relevance and naturalness. Use when the user wants to benchmark on MLS, LibriTTS-R, or asks about evaluating this task. Reports MOS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.01912
  bibtex_key: lyth2024naturallanguage
  confidence: high
---

# tts-nl-guidance-eval

> Natural language guidance of high-fidelity text-to-speech with synthetic annotations — Lyth et al. (2024) (arXiv:2402.01912, 2024)

## What this evaluates

Evaluates a text-to-speech model's ability to generate audio that matches natural language descriptions of speaker attributes (gender, accent, pitch, speaking rate, recording quality) and overall audio fidelity. It measures both objective acoustic metrics and subjective human ratings of relevance and naturalness.

## Datasets

- **MLS** — total ?; splits: test (-1)
- **LibriTTS-R** — total ?; splits: test (-1)

## Metrics

- `PESQ` — range: [0, 4.5]
  - Wideband Perceptual Estimation of Speech Quality. Reference-less estimate computed via the Torchaudio SQUIM model.
- `STOI` — range: [0, 1]
  - Short-Time Objective Intelligibility. Reference-less estimate computed via the Torchaudio SQUIM model.
- `SI-SDR` — range: dB
  - Scale-Invariant Signal-to-Distortion Ratio. Reference-less estimate computed via the Torchaudio SQUIM model.
- `MOS` **(primary)** — range: [1, 5]
  - Mean Opinion Score. Human listeners rate overall naturalness and audio fidelity on a 1-5 scale.
- `REL` — range: [1, 5]
  - Relevance score. Human listeners rate how closely the generated speech matches the natural language description on a 1-5 scale.

## Input / output format

**Input**: Text to synthesize paired with a natural language description specifying target speaker attributes (e.g., gender, accent, pitch, speaking rate, recording conditions).

**Output**: Synthesized audio waveform.

## Scoring recipe

```python
def evaluate(generated_audio, description):
    # Objective SQUIM metrics
    pesq, stoi, si_sdr = torchaudio_squim(generated_audio)
    # Subjective ratings (averaged over 30 listeners)
    mos = mean([listener.rate_naturalness(generated_audio) for _ in range(30)])
    rel = mean([listener.rate_relevance(generated_audio, description) for _ in range(30)])
    return {'PESQ': pesq, 'STOI': stoi, 'SI-SDR': si_sdr, 'MOS': mos, 'REL': rel}
```

## Common pitfalls

- Label noise in test descriptions can cause generated speech to score higher than ground truth on relevance.
- Minor audio artifacts in LibriTTS-R ground truth may make the model's output appear more faithful to the description.
- The model struggles to accurately generate specific acoustic attributes like C50 (reverberation), leading to poor correlation.

## Evidence (verbatim from paper)

> Our final objective evaluation aims to quantify the audio fidelity of our model when asked to produce audio with "excellent recording quality" or similar terms. Here, we use the recently proposed Torchaudio Speech Quality and Intelligibility Measures [34]. This model provides a reference-less estimate of Wideband Perceptual Estimation of Speech Quality (PESQ), Short-Time Objective Intelligibility (STOI), and Scale-Invariant Signal-to-Distortion Ratio (SD-SRD).

## Citation

```bibtex
@misc{lyth2024naturallanguage,
  title={Natural language guidance of high-fidelity text-to-speech with synthetic annotations},
  author={Lyth et al. (2024)},
  year={2024},
  note={arXiv:2402.01912}
}
```

- arXiv: 2402.01912