Libritts R Eval

---
name: libritts-r-eval
description: This protocol evaluates the audio quality and naturalness of a restored multi-speaker TTS corpus (LibriTTS-R) compared to the original LibriTTS dataset. It measures both ground-truth speech fidelity and the downstream impact on multi-speaker TTS model generation quality using human subjective listening tests. Use when the user wants to benchmark on LibriTTS-R, or asks about evaluating this task. Reports MOS.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.18802
  bibtex_key: koizumi2023librittsr
  confidence: high
---

# libritts-r-eval

> LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus — Koizumi et al. (2023) (arXiv:2305.18802, 2023)

## What this evaluates

This protocol evaluates the audio quality and naturalness of a restored multi-speaker TTS corpus (LibriTTS-R) compared to the original LibriTTS dataset. It measures both ground-truth speech fidelity and the downstream impact on multi-speaker TTS model generation quality using human subjective listening tests.

## Datasets

- **LibriTTS-R** — total ?; splits: test-clean (-1), test-other (-1), train-clean-100 (-1), train-clean-360 (-1), train-other-500 (-1)

## Metrics

- `MOS` **(primary)** — range: [1, 5]
  - Mean Opinion Score rating speech naturalness on a 5-point scale (1: Bad, 2: Poor, 3: Fair, 4: Good, 5: Excellent) with 0.5 increments. Computed as the average of human listener ratings.
- `SxS` — range: [-3, 3]
  - Side-by-side preference test score on a 7-point scale (-3 to 3). Positive values indicate preference for LibriTTS-R over LibriTTS. Computed as the average of listener preference scores.
- `WER` — range: [0, 1]
  - Word Error Rate computed using a pre-trained Conformer XXL ASR model to verify text content preservation in restored speech.
- `Speaker Similarity` — range: [-1, 1]
  - Cosine similarity of speaker embeddings calculated between different utterances by the same speaker to verify speaker consistency.

## Input / output format

**Input**: Audio waveforms (ground-truth or TTS-generated) for subjective listening tests; text prompts and speaker IDs for TTS synthesis evaluation.

**Output**: Human listener ratings (MOS scores and SxS preference scores) or synthesized audio waveforms.

## Scoring recipe

```python
def compute_mos(ratings):
    return sum(ratings) / len(ratings)

def compute_sxs(scores):
    return sum(scores) / len(scores)

def compute_wer(gold_text, asr_output):
    return wer(gold_text, asr_output)

def compute_speaker_sim(emb1, emb2):
    return cosine_similarity(emb1, emb2)
```

## Common pitfalls

- The SxS scale is directional: positive scores favor LibriTTS-R, not the original LibriTTS.
- Ground-truth MOS differences between LibriTTS and LibriTTS-R are often statistically insignificant because both are human speech; significant differences appear mainly in SxS tests and TTS-generated samples.
- A small number of LibriTTS-R samples may still be distorted due to restoration failures, which can slightly lower aggregate scores if not filtered.

## Evidence (verbatim from paper)

> To evaluate subjective quality, we rated speech quality through mean-opinion-score (MOS) and side-by-side (SxS) preference tests. We asked to rate the naturalness in MOS test, and "which sound quality is better?" in SxS test. The scale of MOS was a 5-point scale (1: Bad, 2: Poor, 3: Fair, 4: Good, 5: Excellent) with rating increments of 0.5, and that of SxS was a 7-point scale (-3 to 3).

## Citation

```bibtex
@misc{koizumi2023librittsr,
  title={LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus},
  author={Koizumi et al. (2023)},
  year={2023},
  note={arXiv:2305.18802}
}
```

- arXiv: 2305.18802