Libriquote Eval

---
name: libriquote-eval
description: Probes the ability of zero-shot text-to-speech systems to generate expressive, character-specific utterances while preserving reference speaker timbre. It evaluates cross-sentence generation where a narration clip guides the synthesis of a fictional quotation, testing prosodic variability, emotional expressiveness, and speech intelligibility. Use when the user wants to benchmark on LibriQuote, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.04072
  bibtex_key: michel2025libriquote
  confidence: high
---

# libriquote-eval

> LibriQuote: A Speech Dataset of Fictional Character Utterances for Expressive Zero-Shot Speech Synthesis — Michel et al. (2025) (arXiv:2509.04072, 2025)

## What this evaluates

Probes the ability of zero-shot text-to-speech systems to generate expressive, character-specific utterances while preserving reference speaker timbre. It evaluates cross-sentence generation where a narration clip guides the synthesis of a fictional quotation, testing prosodic variability, emotional expressiveness, and speech intelligibility.

## Datasets

- **LibriQuote** — total ?; splits: train (-1), test (-1); repo https://github.com/deezer/libriquote

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate computed using Whisper-large-v3 transcription against the target text. Lower is better.
- `SIM-O` — range: [-1, 1]
  - Cosine similarity between speaker embeddings extracted from a WavLM-large verification model for synthesized and ground-truth audio. Higher is better.
- `MCD` — range: other
  - Mel Cepstral Distortion measuring spectral envelope difference between synthesized and ground-truth speech. Lower is better.
- `FPC` — range: [-1, 1]
  - Pearson correlation coefficient between fundamental frequency (F0) contours of synthesized and ground-truth speech. Higher is better.
- `MOS` — range: [1, 5]
  - Mean Opinion Score rated by human listeners on a scale from 1 to 5 (0.5 intervals) to assess naturalness.
- `CMOS` — range: [-3, 3]
  - Comparative MOS rated from -3 to 3, measuring the degree of expressivity of synthesized speech relative to ground-truth.

## Input / output format

**Input**: Reference narration audio clip (2–15 seconds), target quotation text, and optionally left/right context text tokens.

**Output**: Synthesized audio waveform matching the target quotation text.

## Scoring recipe

```python
# Objective metrics
wer = whisper_transcribe(prediction_audio) / len(gold_text_words)
sim_o = cosine_similarity(wavlm_embed(prediction_audio), wavlm_embed(gold_audio))
mcd = compute_mel_cepstral_distortion(prediction_audio, gold_audio)
fpc = pearson_correlation(f0_contour(prediction_audio), f0_contour(gold_audio))

# Subjective metrics (human evaluation)
mos = mean([rater.score(prediction_audio, scale=[1,5]) for rater in raters])
cmos = mean([rater.score(prediction_audio, reference=gold_audio, scale=[-3,3]) for rater in raters])
```

## Common pitfalls

- High WER on ground-truth quotations is expected due to high expressiveness and fictional delivery, not a data quality issue.
- Speaker similarity (SIM-O) is computed against the original ground-truth speaker, not the reference narration speaker used for zero-shot prompting, which can cause confusion.
- Subjective evaluation relies on a small sample size (30 utterances, 2 per speaker) with only 5 raters, limiting statistical power for CMOS/MOS.

## Evidence (verbatim from paper)

> For objective metrics, we report Word Error Rate (WER) computed with Whisper-large-v3 (Radford et al. [2023]) to measure speech intelligibility and speaker similarity between the synthesized speech and the original ground-truth speech (SIM-O). We employ a WavLM-large based speaker verification model (Chen et al. [2022]) to extract speaker embeddings and calculate cosine similarities. We evaluate prosody similarity with the ground truth quotation using Mel Cepstral Distortion (MCD) and F0 Pearson correlation (FPC) following (Huang et al. [2023]).

## Citation

```bibtex
@misc{michel2025libriquote,
  title={LibriQuote: A Speech Dataset of Fictional Character Utterances for Expressive Zero-Shot Speech Synthesis},
  author={Michel et al. (2025)},
  year={2025},
  note={arXiv:2509.04072}
}
```

- arXiv: 2509.04072