Hall E Tts Eval

---
name: hall-e-tts-eval
description: Evaluates zero-shot text-to-speech synthesis capability for generating minute-long audio from text and a short reference prompt. It measures linguistic accuracy, speaker similarity, audio quality, and temporal dynamics against ground truth speech. Use when the user wants to benchmark on MinutesSpeech, LibriSpeech, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.04380
  bibtex_key: nishimura2024halle
  confidence: high
---

# hall-e-tts-eval

> HALL-E: Hierarchical Neural Codec Language Model for Minute-Long Zero-Shot Text-to-Speech Synthesis — Yuto Nishimura et al. (arXiv:2410.04380, 2024)

## What this evaluates

Evaluates zero-shot text-to-speech synthesis capability for generating minute-long audio from text and a short reference prompt. It measures linguistic accuracy, speaker similarity, audio quality, and temporal dynamics against ground truth speech.

## Datasets

- **MinutesSpeech** — total ?; splits: test-90s (-1), test-180s (-1)
- **LibriSpeech** — total ?; splits: test.clean (-1)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate calculated using a conformer-transducer ASR model. Computed as the edit distance between the predicted and ground truth transcripts divided by the reference length.
- `PESQ` — range: [0, 4.5]
  - Perceptual Evaluation of Speech Quality score measuring audio fidelity between generated and reference speech.
- `SIM` — range: [0, 1]
  - Speaker Similarity score computed as cosine similarity between embeddings extracted by a WavLM-TDNN model from the generated and reference audio.
- `DNSMOS` — range: [1, 5]
  - Deep Noise Suppression Mean Opinion Score predicted by a model trained on ITU-T P.808 human ratings.
- `WD` — range: other
  - Wasserstein distance between the duration distributions of the generated speech and the ground truth speech.
- `QMOS` — range: [1, 5]
  - Subjective Mean Opinion Score for naturalness, rated by 3 native English speakers on a 1–5 scale.
- `SMOS` — range: [1, 5]
  - Subjective Mean Opinion Score for speaker similarity, rated by 3 native English speakers on a 1–5 scale.

## Input / output format

**Input**: Text transcript and a 3-second reference audio prompt.

**Output**: Synthesized audio waveform.

## Scoring recipe

```python
# ASR transcription
pred_text = conformer_transducer(generated_audio)
wer = edit_distance(pred_text, gold_text) / len(gold_text)

# Speaker Similarity
sim = cosine_similarity(wavlm_tdnn(generated_audio), wavlm_tdnn(reference_audio))

# DNSMOS
dnsmos = dns_mos_model(generated_audio)

# Duration Distribution Distance
wd = wasserstein_distance(duration(generated_audio), duration(gold_audio))

# Subjective Scores (QMOS/SMOS)
qmos = mean([rater.rate_naturalness(audio) for rater in raters])
smos = mean([rater.rate_similarity(audio) for rater in raters])
```

## Common pitfalls

- WER is computed using a conformer-transducer model, not standard Whisper-based ASR, which may yield different values than typical TTS benchmarks.
- Subjective metrics (QMOS/SMOS) rely on only 3 raters and 40 utterances per test set, leading to wide confidence intervals.
- Speaker similarity (SIM) degrades as frame rate is reduced to save computation, creating a trade-off between efficiency and acoustic fidelity.

## Evidence (verbatim from paper)

> Two evaluation metrics are used for speech reconstruction experiments: WER and PESQ. WER is calculated using the conformer-transducer. Six evaluation metrics are used for zero-shot TTS experiments: WER, speaker similarity (SIM), deep noise suppression mean opinion score (DNSMOS), Wasserstein distance (WD) with respect to the duration distribution, subjective evaluation of naturalness (QMOS), and subjective evaluation of speaker similarity (SMOS). SIM is calculated using WavLM-TDNN. DNSMOS is calculated using the model trained with ground truth human ratings obtained using ITU-T P.808. WD is calculated between the duration distributions of the generated speech and the ground truth speech. For QMOS and SMOS, 40 utterances were randomly selected from each test set, and three native English speakers rated their naturalness on a scale from 1 to 5.

## Citation

```bibtex
@misc{nishimura2024halle,
  title={HALL-E: Hierarchical Neural Codec Language Model for Minute-Long Zero-Shot Text-to-Speech Synthesis},
  author={Yuto Nishimura et al.},
  year={2024},
  note={arXiv:2410.04380}
}
```

- arXiv: 2410.04380