Speakstream Streaming Tts Eval

---
name: speakstream-streaming-tts-eval
description: Evaluates the quality and latency of streaming text-to-speech models that generate audio incrementally from interleaved text and speech inputs. It probes the model's ability to maintain speech accuracy and naturalness while minimizing first-token latency under streaming constraints. Use when the user wants to benchmark on LJSpeech, LibriSpeech, or asks about evaluating this task. Reports Word Error Rate (WER).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.19206
  bibtex_key: bai2025speakstream
  confidence: high
---

# speakstream-streaming-tts-eval

> SpeakStream: Streaming Text-to-Speech with Interleaved Data — Richard He Bai et al. (2025) (arXiv:2505.19206, 2025)

## What this evaluates

Evaluates the quality and latency of streaming text-to-speech models that generate audio incrementally from interleaved text and speech inputs. It probes the model's ability to maintain speech accuracy and naturalness while minimizing first-token latency under streaming constraints.

## Datasets

- **LJSpeech** — total ?; splits: dev (-1)
- **LibriSpeech** — total ?; splits: dev-clean (-1)

## Metrics

- `Word Error Rate (WER)` **(primary)** — range: percent
  - Calculated by transcribing the generated speech using WhisperX (base.en) and comparing the output text to the ground truth transcript. Standard word-level error rate counting substitutions, insertions, and deletions.
- `TTS Latency` — range: ms
  - Time elapsed between the TTS model receiving its first word and generating its first frame (or first actual phoneme). Measured in milliseconds.
- `Human Naturalness/Coherence` — range: [1, 5]
  - Average rating on a 1 to 5 scale provided by human annotators for each segment's naturalness and coherence.

## Input / output format

**Input**: Streaming text tokens (words) provided sequentially, optionally interleaved with previously generated speech features or audio segments.

**Output**: Streaming Mel spectrogram features (dMel) or 24kHz audio waveform chunks.

## Scoring recipe

```python
def evaluate_wer(generated_audio, ground_truth_text):
    transcribed = whisperx_base_en.transcribe(generated_audio)
    return calculate_wer(transcribed, ground_truth_text)

def evaluate_latency(model, first_word):
    t_start = time.time()
    model.process(first_word)
    _ = model.get_first_audio_chunk()
    t_end = time.time()
    return (t_end - t_start) * 1000
```

## Common pitfalls

- Applying non-streaming TTS models directly to streaming segments causes severe hallucination and WER > 200%.
- Measuring latency from the first audio byte rather than the first actual phoneme overestimates real-world response time.
- Assuming larger context windows (m) always improve accuracy; performance drops beyond m=5 due to text repetition artifacts.

## Evidence (verbatim from paper)

> For TTS evaluation, we utilize WhisperX (“base.en”) to transcribe our generated speech into text and calculate the Word Error Rate (WER). ... We report three latency metrics: 1. Total latency: Time elapsed between the TTS model receiving its first word and the audio player outputting the first waveform chunk. 2. Vocoder latency: Time elapsed between the vocoder receiving its first frame input and generating the first chunk of waveform. 3. TTS latency: Time elapsed between the TTS model receiving its first word and generating its first frame.

## Citation

```bibtex
@misc{bai2025speakstream,
  title={SpeakStream: Streaming Text-to-Speech with Interleaved Data},
  author={Richard He Bai et al. (2025)},
  year={2025},
  note={arXiv:2505.19206}
}
```

- arXiv: 2505.19206