Voxstream Tts Eval

---
name: voxstream-tts-eval
description: Evaluates a streaming text-to-speech model's ability to generate high-quality, speaker-similar, and intelligible audio from text prompts in both non-streaming and full-stream (word-by-word) settings. It probes zero-shot voice cloning, cross-sentence continuity, and real-time synthesis latency. Use when the user wants to benchmark on LibriSpeech test-clean, SEED-TTS test-en, or asks about evaluating this task. Reports Naturalness.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.15969
  bibtex_key: torgashov2025voxstream
  confidence: high
---

# voxstream-tts-eval

> VoXtream: Full-Stream Text-to-Speech with Extremely Low Latency — Torgashov et al. (2025) (arXiv:2509.15969, 2025)

## What this evaluates

Evaluates a streaming text-to-speech model's ability to generate high-quality, speaker-similar, and intelligible audio from text prompts in both non-streaming and full-stream (word-by-word) settings. It probes zero-shot voice cloning, cross-sentence continuity, and real-time synthesis latency.

## Datasets

- **LibriSpeech test-clean** — total ?; splits: test-clean (-1), long (-1)
- **SEED-TTS test-en** — total ?; splits: test-en (-1)

## Metrics

- `WER (%)` — range: percent
  - Word Error Rate between the ASR transcription of synthesized speech and the input text. Whisper-large-v3 is used for SEED-TTS, while a HuBERT-based ASR is used for LibriSpeech.
- `SPK-SIM` — range: [0, 1]
  - Cosine similarity between WavLM-based ECAPA-TDNN embeddings extracted from the reference prompt and the generated audio.
- `UTMOS` — range: [0, 5]
  - MOS predictor score estimating audio quality.
- `Naturalness` **(primary)** — range: [0, 100]
  - MUSHRA-like user study rating on a 0–100 scale, computed as mean ± 95% CI.
- `FPL (ms)` — range: ms
  - First-packet latency, defined as the time to the first speech frame.
- `RTF` — range: other
  - Real-time factor, defined as the ratio of generated speech duration to wall-clock generation time.

## Input / output format

**Input**: Text input (phonemes or raw text) provided sequentially word-by-word to simulate streaming LLM input, plus an audio prompt for speaker reference.

**Output**: Synthesized audio waveform (or acoustic tokens decoded to speech).

## Scoring recipe

```python
def compute_metrics(generated_audio, reference_audio, gold_text):
    # Intelligibility
    transcribed = asr_model(generated_audio)  # Whisper-large-v3 for SEED, HuBERT for LibriSpeech
    wer = word_error_rate(gold_text, transcribed)
    # Speaker Similarity
    ref_emb = speaker_encoder(reference_audio)  # WavLM-based ECAPA-TDNN
    gen_emb = speaker_encoder(generated_audio)
    spk_sim = cosine_similarity(ref_emb, gen_emb)
    # Quality
    utmos = utmos_predictor(generated_audio)
    return {'WER': wer, 'SPK-SIM': spk_sim, 'UTMOS': utmos}
```

## Common pitfalls

- Different ASR backends are used for different datasets (Whisper-large-v3 for SEED-TTS, HuBERT for LibriSpeech), making cross-dataset WER comparisons invalid.
- Naturalness user studies must only include sentences where all baseline models achieve 0% WER to prevent intelligibility failures from biasing perceived naturalness scores.
- In full-stream evaluation, some baselines (e.g., CosyVoice2) re-synthesize the prompt when target text is shorter, artificially inflating WER; these cases were explicitly excluded by the authors.

## Evidence (verbatim from paper)

> We used three reproducible model-based metrics. For intelligibility, we report WER between the transcription of synthesized speech and the input text. For SEED-TTS test-en we used Whisper-large-v3 and followed metric calculation from the official SEED test repository. For LibriSpeech test-clean we used a HuBERT-based ASR and prepended the audio prompt to the generated continuation for WER, as in [15]. For speaker similarity, we computed cosine similarity (SPK-SIM) between embeddings from a WavLM-based ECAPA-TDNN for the prompt and synthesized speech. For quality, we used the UTMOS MOS predictor.

## Citation

```bibtex
@misc{torgashov2025voxstream,
  title={VoXtream: Full-Stream Text-to-Speech with Extremely Low Latency},
  author={Torgashov et al. (2025)},
  year={2025},
  note={arXiv:2509.15969}
}
```

- arXiv: 2509.15969