Voxtream2 Eval

---
name: voxtream2-eval
description: This evaluation probes a full-stream text-to-speech model's ability to generate intelligible, natural-sounding speech while dynamically controlling the speaking rate in real-time. It measures objective intelligibility, speaker similarity, audio quality, generation latency, and the accuracy of speaking-rate control against target rates. Use when the user wants to benchmark on Emilia speaking-rate dataset, or asks about evaluating this task. Reports WER (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.13518
  bibtex_key: [REDACTED]
  confidence: high
---

# voxtream2-eval

> VoXtream2: Full-stream TTS with dynamic speaking rate control — Torgashov et al. (2026) (arXiv:2603.13518, 2026)

## What this evaluates

This evaluation probes a full-stream text-to-speech model's ability to generate intelligible, natural-sounding speech while dynamically controlling the speaking rate in real-time. It measures objective intelligibility, speaker similarity, audio quality, generation latency, and the accuracy of speaking-rate control against target rates.

## Datasets

- **Emilia speaking-rate dataset** — total ?; splits: test (-1)

## Metrics

- `WER (%)` **(primary)** — range: percent
  - Word Error Rate calculated as the percentage of incorrectly transcribed words relative to the ground truth text.
- `SPK-SIM` — range: [0, 1]
  - Speaker Similarity score computed as the cosine similarity between speaker embeddings of the generated audio and the reference acoustic prompt.
- `UTMOS` — range: other
  - Universal TTS Mean Opinion Score, an objective predictor of perceived audio quality and naturalness.
- `FPL (ms)` — range: ms
  - First Packet Latency, measuring the time elapsed from input submission to the generation of the first audio packet.
- `RTF` — range: [0, 1]
  - Real-Time Factor, calculated as the ratio of total generated audio duration to the total inference time.
- `Corr.` — range: [0, 1]
  - Pearson correlation coefficient between the target speaking rate (SPS) control signal and the actual synthesized speaking rate extracted via phoneme alignment.

## Input / output format

**Input**: Text input, acoustic prompt (reference speech), and target speaking rate or control signal (e.g., SPS value or linearly varying signal).

**Output**: Synthesized audio waveform.

## Scoring recipe

```python
def compute_metrics(audio, text, prompt_audio, target_sps, control_signal):
    wer = wer_engine(transcribe(audio), text)
    spk_sim = cosine_similarity(speaker_encoder(audio), speaker_encoder(prompt_audio))
    utmos = utmos_model(audio)
    fpl = time_to_first_packet()
    rtf = len(audio) / inference_time()
    target_rates = extract_sps(control_signal)
    gen_rates = extract_sps_from_alignments(audio)
    corr = pearsonr(target_rates, gen_rates)
    return {'WER': wer, 'SPK-SIM': spk_sim, 'UTMOS': utmos, 'FPL': fpl, 'RTF': rtf, 'Corr.': corr}
```

## Common pitfalls

- WER can be artificially inflated at slow speaking rates due to model-generated fillers and word repetitions, which sound natural but penalize transcription accuracy.
- SPK-SIM and UTMOS are not reported for the full-stream setting because they showed no significant difference compared to non-streaming evaluation.
- Spark-TTS is excluded from speaker similarity (SPK-SIM) evaluation because its controllable version does not support voice cloning.
- UTMOS ratings are sensitive to evaluation design and may reflect overall audio quality or background noise rather than strictly naturalness.

## Evidence (verbatim from paper)

> We benchmark the performance of our model against prior work in Table 4. VoXtream2 achieves the lowest FPL and RTF among publicly available full-stream models. We also evaluate different GPU inference optimizations. While torch.compile provides the best runtime performance, it requires longer graph compilation time compared to CUDA Graphs.

## Citation

```bibtex
@misc{torgashov2026voxstream2,
  title={VoXtream2: Full-stream TTS with dynamic speaking rate control},
  author={Torgashov et al. (2026)},
  year={2026},
  note={arXiv:2603.13518}
}
```

- arXiv: 2603.13518