Ultravoice Style Control Eval

---
name: ultravoice-style-control-eval
description: Evaluates spoken dialogue models' ability to follow fine-grained speech style instructions (emotion, speed, volume, accent, language, composite) while maintaining general conversational competence. It measures both subjective audio quality/naturalness and objective content/emotion alignment against ground-truth style specifications. Use when the user wants to benchmark on UltraVoice Test Set, URO-Bench, or asks about evaluating this task. Reports MOS, IFR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.22588
  bibtex_key: tu2025ultravoice
  confidence: high
---

# ultravoice-style-control-eval

> UltraVoice: Scaling Fine-Grained Style-Controlled Speech Conversations for Spoken Dialogue Models — Tu et al. (2025) (arXiv:2510.22588, 2025)

## What this evaluates

Evaluates spoken dialogue models' ability to follow fine-grained speech style instructions (emotion, speed, volume, accent, language, composite) while maintaining general conversational competence. It measures both subjective audio quality/naturalness and objective content/emotion alignment against ground-truth style specifications.

## Datasets

- **UltraVoice Test Set** — total 2300; splits: test (2300); repo https://github.com/bigai-nlco/UltraVoice
- **URO-Bench** — total ?; splits: test (-1)

## Metrics

- `MOS` **(primary)** — range: [1, 5]
  - Mean Opinion Score generated automatically by Gemini-2.5-Flash using custom prompts. Models rate audio on a scale reflecting naturalness and style adherence.
- `IFR` **(primary)** — range: percent
  - Instruction-Following Rate, calculated as the percentage of generated responses that successfully satisfy the specified style instruction according to the ALM evaluator.
- `WER` — range: percent
  - Word Error Rate computed by transcribing generated speech with Whisper-large-v3 and comparing against the ground-truth text transcript.
- `Emotion Similarity` — range: [0, 1]
  - Cosine similarity between emotion2vec embeddings of the generated speech and the ground-truth reference speech.
- `Emotion Recall Rate` — range: percent
  - Accuracy of an emotion classifier predicting the intended emotion from the generated speech compared to the ground-truth label.
- `UTMOS` — range: [1, 5]
  - Perceptual audio quality and naturalness score predicted by the UTMOS model.

## Input / output format

**Input**: Text prompt containing dialogue context and explicit style control instructions (e.g., emotion, speed, volume), optionally with prior audio context.

**Output**: Synthesized speech audio response matching the requested style and content.

## Scoring recipe

```python
def score_ultravoice(predictions, golds):
    scores = {'MOS': [], 'IFR': [], 'WER': [], 'Emotion_Sim': [], 'Emotion_Recall': [], 'UTMOS': []}
    for pred_audio, gold in zip(predictions, golds):
        mos, ifr = evaluate_alm_gemini(pred_audio, gold['instruction'])
        scores['MOS'].append(mos); scores['IFR'].append(ifr)
        pred_text = whisper_transcribe(pred_audio)
        scores['WER'].append(calculate_wer(pred_text, gold['text']))
        pred_emb = emotion2vec_embed(pred_audio)
        gold_emb = emotion2vec_embed(gold['reference_audio'])
        scores['Emotion_Sim'].append(cosine_similarity(pred_emb, gold_emb))
        pred_emotion = emotion_classifier(pred_audio)
        scores['Emotion_Recall'].append(pred_emotion == gold['emotion'])
        scores['UTMOS'].append(utmos_model(pred_audio))
    return {k: mean(v) for k, v in scores.items()}
```

## Common pitfalls

- ALM-based MOS/IFR scores depend heavily on prompt design and may exhibit bias toward certain acoustic characteristics or languages.
- WER computed via Whisper can be inflated for fast-speed or accented speech due to transcription model limitations, not necessarily model failure.
- Emotion2vec similarity measures acoustic embedding distance, which may not perfectly correlate with human perceptual style control.

## Evidence (verbatim from paper)

> To construct our evaluation benchmark, we randomly sampled 100 examples from each fine-grained dimension within the six major control dimensions defined in UltraVoice, resulting in a test set of 2,300 samples. The test set has no overlap with the training data. To further evaluate whether SFT on UltraVoice impacts general spoken dialogue capabilities such as natural conversation, comprehension, and reasoning, we utilized the URO-Bench (Yan et al., [2025]), which assesses models across three dimensions: Oral Conversation, Understanding, and Reasoning. It allows us to analyze whether core dialogue competencies are preserved and whether expressive performance improves after fine-tuning. Audio-Language Model (ALM) based Metric. Following the evaluation paradigm similar to methodologies proposed by Yan et al. ([2025]); Yang et al. ([2025]), we employed Gemini-2.5-Flash (Comanici et al., [2025]) as our automatic evaluator to automatically generate Mean Opinion Scores (MOS) and compute the instruction-following rate (IFR) for each control dimension.

## Citation

```bibtex
@misc{tu2025ultravoice,
  title={UltraVoice: Scaling Fine-Grained Style-Controlled Speech Conversations for Spoken Dialogue Models},
  author={Tu et al. (2025)},
  year={2025},
  note={arXiv:2510.22588}
}
```

- arXiv: 2510.22588