Stylebench Eval

---
name: stylebench-eval
description: Evaluates speech language models' ability to control four paralinguistic dimensions (emotion, speed, volume, pitch) across multi-turn dialogues. It probes whether models can follow gradational style-intensity instructions while preserving fixed semantic content and maintaining coherent intensity trajectories across turns. Use when the user wants to benchmark on StyleBench, or asks about evaluating this task. Reports dimension-specific metrics.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.07599
  bibtex_key: zhao2026stylebench
  confidence: medium
---

# stylebench-eval

> StyleBench: Evaluating Speech Language Models on Conversational Speaking Style Control — Zhao et al. (2026) (arXiv:2603.07599, 2026)

## What this evaluates

Evaluates speech language models' ability to control four paralinguistic dimensions (emotion, speed, volume, pitch) across multi-turn dialogues. It probes whether models can follow gradational style-intensity instructions while preserving fixed semantic content and maintaining coherent intensity trajectories across turns.

## Datasets

- **StyleBench** — total 86400; splits: test (-1)

## Metrics

- `dimension-specific metrics` **(primary)** — range: other
  - Quantitative measures applied per paralinguistic dimension (emotion, speed, volume, pitch) to assess how closely generated speech matches the target intensity level specified in the prompt.
- `human evaluation` — range: percent
  - Human raters assess the controllability and naturalness of the generated speech across the four style dimensions and intensity trajectories.

## Input / output format

**Input**: Three-turn conversational dialogue. Each turn contains a user prompt with gradational style-intensity instructions and a fixed semantic answer text. Semantic content remains identical across turns; only the style instruction changes.

**Output**: Synthesized speech audio corresponding to the answer text, modulated to match the requested paralinguistic style and intensity level specified in the prompt.

## Scoring recipe

```python
for dialogue in StyleBench:
    for turn in dialogue.turns:
        audio = model.generate(turn.prompt, turn.text)
        score = compute_dimension_metric(audio, target_intensity)
        scores.append(score)
final_score = aggregate(scores)
# Human evaluation: raters score controllability and naturalness on a scale
```

## Common pitfalls

- Models may alter the semantic content of the answer instead of just modulating the acoustic style, violating the fixed-text protocol.
- Evaluators might treat style control as categorical rather than gradational/intensity-based, missing the benchmark's core trajectory design.
- Ignoring the multi-turn intensity trajectory (increasing vs. decreasing) when scoring, which is a key controllability measure.

## Evidence (verbatim from paper)

> Evaluating the conversational speaking style control ability of SLMs necessitates coherent multi-turn dialogue modeling alongside accurate stylistic speech generation. We therefore construct a multi-turn QA dataset designed for controlling style and intensity. In this section, we outline StyleBench, a bilingual three-turn dialogue dataset that enables the evaluation of SLMs on: (1) understanding natural language instructions with gradational style variations, and (2) preserving hierarchical intensity control across dialogue turns. ... human evaluation and dimension-specific metrics providing fine-grained insights into controllability.

## Citation

```bibtex
@misc{zhao2026stylebench,
  title={StyleBench: Evaluating Speech Language Models on Conversational Speaking Style Control},
  author={Zhao et al. (2026)},
  year={2026},
  note={arXiv:2603.07599}
}
```

- arXiv: 2603.07599