Batonvoice Eval

---
name: batonvoice-eval
description: Evaluates a controllable text-to-speech model's ability to generate intelligible speech and accurately convey specific emotional tones based on text instructions. It probes zero-shot cross-lingual generalization and instruction-following capabilities in speech synthesis. Use when the user wants to benchmark on Seed-TTS, Emotion dataset, or asks about evaluating this task. Reports Emotion Classification Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.26514
  bibtex_key: [REDACTED]
  confidence: high
---

# batonvoice-eval

> BatonVoice: An Operationalist Framework for Enhancing Controllable Speech Synthesis with Linguistic Intelligence from LLMs — Wang et al. (2025) (arXiv:2509.26514, 2025)

## What this evaluates

Evaluates a controllable text-to-speech model's ability to generate intelligible speech and accurately convey specific emotional tones based on text instructions. It probes zero-shot cross-lingual generalization and instruction-following capabilities in speech synthesis.

## Datasets

- **Seed-TTS** — total ?; splits: test (-1)
- **Emotion dataset** — total 500; splits: test (500)

## Metrics

- `Word Error Rate (WER)` — range: percent
  - Calculated by transcribing the generated speech with a pre-trained ASR model (Whisper-large-v3) and comparing it to the reference text. Lower values indicate higher intelligibility.
- `Emotion Classification Accuracy` **(primary)** — range: [0, 1]
  - Percentage of synthesized utterances correctly classified into one of five target emotions (joy, sadness, anger, surprise, fear) by an LLM judge (Gemini-2.5-Pro). Higher values indicate better emotional control.

## Input / output format

**Input**: Text instructions (emotion labels or free-form descriptive prompts) and, for Seed-TTS, a short reference speech prompt.

**Output**: Synthesized speech audio (discrete vocal tokens or waveform) conditioned on the input text and instructed emotional style.

## Scoring recipe

```python
# Word Error Rate (WER)
asr_transcript = whisper_large_v3.transcribe(generated_audio)
wer = wer_score(reference_text, asr_transcript)

# Emotion Classification Accuracy
prompt = f"Classify the emotion of this speech: {generated_audio}"
pred_emotion = gemini_2_5_pro.classify(prompt)
accuracy = sum(pred_emotion == gold_emotion) / len(gold_emotion)
```

## Common pitfalls

- WER scores depend heavily on the ASR backbone used (here, Whisper-large-v3), preventing direct cross-paper comparison without re-evaluation.
- Emotion accuracy uses a single LLM-as-a-judge (Gemini-2.5-Pro) instead of human raters, risking systematic bias or inconsistency.
- The emotion benchmark only covers five basic emotions, limiting assessment of nuanced or blended emotional control.

## Evidence (verbatim from paper)

> We measure performance using Emotion Classification Accuracy. This metric is derived by employing Google’s Gemini-2.5-Pro to classify the emotion of the synthesized speech. A higher accuracy indicates a greater success rate in generating perceptually accurate emotional speech.

## Citation

```bibtex
@misc{wang2025batonvoice,
  title={BatonVoice: An Operationalist Framework for Enhancing Controllable Speech Synthesis with Linguistic Intelligence from LLMs},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2509.26514}
}
```

- arXiv: 2509.26514