Instructtts Eval

---
name: instructtts-eval
description: Evaluates a text-to-speech model's capability to generate realistic vocal timbres that accurately follow complex natural-language style instructions. It probes fine-grained acoustic control, generalization to unstructured descriptions, and contextual role-play inference. Use when the user wants to benchmark on InstructTTSEval, or asks about evaluating this task. Reports Instruction-following accuracy (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.28086
  bibtex_key: huang2026mossvoicegenerator
  confidence: high
---

# instructtts-eval

> MOSS-VoiceGenerator: Create Realistic Voices with Natural Language Descriptions — Huang et al. (2026) (arXiv:2603.28086, 2026)

## What this evaluates

Evaluates a text-to-speech model's capability to generate realistic vocal timbres that accurately follow complex natural-language style instructions. It probes fine-grained acoustic control, generalization to unstructured descriptions, and contextual role-play inference.

## Datasets

- **InstructTTSEval** — total 6000; splits: test (6000)

## Metrics

- `Instruction-following accuracy (%)` **(primary)** — range: percent
  - Percentage of generated audio clips that correctly satisfy the specified acoustic attributes, descriptive styles, or role-play scenarios in the instruction. Computed separately for each task (APS, DSD, RP) and language (EN, ZH).
- `Pairwise preference win rate` — range: percent
  - Percentage of comparisons where the model's output is preferred over a baseline's output by human annotators, calculated as (wins / total comparisons) * 100 across dimensions like Overall Performance, Instruction Following, and Naturalness.

## Input / output format

**Input**: Natural language instruction describing vocal style/attributes + target text to synthesize.

**Output**: Generated audio clip.

## Scoring recipe

```python
# Objective: Instruction-following accuracy
correct = sum(1 for pred, gold in zip(predictions, golds) if pred.matches_instruction(gold))
accuracy = (correct / len(golds)) * 100

# Subjective: Pairwise preference win rate
wins = sum(1 for w, t, l in pairwise_results if w)
win_rate = (wins / len(pairwise_results)) * 100
```

## Common pitfalls

- Test-set contamination must be prevented via fuzzy matching on transcripts, as training data overlaps with benchmark sources.
- Comparisons are invalid if baselines only support fixed voice editing rather than free-form instruction-driven design.
- Subjective evaluation uses majority vote from 3 annotators per item, with each annotator scoring only one dimension to reduce bias.

## Evidence (verbatim from paper)

> We evaluate MOSS-VoiceGenerator on InstructTTSEval, a public benchmark designed to assess TTS models’ ability to follow complex natural-language style instructions. InstructTTSEval comprises 6,000 test cases (3 tasks × 2 languages × 1,000 samples) drawn from movies, TV dramas, and variety shows, each paired with a reference audio clip. ... Table 1: Instruction-following accuracy (%) on InstructTTSEval.

## Citation

```bibtex
@misc{huang2026mossvoicegenerator,
  title={MOSS-VoiceGenerator: Create Realistic Voices with Natural Language Descriptions},
  author={Huang et al. (2026)},
  year={2026},
  note={arXiv:2603.28086}
}
```

- arXiv: 2603.28086