Instruct Tts Eval

---
name: instruct-tts-eval
description: Evaluates a text-to-speech system's ability to follow complex natural-language instructions for acoustic parameter specification, descriptive style direction, and role-play scenarios. It probes fine-grained prosodic control, open-ended style inference, and high-level scenario-based emotional/character expression. Use when the user wants to benchmark on InstructTTSEval, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.16381
  bibtex_key: huang2025instructtts
  confidence: high
---

# instruct-tts-eval

> InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems — Huang et al. (2025) (arXiv:2506.16381, 2025)

## What this evaluates

Evaluates a text-to-speech system's ability to follow complex natural-language instructions for acoustic parameter specification, descriptive style direction, and role-play scenarios. It probes fine-grained prosodic control, open-ended style inference, and high-level scenario-based emotional/character expression.

## Datasets

- **InstructTTSEval** — total ?; splits: EN-subset (-1), ZH-subset (-1); repo https://github.com/KexinHUANG19/InstructTTSEval

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of generated audio samples correctly matched to their corresponding natural-language instruction by the LLM judge (Gemini). Calculated as (number of correct matches / total evaluated samples) * 100.

## Input / output format

**Input**: Natural-language instruction specifying acoustic parameters, style directives, or role-play scenarios.

**Output**: Synthesized audio file generated by the TTS model.

## Scoring recipe

```python
def compute_accuracy(predictions, instructions):
    correct = 0
    for audio, instruction in zip(predictions, instructions):
        judge_output = gemini_judge(audio, instruction)
        if judge_output == 'match':
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- LLM judges may exhibit self-preference bias, inflating scores for models from the same family (e.g., Gemini models scoring higher on Gemini-generated audio).
- Mismatched instruction-audio pairs might still partially align due to the many-to-many nature of speech and description, leading to false negatives in consistency checks.
- Safety filters in commercial models (e.g., Hume blocking child voices) can artificially lower scores unrelated to instruction-following capability.

## Evidence (verbatim from paper)

> Accuracy | APS | DSD | RP | Avg.
| --- | --- | --- | --- | --- |
| EN | 86% | 78% | 66% | 76.7% |

## Citation

```bibtex
@misc{huang2025instructtts,
  title={InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems},
  author={Huang et al. (2025)},
  year={2025},
  note={arXiv:2506.16381}
}
```

- arXiv: 2506.16381