Instructttseval Zh Eval

---
name: instructttseval-zh-eval
description: Evaluates a text-to-speech model's ability to follow natural-language instructions for voice design, specifically controlling acoustic parameters, descriptive styles, and role-play characteristics. It measures how accurately synthesized speech adheres to explicit semantic and stylistic requirements. Use when the user wants to benchmark on InstructTTSEval-Zh, or asks about evaluating this task. Reports AVG.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.10629
  bibtex_key: hu2026voicesculptor
  confidence: high
---

# instructttseval-zh-eval

> VoiceSculptor: Your Voice, Designed By You — Jingbin Hu et al. (2026) (arXiv:2601.10629, 2026)

## What this evaluates

Evaluates a text-to-speech model's ability to follow natural-language instructions for voice design, specifically controlling acoustic parameters, descriptive styles, and role-play characteristics. It measures how accurately synthesized speech adheres to explicit semantic and stylistic requirements.

## Datasets

- **InstructTTSEval-Zh** — total ?; splits: test (-1)

## Metrics

- `APS` — range: percent
  - Acoustic-Parameter Specification score (%). Measures adherence to explicit acoustic attribute instructions (e.g., pitch, rate, age).
- `DSD` — range: percent
  - Descriptive-Style Directive score (%). Measures adherence to descriptive style instructions (e.g., emotion, tone).
- `RP` — range: percent
  - Role-Play score (%). Measures adherence to role-playing or character-specific voice instructions.
- `AVG` **(primary)** — range: percent
  - Average of APS, DSD, and RP scores (%). Serves as the headline metric for overall instruction-following performance.
- `IMOS` — range: other
  - Instruction-following Mean Opinion Score. A human subjective rating on a standardized MOS-style scale assessing how well synthesized speech adheres to semantic and stylistic instructions.

## Input / output format

**Input**: Natural language instruction specifying voice attributes (e.g., pitch, emotion, style) and optionally a target text for synthesis.

**Output**: Synthesized audio waveform matching the instructed voice characteristics.

## Scoring recipe

```python
def score_automated(predictions, instructions):
    aps_scores, dsd_scores, rp_scores = [], [], []
    for pred, inst in zip(predictions, instructions):
        res = llm_evaluate(audio=pred, instruction=inst) # LLM (e.g., Gemini 2.5 Pro) scores adherence
        aps_scores.append(res['APS'])
        dsd_scores.append(res['DSD'])
        rp_scores.append(res['RP'])
    aps = mean(aps_scores)
    dsd = mean(dsd_scores)
    rp = mean(rp_scores)
    avg = mean([aps, dsd, rp])
    return aps, dsd, rp, avg

def score_human(predictions, instructions):
    ratings = []
    for pred, inst in zip(predictions, instructions):
        rating = human_listener_rate(audio=pred, instruction=inst) # Standardized MOS scale
        ratings.append(rating)
    imos = mean(ratings)
    return imos
```

## Common pitfalls

- LLM evaluator may exhibit bias toward commercial models or specific instruction phrasings.
- Evaluation is conducted exclusively on Chinese data, limiting claims about cross-lingual generalization.
- Commercial baselines are evaluated via official APIs while open-source models use local inference, potentially introducing infrastructure or latency biases.

## Evidence (verbatim from paper)

> To this end, we employ InstructTTSEval-Zh, a Chinese instruction-based TTS evaluation benchmark designed to measure how well a model follows natural-language instructions in speech synthesis. The benchmark evaluates multiple aspects of instruction controllability, including Acoustic-Parameter Specification (APS), Descriptive-Style Directive (DSD), and Role-Play (RP). These metrics are computed by synthesizing speech from instruction prompts and assessing the generated audio using a unified evaluation protocol with a large language model as the evaluator.

## Citation

```bibtex
@misc{hu2026voicesculptor,
  title={VoiceSculptor: Your Voice, Designed By You},
  author={Jingbin Hu et al. (2026)},
  year={2026},
  note={arXiv:2601.10629}
}
```

- arXiv: 2601.10629