Instructaudio Eval

---
name: instructaudio-eval
description: This evaluation probes a model's ability to generate speech and music conditioned on natural language instructions describing acoustic and musical attributes. It measures text-to-audio fidelity, attribute control accuracy, and perceptual quality across short-form generation tasks. Use when the user wants to benchmark on Seed-TTS benchmark, InstructAudio internal test set, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.18487
  bibtex_key: qiang2025instructaudio
  confidence: high
---

# instructaudio-eval

> InstructAudio: Unified speech and music generation with natural language instruction — Qiang et al. (2025) (arXiv:2511.18487, 2025)

## What this evaluates

This evaluation probes a model's ability to generate speech and music conditioned on natural language instructions describing acoustic and musical attributes. It measures text-to-audio fidelity, attribute control accuracy, and perceptual quality across short-form generation tasks.

## Datasets

- **Seed-TTS benchmark** — total ?; splits: test (-1)
- **InstructAudio internal test set** — total 500; splits: test (500)

## Metrics

- `WER` **(primary)** — range: percent
  - Word Error Rate: percentage of incorrectly recognized words relative to the reference transcript, computed via edit distance alignment.
- `Classification Control Accuracy` — range: percent
  - Percentage of generated samples where a classifier correctly identifies the target attribute (e.g., gender, age, emotion) matching the instruction.
- `SongEval` — range: other
  - Benchmark scoring coherence, musicality, memorability, clarity, and naturalness on a 1-5 scale based on human listening tests.
- `QMOS/NMOS/MMOS` — range: other
  - Mean Opinion Scores from trained evaluators rating audio quality, naturalness, or musicality respectively.

## Input / output format

**Input**: Natural language instruction specifying target attributes (e.g., gender, age, emotion, style, accent for speech; genre, instrument, rhythm, atmosphere for music), optionally paired with ground truth text/lyrics.

**Output**: 44.1kHz audio waveform clip (2–20 seconds).

## Scoring recipe

```python
def compute_wer(predictions, gold):
    alignment = edit_distance_align(predictions, gold)
    errors = sum(alignment.substitutions + alignment.deletions + alignment.insertions)
    total = len(gold.split())
    return (errors / total) * 100 if total > 0 else 0.0

def compute_control_accuracy(predictions, gold_attributes):
    correct = 0
    for pred_audio, target_attr in zip(predictions, gold_attributes):
        predicted_attr = classifier.predict(pred_audio)
        if predicted_attr == target_attr:
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Using reference audio for baseline models (e.g., CosyVoice2) while InstructAudio uses text-only conditioning creates an unfair timbre control comparison.
- Truncating long music generations to 5–20s clips biases evaluation against models optimized for full-length tracks.
- Short clip duration limits assessment of temporal coherence and long-form structural quality.

## Evidence (verbatim from paper)

> We employ comprehensive objective and subjective metrics to ensure thorough evaluation. Objective metrics include Word Error Rate (WER) using Seed-TTS, Speaker Similarity, Emotion Similarity, Log-Spectral Distance (LSD), Mel-Cepstral Distortion (MCD), Mean Squared Error of Pitch (MSEP), Voiced/Unvoiced Mismatch Rate (MR), SongEval music evaluation benchmark, and classification control accuracy through perceptual consistency assessment.

## Citation

```bibtex
@misc{qiang2025instructaudio,
  title={InstructAudio: Unified speech and music generation with natural language instruction},
  author={Qiang et al. (2025)},
  year={2025},
  note={arXiv:2511.18487}
}
```

- arXiv: 2511.18487