Step Audio Editx Eval

---
name: step-audio-editx-eval
description: Probes a model's capability to accurately edit or synthesize audio with specific emotional tones, speaking styles, and paralinguistic elements (e.g., laughter, sighs) using zero-shot voice cloning or iterative refinement. It measures how well the model preserves linguistic content while transferring or inserting target audio attributes. Use when the user wants to benchmark on Step-Audio-Edit-Test, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.03601
  bibtex_key: yan2025stepaudioeditx
  confidence: high
---

# step-audio-editx-eval

> Step-Audio-EditX Technical Report — Yan et al. (2025) (arXiv:2511.03601, 2025)

## What this evaluates

Probes a model's capability to accurately edit or synthesize audio with specific emotional tones, speaking styles, and paralinguistic elements (e.g., laughter, sighs) using zero-shot voice cloning or iterative refinement. It measures how well the model preserves linguistic content while transferring or inserting target audio attributes.

## Datasets

- **Step-Audio-Edit-Test** — total ?; splits: test (-1); repo https://github.com/stepfun-ai/Step-Audio-EditX

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly classified audio samples out of the total, averaged across all speakers and target categories (5 emotions, 7 styles).
- `paralinguistic_score` — range: [1, 3]
  - Average score assigned by the Gemini-2.5-Pro LLM judge on a 1–3 scale (3 = perfect, 2 = flawed, 1 = failed) for paralinguistic insertion accuracy.

## Input / output format

**Input**: Generated audio clips (from zero-shot voice cloning or closed-source TTS) paired with the target instruction or category label (e.g., emotion, style, or paralinguistic tag).

**Output**: Categorical classification label for emotion/style tasks; integer score from 1 to 3 for paralinguistic tasks.

## Scoring recipe

```python
# Emotion/Style Accuracy
correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
accuracy = (correct / len(gold_labels)) * 100

# Paralinguistic Score
llm_scores = [gemini_judge_score(audio, instruction) for audio, instruction in zip(audios, instructions)]
paralinguistic_score = sum(llm_scores) / len(llm_scores)
```

## Common pitfalls

- LLM-as-a-judge (Gemini-2.5-Pro) may favor fluent but content-less audio, leading to inflated scores for paralinguistic omission.
- Iterative editing setup (N=3) means final scores depend heavily on the prompt audio provided in each step; holding prompt audio constant significantly alters results.
- Baseline 'Iter0' scores reflect zero-shot cloning capability, not editing performance, and should not be conflated with the model's editing accuracy.

## Evidence (verbatim from paper)

> We introduce Step-Audio-Edit-Test, a benchmark that leverages LLM-as-a-judge model to evaluate model performance on emotion, speaking style, and paralinguistics. All evaluation audio is generated via zero-shot voice cloning and subsequently scored using the Gemini-2.5-Pro model... The final accuracy for each category is calculated as the average across all speakers... employing a rigorous 1–3 scoring scale (3 = perfect, 2 = flawed, 1 = failed)... Finally, model performance in the paralinguistic editing task is assessed by calculating the overall average score generated by Gemini-2.5-Pro model.

## Citation

```bibtex
@misc{yan2025stepaudioeditx,
  title={Step-Audio-EditX Technical Report},
  author={Yan et al. (2025)},
  year={2025},
  note={arXiv:2511.03601}
}
```

- arXiv: 2511.03601