Step Audio R1 Eval

---
name: step-audio-r1-eval
description: Evaluates audio language models on speech understanding, reasoning, and real-time interactive dialogue capabilities using raw acoustic signals rather than textual transcriptions. It measures both comprehension accuracy across multiple audio benchmarks and real-time generation fluency. Use when the user wants to benchmark on Big Bench Audio, Spoken MQA, MMSU, MMAU, Wild Speech, or asks about evaluating this task. Reports Average Score (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.15848
  bibtex_key: [REDACTED]
  confidence: high
---

# step-audio-r1-eval

> Step-Audio-R1 Technical Report — Tian et al. (2025) (arXiv:2511.15848, 2025)

## What this evaluates

Evaluates audio language models on speech understanding, reasoning, and real-time interactive dialogue capabilities using raw acoustic signals rather than textual transcriptions. It measures both comprehension accuracy across multiple audio benchmarks and real-time generation fluency.

## Datasets

- **Big Bench Audio** — total ?; splits: test (-1); HF `ArtificialAnalysis/big_bench_audio`
- **Spoken MQA** — total ?; splits: test (-1)
- **MMSU** — total ?; splits: test (-1)
- **MMAU** — total ?; splits: test (-1)
- **Wild Speech** — total ?; splits: test (-1)

## Metrics

- `Average Score (%)` **(primary)** — range: percent
  - Arithmetic mean of the percentage scores across Big Bench Audio, Spoken MQA, MMSU, MMAU, and Wild Speech benchmarks.
- `Speech Reasoning Performance Score (%)` — range: percent
  - Percentage score measuring the model’s reasoning ability over spoken content in the speech-to-speech benchmark.
- `Latency (seconds)` — range: seconds
  - Time in seconds to generate the first audio packet, quantifying response speed as an indicator of dialogue fluency.

## Input / output format

**Input**: Raw audio recordings of spoken speech or acoustic signals.

**Output**: Text transcriptions or answers for speech-to-text tasks; spoken audio responses for speech-to-speech tasks.

## Scoring recipe

```python
# Speech-to-text evaluation
benchmarks = [BigBenchAudio, SpokenMQA, MMSU, MMAU, WildSpeech]
scores = [get_accuracy(model, bench) for bench in benchmarks]
avg_score = sum(scores) / len(scores)

# Speech-to-speech evaluation
reasoning_score = evaluate_reasoning(audio_input, audio_output)
latency = measure_first_packet_time(audio_output)
```

## Common pitfalls

- Evaluating on transcribed text instead of raw acoustic signals, which defeats the purpose of modality-grounded reasoning.
- Comparing the standard Step-Audio-R1 model directly to the 'Realtime' variant's latency metrics without noting the architectural adaptation.
- Assuming the benchmarks are single-domain tests rather than a multi-task suite requiring an average score.

## Evidence (verbatim from paper)

> This benchmark comprises two evaluation metrics: the Speech Reasoning Performance Score (%), measuring the model’s reasoning ability over spoken content, and the first-packet Latency (seconds) metric, quantifying response speed as an indicator of dialogue fluency.

## Citation

```bibtex
@misc{tian2025stepaudior1,
  title={Step-Audio-R1 Technical Report},
  author={Tian et al. (2025)},
  year={2025},
  note={arXiv:2511.15848}
}
```

- arXiv: 2511.15848