Uro Bench Eval

---
name: uro-bench-eval
description: Evaluates end-to-end speech-to-speech dialogue models across three core dimensions: understanding, reasoning, and oral conversation. It probes multilingual proficiency, multi-turn dialogue handling, and the ability to generate paralinguistic and emotional cues in audio responses. Use when the user wants to benchmark on URO-Bench, or asks about evaluating this task. Reports Task Accomplish Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.17810
  bibtex_key: yan2025urobench
  confidence: medium
---

# uro-bench-eval

> URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Models — Yan et al. (2025) (arXiv:2502.17810, 2025)

## What this evaluates

Evaluates end-to-end speech-to-speech dialogue models across three core dimensions: understanding, reasoning, and oral conversation. It probes multilingual proficiency, multi-turn dialogue handling, and the ability to generate paralinguistic and emotional cues in audio responses.

## Datasets

- **URO-Bench** — total 40; splits: basic (-1), pro (-1)

## Metrics

- `Task Accomplish Score` **(primary)** — range: percent
  - A composite metric measuring task completion and content quality across understanding, reasoning, and oral conversation dimensions.
- `UTMOS` — range: [0, 5]
  - Automatic mean opinion score for predicting natural speech quality.
- `Matching Rate` — range: percent
  - Percentage of pairwise model preferences where automated (GPT or rule-based) scores align with human evaluations.

## Input / output format

**Input**: Audio prompts containing questions, instructions, or conversational turns in spoken dialogue.

**Output**: Audio responses generated by the end-to-end spoken dialogue model.

## Scoring recipe

```python
# 1. Task Accomplish Score & Content Quality
task_score = evaluate_task_completion(predictions, gold)
# 2. UTMOS for speech quality
utmos_val = utmos_model.predict(predictions.audio)
# 3. Matching Rate (Human vs AI consistency)
human_prefs = get_human_pairwise_prefs(config)
ai_scores = gpt_or_rule_based_score(predictions, gold)
matching_rate = sum(1 for h, a in zip(human_prefs, ai_scores) if h == a) / len(human_prefs)
```

## Common pitfalls

- Confusing the basic and pro difficulty tracks, which test different levels of reasoning and audio understanding.
- Overlooking paralinguistic and emotional generation capabilities, which require specific rule-based or TTS-integrated scoring rather than pure text metrics.
- Assuming cascaded ASR+LLM+TTS pipelines are directly comparable to end-to-end SDMs without accounting for error propagation and speech-text alignment differences.

## Evidence (verbatim from paper)

> UTMOS of GPT-4o-Audio-Preview is lower than that of SLAM-Omni (Chen et al., 2024a) and GLM-4-Voice, but it achieves better speech-text alignment.

## Citation

```bibtex
@misc{yan2025urobench,
  title={URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Models},
  author={Yan et al. (2025)},
  year={2025},
  note={arXiv:2502.17810}
}
```

- arXiv: 2502.17810