Ultraeval Audio Eval

---
name: ultraeval-audio-eval
description: Evaluates audio foundation models across understanding, generation, and codec capabilities. It probes semantic accuracy, timbre fidelity, acoustic quality, and multilingual speech comprehension using a unified taxonomy and standardized benchmarks. Use when the user wants to benchmark on SpeechCMMLU, SpeechHSK, LibriSpeech, AISHELL-1, or asks about evaluating this task. Reports WER.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.01373
  bibtex_key: shi2026ultraevalaudio
  confidence: high
---

# ultraeval-audio-eval

> UltraEval-Audio: A Unified Framework for Comprehensive Evaluation of Audio Foundation Models — Shi et al. (2026) (arXiv:2601.01373, 2026)

## What this evaluates

Evaluates audio foundation models across understanding, generation, and codec capabilities. It probes semantic accuracy, timbre fidelity, acoustic quality, and multilingual speech comprehension using a unified taxonomy and standardized benchmarks.

## Datasets

- **SpeechCMMLU** — total 3519; splits: test (3519); repo https://github.com/OpenBMB/UltraEval-Audio
- **SpeechHSK** — total 170; splits: test (170); repo https://github.com/OpenBMB/UltraEval-Audio
- **LibriSpeech** — total ?; splits: test (-1)
- **AISHELL-1** — total ?; splits: test (-1)

## Metrics

- `WER` **(primary)** — range: [0, 1]
  - Word Error Rate: minimum number of word edits (insertions, deletions, substitutions) required to change the predicted transcript to the reference transcript, normalized by reference length.
- `ASR-WER` — range: [0, 1]
  - Computed by transcribing generated or reconstructed audio with a high-performance ASR model (e.g., Whisper-large-v3) and calculating WER on the transcriptions.
- `SIM` — range: [-1, 1]
  - Speaker embedding cosine similarity: cosine distance between WavLM-large embeddings of original and reconstructed audio to measure timbre fidelity.
- `UTMOS` — range: [1, 5]
  - Objective metric predicting overall speech naturalness and perceptual comfort.

## Input / output format

**Input**: Audio files (speech, music, or environmental sounds) or text prompts, depending on the task (understanding, generation, or codec).

**Output**: Text transcriptions, translations, or answers for understanding tasks; synthesized or reconstructed audio files for generation/codec tasks.

## Scoring recipe

```python
def compute_wer(pred_text, ref_text):
    edits = levenshtein_distance(pred_text.split(), ref_text.split())
    return edits / max(len(ref_text.split()), 1)

def compute_sim(orig_audio, recon_audio):
    emb_orig = wavlm_large.encode(orig_audio)
    emb_recon = wavlm_large.encode(recon_audio)
    return cosine_similarity(emb_orig, emb_recon)

def compute_utmos(audio):
    return utmos_model.predict(audio)
```

## Common pitfalls

- Inconsistent use of ASR models for computing ASR-WER across different benchmarks can skew semantic accuracy comparisons.
- Codec evaluation metrics (WER, SIM, UTMOS) measure different dimensions; relying solely on WER ignores timbre and acoustic quality degradation.
- TTS synthesis artifacts in SpeechCMMLU may introduce pronunciation errors if CER filtering is not strictly applied during dataset construction.

## Evidence (verbatim from paper)

> For semantics, we measure how well reconstructed audio preserves the original content using WER. Specifically, the reconstructed audio is transcribed by high-performance ASR models and compared to the original transcript. We employ Whisper-large-v3 for English and Paraformer-zh for Chinese. For timbre fidelity, we extract audio embeddings using WavLM-large fine-tuned on speaker verification, and compute the cosine similarity between embeddings of the original and reconstructed audio.

## Citation

```bibtex
@misc{shi2026ultraevalaudio,
  title={UltraEval-Audio: A Unified Framework for Comprehensive Evaluation of Audio Foundation Models},
  author={Shi et al. (2026)},
  year={2026},
  note={arXiv:2601.01373}
}
```

- arXiv: 2601.01373