Audiorole Eval

---
name: audiorole-eval
description: Evaluates a model's ability to generate audio responses that simultaneously maintain contextual semantic appropriateness and preserve the specific acoustic identity (timbre, prosody) of a target character given reference audio. Use when the user wants to benchmark on AudioRole-Demo, or asks about evaluating this task. Reports Acoustic Personalization (AP).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.23435
  bibtex_key: li2025audiorole
  confidence: high
---

# audiorole-eval

> AudioRole: An Audio Dataset for Character Role-Playing in Large Language Models — Li et al. (2025) (arXiv:2509.23435, 2025)

## What this evaluates

Evaluates a model's ability to generate audio responses that simultaneously maintain contextual semantic appropriateness and preserve the specific acoustic identity (timbre, prosody) of a target character given reference audio.

## Datasets

- **AudioRole-Demo** — total 11328; splits: train (10704), test (624)

## Metrics

- `Acoustic Quality (AQ)` — range: [0, 10]
  - Score from 0 to 10 computed by Audiobox’s pre-trained aesthetic scoring model, quantifying signal-to-noise ratio, harmonic-to-noise ratio, and spectral flatness.
- `Content Quality (CQ)` — range: [0, 2]
  - Score from 0 to 2 obtained by transcribing the output audio with whisper-turbo and evaluating semantic alignment/contextual appropriateness using GPT-4o.
- `Acoustic Personalization (AP)` **(primary)** — range: other
  - Cosine distance $D_{\text{cos}}$ between speaker embeddings of the reference audio and synthesized output, extracted via PyAnnotate’s wespeaker-voxceleb-resnet34-LM model. Lower values indicate better preservation.
- `Content Personalization (CP)` — range: [0, 2]
  - Score from 0 to 2 generated by GPT-4o-audio, which analyzes whether the synthesized audio and reference audio exhibit the same character’s stylistic consistency.

## Input / output format

**Input**: Reference audio of the target character ($X_c$) and an input audio prompt ($X_a$).

**Output**: Synthesized audio response ($X_b$) matching the prompt and character identity.

## Scoring recipe

```python
def evaluate(X_c, X_a, X_b):
    aq = audiobox_aesthetic_score(X_b)  # [0, 10]
    text = whisper_turbo.transcribe(X_b)
    cq = gpt4o_semantic_alignment_score(text, context)  # [0, 2]
    ref_emb = pyannote_wespeaker_embedding(X_c)
    gen_emb = pyannote_wespeaker_embedding(X_b)
    ap = cosine_distance(ref_emb, gen_emb)  # lower is better
    cp = gpt4o_audio_style_score(X_b, X_c)  # [0, 2]
    return {'AQ': aq, 'CQ': cq, 'AP': ap, 'CP': cp}
```

## Common pitfalls

- Acoustic Personalization (AP) reports cosine distance where lower values indicate better performance, which is the inverse of typical similarity scores.
- Content Quality (CQ) and Content Personalization (CP) rely on LLM judges (GPT-4o), making results sensitive to prompt phrasing and non-deterministic sampling.
- The test split is highly imbalanced across characters (e.g., 22 turns for Tyrion vs. 250 for Sheldon), so macro-averaging may skew overall results.

## Evidence (verbatim from paper)

> Acoustic Personalization (AP) quantifies voice characteristic preservation through PyAnnotate’s speaker embedding model111111[https://huggingface.co/pyannote/wespeaker-voxceleb-resnet34-LM ""]. Given reference audio samples $X_{c}$ of $C$ and synthesized $X_{b}$, we calculate voice similarity as $D_{\text{cos}}$. Lower $D_{\text{cos}}$ values indicate better preservation of $C$’s vocal timbre and prosodic patterns. Content Personalization (CP) evaluates stylistic consistency using GPT-4o-audio multi-modal reasoning ability. The model takes in $X_{b}$ and a reference audio sample $X_{c}$, and a prompt to analyze whether they show the same character’s style in a range of 0 to 2.

## Citation

```bibtex
@misc{li2025audiorole,
  title={AudioRole: An Audio Dataset for Character Role-Playing in Large Language Models},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2509.23435}
}
```

- arXiv: 2509.23435