Speech Drame Realism Eval

---
name: speech-drame-realism-eval
description: Evaluates a speech foundation model's ability to generate role-play responses grounded in bottom-up, human-grounded realism. It measures fine-grained aspects such as prosodic dynamics, emotional fidelity, character consistency, and contextual fit based on realistic dialogue and media sources. Use when the user wants to benchmark on DRAME-RoleBench (Realism), or asks about evaluating this task. Reports realism_score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.01261
  bibtex_key: shi2025speechdrame
  confidence: medium
---

# speech-drame-realism-eval

> Speech-DRAME: A Framework for Human-Aligned Benchmarks in Speech Role-Play — Jiatong Shi et al. (2025) (arXiv:2511.01261, 2025)

## What this evaluates

Evaluates a speech foundation model's ability to generate role-play responses grounded in bottom-up, human-grounded realism. It measures fine-grained aspects such as prosodic dynamics, emotional fidelity, character consistency, and contextual fit based on realistic dialogue and media sources.

## Datasets

- **DRAME-RoleBench (Realism)** — total ?; splits: test (-1); repo https://github.com/Anuttacon/speech_drame

## Metrics

- `realism_score` **(primary)** — range: [0, 1]
  - A multi-dimensional scalar score assessing prosodic dynamics, emotional fidelity, character consistency, and contextual fit. Aggregated scores are correlated with human annotations using Pearson correlation to measure human alignment.

## Input / output format

**Input**: Character profile ($\bm{C}_{\text{profile}}$), scene specification ($\bm{C}_{\text{scene}}$), and optionally preceding dialogue context ($\bm{C}_{\text{speech}}$).

**Output**: Generated speech response ($\bm{S}_r$), represented as a waveform or latent representation.

## Scoring recipe

```python
# For each instance in the benchmark:
response = model.generate(profile, scene, context)
score = sem.evaluate(response, profile, scene)
# Aggregate scores across the dataset
human_scores = get_human_annotations(instance)
pearson_corr = pearsonr(score, human_scores)
return pearson_corr
```

## Common pitfalls

- Requires careful curation and more human annotation resources compared to archetype-based evaluation.
- Judgments are multi-dimensional, making it harder to isolate specific paralinguistic failures without detailed error analysis.
- Single-turn evaluation may not fully capture long-form conversational realism.

## Evidence (verbatim from paper)

> The second strategy adopts a bottom-up design, motivated by real-world performance and real human speech. Instead of relying solely on stereotypes, this approach grounds evaluation in recordings from professional and non-professional speakers, drawn from realistic dialogue and media sources. Judgments in this setting are more fine-grained and multi-dimensional, addressing aspects such as prosodic dynamics, emotional fidelity, character consistency, and contextual fit.

## Citation

```bibtex
@misc{shi2025speechdrame,
  title={Speech-DRAME: A Framework for Human-Aligned Benchmarks in Speech Role-Play},
  author={Jiatong Shi et al. (2025)},
  year={2025},
  note={arXiv:2511.01261}
}
```

- arXiv: 2511.01261