Speech Drame Archetype Eval

---
name: speech-drame-archetype-eval
description: Evaluates a speech foundation model's ability to generate role-play responses that align with top-down, stereotype-driven character archetypes within specific scene contexts. It measures how well the model captures broad, accessible scoring criteria and general impressions of character consistency without relying on fine-grained prosodic nuance. Use when the user wants to benchmark on DRAME-RoleBench (Archetype), or asks about evaluating this task. Reports archetype_score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.01261
  bibtex_key: shi2025speechdrame
  confidence: medium
---

# speech-drame-archetype-eval

> Speech-DRAME: A Framework for Human-Aligned Benchmarks in Speech Role-Play — Jiatong Shi et al. (2025) (arXiv:2511.01261, 2025)

## What this evaluates

Evaluates a speech foundation model's ability to generate role-play responses that align with top-down, stereotype-driven character archetypes within specific scene contexts. It measures how well the model captures broad, accessible scoring criteria and general impressions of character consistency without relying on fine-grained prosodic nuance.

## Datasets

- **DRAME-RoleBench (Archetype)** — total ?; splits: test (-1); repo https://github.com/Anuttacon/speech_drame

## Metrics

- `archetype_score` **(primary)** — range: [0, 1]
  - A scalar score reflecting alignment with stereotypical archetypes and scene contexts. Model scores are aggregated and compared to human annotations using Pearson correlation to measure human alignment.

## Input / output format

**Input**: Character profile ($\bm{C}_{\text{profile}}$), scene specification ($\bm{C}_{\text{scene}}$), and optionally preceding dialogue context ($\bm{C}_{\text{speech}}$).

**Output**: Generated speech response ($\bm{S}_r$), represented as a waveform or latent representation.

## Scoring recipe

```python
# For each instance in the benchmark:
response = model.generate(profile, scene, context)
score = sem.evaluate(response, profile, scene)
# Aggregate scores across the dataset
human_scores = get_human_annotations(instance)
pearson_corr = pearsonr(score, human_scores)
return pearson_corr
```

## Common pitfalls

- Relies on broad stereotypes which may limit fine-grained prosodic or emotional nuance capture.
- Single-turn formulation ignores multi-turn dialogue context aggregation, which may skew long-conversation evaluations.
- Human alignment correlation (Pearson) is reported as a secondary metric, not the direct generation score.

## Evidence (verbatim from paper)

> The first strategy follows a top-down design, inspired by prior text-based role-play benchmarks. Here, role-play is judged with respect to stereotypical archetypes (e.g., “firefighter” or “ER doctor”), using scene contexts (e.g., “comfort a child trapped in fire” or “ask for help in a serious operation”). This approach provides accessible, general-purpose scoring that is scalable to a wide range of scenarios, making it suitable for large-scale benchmarking.

## Citation

```bibtex
@misc{shi2025speechdrame,
  title={Speech-DRAME: A Framework for Human-Aligned Benchmarks in Speech Role-Play},
  author={Jiatong Shi et al. (2025)},
  year={2025},
  note={arXiv:2511.01261}
}
```

- arXiv: 2511.01261