Paras2s Eval

---
name: paras2s-eval
description: Evaluates how well speech-to-speech models adapt to and reflect paralinguistic styles (age, emotion, gender, sarcasm) in spoken responses. It measures both content appropriateness and stylistic alignment against ground-truth or human-annotated references. Use when the user wants to benchmark on ParaS2SBench, IEMOCAP, MELD, or asks about evaluating this task. Reports ParaS2SBench score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.08723
  bibtex_key: yang2025paras2s
  confidence: high
---

# paras2s-eval

> ParaS2S: Benchmarking and Aligning Spoken Language Models for Paralinguistic-aware Speech-to-Speech Interaction — Shu-wen Yang et al. (2025) (arXiv:2511.08723, 2025)

## What this evaluates

Evaluates how well speech-to-speech models adapt to and reflect paralinguistic styles (age, emotion, gender, sarcasm) in spoken responses. It measures both content appropriateness and stylistic alignment against ground-truth or human-annotated references.

## Datasets

- **ParaS2SBench** — total ?; splits: test (-1)
- **IEMOCAP** — total ?; splits: test (-1)
- **MELD** — total ?; splits: test (-1)

## Metrics

- `ParaS2SBench score` **(primary)** — range: [1, 5]
  - Average Likert-scale score (1–5) assigned to each prompt–response pair across four paralinguistic categories: Age, Emotion, Gender, and Sarcasm. Scores are generated by either human experts or an automatic GPT-based judge.

## Input / output format

**Input**: Audio speech prompts containing specific paralinguistic styles (age, emotion, gender, sarcasm).

**Output**: Audio speech waveform representing the model's spoken response.

## Scoring recipe

```python
scores = []
for prompt, response in dataset:
    s_age = judge_score(prompt, response, "Age")
    s_emotion = judge_score(prompt, response, "Emotion")
    s_gender = judge_score(prompt, response, "Gender")
    s_sarcasm = judge_score(prompt, response, "Sarcasm")
    scores.append((s_age + s_emotion + s_gender + s_sarcasm) / 4)
return sum(scores) / len(scores)
```

## Common pitfalls

- Models frequently produce 'tone-deaf' responses to contrasting styles, causing scores to average around 3.0 even when one response is highly appropriate.
- Automatic GPT-based scoring correlates strongly with human evaluation (>0.7), making human annotation unnecessary for routine benchmarking but critical only for validation.
- Comparing SFT and RL requires careful budget allocation; RL consistently outperforms SFT even when SFT uses 10x more data, so raw data volume comparisons are misleading.

## Evidence (verbatim from paper)

> The ParaS2SBench score is the average across 4 categories. ... Each prompt–response pair is scored by three human experts on a Likert scale... We also apply automatic scoring to study alignment.

## Citation

```bibtex
@misc{yang2025paras2s,
  title={ParaS2S: Benchmarking and Aligning Spoken Language Models for Paralinguistic-aware Speech-to-Speech Interaction},
  author={Shu-wen Yang et al. (2025)},
  year={2025},
  note={arXiv:2511.08723}
}
```

- arXiv: 2511.08723