S2s Arena Eval

---
name: s2s-arena-eval
description: Evaluates speech-to-speech models on instruction following, assessing both semantic correctness and paralinguistic/speech quality in a reference-free, head-to-head comparison. Use when the user wants to benchmark on S2S-Arena, or asks about evaluating this task. Reports ELO score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.05085
  bibtex_key: jiang2025s2sarena
  confidence: high
---

# s2s-arena-eval

> S2S-Arena, Evaluating Speech2Speech Protocols on Instruction Following with Paralinguistic Information — Feng Jiang et al. (2025) (arXiv:2503.05085, 2025)

## What this evaluates

Evaluates speech-to-speech models on instruction following, assessing both semantic correctness and paralinguistic/speech quality in a reference-free, head-to-head comparison.

## Datasets

- **S2S-Arena** — total ?; splits: test (-1); repo https://github.com/FreedomIntelligence/S2S-Arena

## Metrics

- `ELO score` **(primary)** — range: other
  - Standard ELO rating system for pairwise comparisons. Ratings update after each match based on expected vs. actual outcome: R_new = R_old + K * (S - E), where S is actual score (1 for win, 0 for loss) and E is expected win probability derived from the rating difference.
- `Pairwise win rate` — range: percent
  - Percentage of pairwise comparisons won by a model against another specific model.

## Input / output format

**Input**: Speech audio containing an instruction or query.

**Output**: Speech audio response generated by the model.

## Scoring recipe

```python
ratings = {model: 1000 for model in models}
for pair in comparisons:
    winner, loser = pair.winner, pair.loser
    expected = 1 / (1 + 10**((ratings[loser] - ratings[winner]) / 400))
    ratings[winner] += 32 * (1 - expected)
    ratings[loser] += 32 * (0 - expected)
return ratings
```

## Common pitfalls

- Relying on automatic text-based evaluation (LLM-as-judge) loses critical paralinguistic and speech quality information.
- Speech-based automatic evaluators are unreliable and biased; human arena-style comparison is required.
- ELO scores are relative to the specific pool of models and evaluators, not absolute performance metrics.

## Evidence (verbatim from paper)

> Therefore, we adopt a manual arena-style approach with ELO ranking to more directly and comprehensively evaluate the performance of various speech models. More Details of ELO ranking calculation can be seen in the Appendix[C]. Followed by Chat-Arena, we build a S2S-Arena web-based evaluation tool for evaluators to perform a reference-free comparison. Given a speech as the input, we invite human evaluators to rank two speech outputs generated by different speech models, considering both semantics and speech quality, as shown in Figure[3].

## Citation

```bibtex
@misc{jiang2025s2sarena,
  title={S2S-Arena, Evaluating Speech2Speech Protocols on Instruction Following with Paralinguistic Information},
  author={Feng Jiang et al. (2025)},
  year={2025},
  note={arXiv:2503.05085}
}
```

- arXiv: 2503.05085