Multi Bench Eval

---
name: multi-bench-eval
description: Evaluates the emotional intelligence (EI) capabilities of spoken dialogue models in multi-turn interactive settings. It probes basic emotion understanding, advanced emotion support, paralinguistic analysis, and style inference across both Chinese and English dialogues. Use when the user wants to benchmark on MULTI-Bench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.00850
  bibtex_key: deng2025multibench
  confidence: high
---

# multi-bench-eval

> MULTI-Bench: A Multi-Turn Interactive Benchmark for Assessing Emotional Intelligence ability of Spoken Dialogue Models — Deng et al. (2025) (arXiv:2511.00850, 2025)

## What this evaluates

Evaluates the emotional intelligence (EI) capabilities of spoken dialogue models in multi-turn interactive settings. It probes basic emotion understanding, advanced emotion support, paralinguistic analysis, and style inference across both Chinese and English dialogues.

## Datasets

- **MULTI-Bench** — total 1500; splits: test (1500)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correct predictions on multi-choice tasks (e.g., emotion recognition, style inference, paralinguistic emotion inference). Calculated as the number of correct predictions divided by the total number of instances.
- `LLM-judge score` — range: other
  - Score assigned by Gemini-2.5-Pro (acoustic focus) or DeepSeek (textual focus) to model responses in multi-turn dialogues, assessing emotional accuracy and contextual responsiveness on a continuous scale.

## Input / output format

**Input**: Multi-turn dialogue context (up to 10 turns) containing user prompts and model history, provided as audio and/or text.

**Output**: Model-generated spoken or textual response to the user's latest prompt.

## Scoring recipe

```python
def compute_metrics(predictions, golds, dialogue_turns):
    # Multi-choice tasks
    accuracy = sum(1 for pred, gold in zip(predictions, golds) if pred == gold) / len(golds)
    
    # Interactive dialogue (LLM judges)
    scores = []
    for turn in dialogue_turns:
        gemini_score = judge_gemini.evaluate(turn.response, turn.context, task_type)
        deepseek_score = judge_deepseek.evaluate(turn.response, turn.context, task_type)
        scores.append((gemini_score + deepseek_score) / 2)
    return accuracy, mean(scores)
```

## Common pitfalls

- Multi-choice evaluation tasks consistently yield low accuracy across all systems, with models typically identifying only one correct option per question.
- Models often struggle with maintaining conversational memory across multi-turn interactions, leading to degraded performance in later turns.
- Language mixing occurs frequently in English dialogue tasks for models primarily trained on Chinese data.

## Evidence (verbatim from paper)

> Step Audio 2 leads in emotion recognition with 70.80%. Furthermore, it performs well on reasoning tasks. For example, it reaches 56.14% accuracy in best response style inference compared with 55.29% for Kimi Audio, and 40.00% in paralinguistic emotion inference compared with 35.28% for Kimi Audio.

## Citation

```bibtex
@misc{deng2025multibench,
  title={MULTI-Bench: A Multi-Turn Interactive Benchmark for Assessing Emotional Intelligence ability of Spoken Dialogue Models},
  author={Deng et al. (2025)},
  year={2025},
  note={arXiv:2511.00850}
}
```

- arXiv: 2511.00850