Emoval Emotion Dialogue Eval

---
name: emoval-emotion-dialogue-eval
description: Evaluates an omni-modal language model's ability to engage in end-to-end spoken dialogue with vivid emotional control. It probes the model's dialogue quality, text generation accuracy under different input modalities, and its capability to control and classify speech styles/emotions. Use when the user wants to benchmark on EMOVA-EmotionDialogue-Test, or asks about evaluating this task. Reports end-to-end spoken dialogue score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.18042
  bibtex_key: chen2024emova
  confidence: high
---

# emoval-emotion-dialogue-eval

> EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions — Kai Chen et al. (2024) (arXiv:2409.18042, 2024)

## What this evaluates

Evaluates an omni-modal language model's ability to engage in end-to-end spoken dialogue with vivid emotional control. It probes the model's dialogue quality, text generation accuracy under different input modalities, and its capability to control and classify speech styles/emotions.

## Datasets

- **EMOVA-EmotionDialogue-Test** — total ?; splits: test (-1)

## Metrics

- `end-to-end spoken dialogue score` **(primary)** — range: [0, 10]
  - Average score from 0 to 10 assigned by GPT-4o to assess dialogue performance based on generated speech quality and emotional expression.
- `unit-input-text-output score` — range: [0, 10]
  - Score assessing the quality of textual responses when the model receives speech units as input, bypassing speech synthesis errors.
- `text-input-text-output score` — range: [0, 10]
  - Score assessing textual response quality when the model receives ground-truth user instruction texts as input.
- `style label classification accuracy` — range: [0, 1]
  - Percentage of correctly predicted style/emotion labels from the model's output compared to ground truth.
- `style controllability` — range: other
  - Measured using a confusion matrix comparing conditional style labels provided to the detokenizer against recognized style labels in the generated speech.

## Input / output format

**Input**: Input image and user instructions provided as speech units (or ground-truth text for text-input variant).

**Output**: Text responses, style labels, and corresponding speech units.

## Scoring recipe

```python
def evaluate(predictions, gold):
    # LLM-as-judge for dialogue scores (0-10)
    scores = [gpt4o_judge(f"Rate dialogue 0-10: {inp}, {pred}") for inp, pred in zip(inputs, predictions)]
    avg_score = sum(scores) / len(scores)
    
    # Style classification accuracy
    correct = sum(1 for p, g in zip(predictions['style_labels'], gold['style_labels']) if p == g)
    accuracy = correct / len(gold['style_labels'])
    
    # Controllability via confusion matrix
    confusion = compute_confusion_matrix(predictions['generated_styles'], gold['recognized_styles'])
    return avg_score, accuracy, confusion
```

## Common pitfalls

- The evaluation relies entirely on GPT-4o for automated scoring, which may introduce judge bias or inconsistency compared to human evaluation.
- ASR/TTS errors in the end-to-end pipeline can unfairly penalize the dialogue score; the protocol explicitly disentangles this by comparing unit-input vs text-input scores.
- The test set is split from the model's own synthesized instruction-tuning data, raising potential data leakage or distribution shift concerns.

## Evidence (verbatim from paper)

> To ensure comprehensive evaluation, we propose the following evaluation metrics: 1. End-to-end spoken dialogue score assesses the model’s dialogue performance based on the generated speeches, with a score ranging from 0 to 10, reporting the average. ... Due to the lack of emotionally rich spoken dialogue evaluation datasets, we split a test set from our synthesized omni-modal instruction-tuning data (Sec.[4.1]). GPT-4o is used for automated evaluation.

## Citation

```bibtex
@misc{chen2024emova,
  title={EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions},
  author={Kai Chen et al. (2024)},
  year={2024},
  note={arXiv:2409.18042}
}
```

- arXiv: 2409.18042