Humdial Eibench Eval

---
name: humdial-eibench-eval
description: Evaluates the emotional intelligence of audio language models across multi-turn dialogues. It probes four core capabilities: tracking emotional trajectories over time, reasoning about implicit emotional causes, generating empathetic responses, and resolving conflicts between acoustic and textual emotional signals. Use when the user wants to benchmark on HumDial-EIBench, or asks about evaluating this task. Reports Accuracy (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.11594
  bibtex_key: wang2026humdialeibench
  confidence: high
---

# humdial-eibench-eval

> HumDial-EIBench: A Human-Recorded Multi-Turn Emotional Intelligence Benchmark for Audio Language Models — Wang et al. (2026) (arXiv:2604.11594, 2026)

## What this evaluates

Evaluates the emotional intelligence of audio language models across multi-turn dialogues. It probes four core capabilities: tracking emotional trajectories over time, reasoning about implicit emotional causes, generating empathetic responses, and resolving conflicts between acoustic and textual emotional signals.

## Datasets

- **HumDial-EIBench** — total ?; splits: test (-1); repo https://github.com/ASLP-lab/HumDial-EIBench

## Metrics

- `Accuracy (%)` **(primary)** — range: percent
  - Percentage of correctly selected options in multiple-choice tasks (Tasks 1, 2, and 4). Calculated as (number of correct predictions / total samples) * 100.
- `Empathy/Naturalness Score` — range: [1, 5]
  - Average rating on a 1-5 scale for text empathy (judged by LLMs), acoustic empathy, and response naturalness (judged by humans) in open-ended generation tasks (Task 3).

## Input / output format

**Input**: Multi-turn audio-text dialogues. For Tasks 1, 2, and 4, the model receives context audio/text and multiple-choice options. For Task 3, the model receives conversational prompts and must generate a response.

**Output**: For Tasks 1, 2, and 4: a single selected multiple-choice option. For Task 3: a generated audio/text response.

## Scoring recipe

```python
# Tasks 1, 2, 4 (Multiple Choice)
correct = sum(1 for pred, gold in zip(predictions, golds) if pred == gold)
accuracy = (correct / len(golds)) * 100

# Task 3 (Open-ended)
# LLM judges score text empathy (D1) on a scale (e.g., 1-5)
# Human annotators score acoustic empathy (D2) and naturalness (D3) on a scale (e.g., 1-5)
empathy_score = (llm_judge_score + human_acoustic_score + human_naturalness_score) / 3
```

## Common pitfalls

- Text-dominance bias: Models heavily over-rely on literal text semantics when acoustic and textual signals contradict, causing severe accuracy drops on conflict samples.
- LLM-as-a-judge instability: Automated text empathy scoring shows high variance (up to 1.0 point difference between judges), making it unreliable without human calibration.
- Cross-lingual interference: Models may generate responses in the wrong language (e.g., Chinese to English prompts), artificially deflating empathy scores.

## Evidence (verbatim from paper)

> Table 3: Accuracy (%) on Tasks 1 and 2. Bold: best in column; underline: second best.

## Citation

```bibtex
@misc{wang2026humdialeibench,
  title={HumDial-EIBench: A Human-Recorded Multi-Turn Emotional Intelligence Benchmark for Audio Language Models},
  author={Wang et al. (2026)},
  year={2026},
  note={arXiv:2604.11594}
}
```

- arXiv: 2604.11594