Multivox Eval

---
name: multivox-eval
description: Evaluates voice assistants' ability to jointly ground visual and paralinguistic speech cues (e.g., pitch, emotion, volume, background sounds) in context-aware responses. It specifically tests robustness against confounding samples that flip speech properties to prevent overreliance on unimodal priors. Use when the user wants to benchmark on MultiVox, or asks about evaluating this task. Reports visual grounding and non-verbal speech signals.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.10859
  bibtex_key: selvakumar2025multivox
  confidence: medium
---

# multivox-eval

> MultiVox: A Benchmark for Evaluating Voice Assistants for Multimodal Interactions — Selvakumar et al. (2025) (arXiv:2507.10859, 2025)

## What this evaluates

Evaluates voice assistants' ability to jointly ground visual and paralinguistic speech cues (e.g., pitch, emotion, volume, background sounds) in context-aware responses. It specifically tests robustness against confounding samples that flip speech properties to prevent overreliance on unimodal priors.

## Datasets

- **MultiVox** — total 1000; splits: test (-1)

## Metrics

- `visual grounding and non-verbal speech signals` **(primary)** — range: other
  - Not explicitly defined in the provided text. The benchmark evaluates performance on jointly grounding visual and paralinguistic speech cues in context-aware responses.

## Input / output format

**Input**: Speech-video or speech-image pairs paired with human-annotated dialogue turns.

**Output**: Context-aware responses from the evaluated voice assistant.

## Scoring recipe

```python
# Scoring recipe not specified in the provided text.
# Based on the benchmark description, evaluation likely involves:
# 1. Feed speech-video or speech-image pairs to the OVA.
# 2. Collect the assistant's context-aware response.
# 3. Assess response quality on visual grounding and non-verbal speech signal interpretation.
# 4. Compare against human-annotated dialogues or rubric-based scoring.
# Exact metrics and formulas are not included in this excerpt.
```

## Common pitfalls

- Models may overrely on unimodal visual priors if confounding speech samples are not properly handled.
- Persistent failure in interpreting non-verbal speech signals (pitch, emotion, volume, background sounds) despite advances in multimodal reasoning.

## Evidence (verbatim from paper)

> MultiVox introduces the first human-spoken, multimodal benchmark for omni-modal voice assistants (OVAs), evaluating their ability to jointly ground visual and paralinguistic speech cues (e.g., pitch, emotion, volume, background sounds) in context-aware responses. It features 1,000 human-annotated dialogues with speech-video or speech-image pairs, including confounding samples that flip speech properties to prevent model overreliance on unimodal priors. Evaluation of 9 state-of-the-art OLMs reveals strong visual grounding but persistent failure in interpreting non-verbal speech signals, exposing a critical gap in speech understanding despite advances in multimodal reasoning.

## Citation

```bibtex
@misc{selvakumar2025multivox,
  title={MultiVox: A Benchmark for Evaluating Voice Assistants for Multimodal Interactions},
  author={Selvakumar et al. (2025)},
  year={2025},
  note={arXiv:2507.10859}
}
```

- arXiv: 2507.10859