Stream Omni Eval

---
name: stream-omni-eval
description: Evaluates multimodal capabilities across visual understanding, speech interaction, and vision-grounded speech tasks. It measures accuracy on standard VQA and knowledge-grounded QA benchmarks, and uses LLM-based scoring for open-ended spoken interactions. Use when the user wants to benchmark on VQA-v2, GQA, VizWiz, ScienceQA-IMG, TextVQA, POPE, MME, MMBench, SEED-Bench, LLaVA-Bench-in-the-Wild, MM-Vet, Llama Questions, Web Questions, SpokenVisIT, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.13642
  bibtex_key: zhang2025streamomni
  confidence: high
---

# stream-omni-eval

> Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model — Zhang et al. (2025) (arXiv:2506.13642, 2025)

## What this evaluates

Evaluates multimodal capabilities across visual understanding, speech interaction, and vision-grounded speech tasks. It measures accuracy on standard VQA and knowledge-grounded QA benchmarks, and uses LLM-based scoring for open-ended spoken interactions.

## Datasets

- **VQA-v2** — total ?; splits: test (-1)
- **GQA** — total ?; splits: test (-1)
- **VizWiz** — total ?; splits: test (-1)
- **ScienceQA-IMG** — total ?; splits: test (-1)
- **TextVQA** — total ?; splits: test (-1)
- **POPE** — total ?; splits: test (-1)
- **MME** — total ?; splits: test (-1)
- **MMBench** — total ?; splits: test (-1)
- **SEED-Bench** — total ?; splits: test (-1)
- **LLaVA-Bench-in-the-Wild** — total ?; splits: test (-1)
- **MM-Vet** — total ?; splits: test (-1)
- **Llama Questions** — total ?; splits: test (-1)
- **Web Questions** — total ?; splits: test (-1)
- **SpokenVisIT** — total ?; splits: test (-1); HF `ICTNLP/SpokenVisIT`

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Exact-match or normalized accuracy measuring whether the model's generated response matches the ground-truth answer.
- `GPT-4o score` — range: [1, 5]
  - LLM-as-a-judge score from 1 to 5 assigned by GPT-4o to evaluate response quality on open-ended vision-grounded speech tasks.

## Input / output format

**Input**: Image + text question (vision), audio + text prompt (speech), or image + audio question (vision-grounded speech).

**Output**: Text response for VQA/QA tasks; audio response for speech-to-speech settings.

## Scoring recipe

```python
if dataset in ['Llama Q.', 'Web Q.', 'VQA-v2', 'GQA', 'VizWiz', 'ScienceQA-IMG', 'TextVQA', 'POPE', 'MME', 'MMBench', 'SEED-Bench', 'LLaVA-Bench-in-the-Wild', 'MM-Vet']:
    score = 1.0 if normalize(prediction) == normalize(gold) else 0.0
    metric = mean(score)
elif dataset == 'SpokenVisIT':
    prompt = f'Score the response to the question on a scale of 1-5: Q: {question}, A: {prediction}'
    score = gpt4o_score(prompt)
    metric = mean(score)
return metric
```

## Common pitfalls

- Speech-to-speech evaluations require transcribing generated audio with Whisper-large-v3 before scoring, which can introduce transcription errors.
- SpokenVisIT uses GPT-4o for scoring, which may vary based on prompt wording and model version.
- Vision benchmarks follow LLaVA's specific evaluation scripts; using standard VQA evaluation scripts may yield different results.

## Evidence (verbatim from paper)

> For speech evaluation, we assess the model’s knowledge-grounded speech interaction on spoken question answering benchmarks, Llama Questions (Llama Q.) *[[62]]* and Web Questions (Web Q.) *[[63]]*, where the metric is the accuracy that whether the model’s response matches the ground-truth answer. Following *Fang et al. [[9]]*, the evaluation for SpokenVisIT employs the GPT model (gpt-4o version) to assign a score ranging from 1 to 5 for response.

## Citation

```bibtex
@misc{zhang2025streamomni,
  title={Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2506.13642}
}
```

- arXiv: 2506.13642