Omnivideobench Eval

---
name: omnivideobench-eval
description: Evaluates multimodal large language models' ability to jointly reason across visual and audio modalities in long-duration videos. It probes capabilities like cross-modal alignment, temporal dependency modeling, and understanding of low-semantic acoustic cues such as music and ambient sounds. Use when the user wants to benchmark on OmniVideoBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.10689
  bibtex_key: li2025omnivideobench
  confidence: high
---

# omnivideobench-eval

> OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs — Li et al. (2025) (arXiv:2510.10689, 2025)

## What this evaluates

Evaluates multimodal large language models' ability to jointly reason across visual and audio modalities in long-duration videos. It probes capabilities like cross-modal alignment, temporal dependency modeling, and understanding of low-semantic acoustic cues such as music and ambient sounds.

## Datasets

- **OmniVideoBench** — total 1000; splits: test (1000)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Standard classification accuracy: the proportion of correctly answered questions out of the total number of questions. Calculated separately for multiple-choice questions (MCQ) and open-ended question-answering (QA).

## Input / output format

**Input**: Long-duration videos (up to 30 minutes) with synchronized audio, accompanied by a natural language question. Inputs may be provided as raw video frames + audio, silent frames + ASR transcripts, or varying frame counts (e.g., 32, 64, 128, 256 frames).

**Output**: For MCQ: a single selected option letter/text. For open-ended QA: a free-form textual response.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if pred.strip().lower() == gold.strip().lower():
            correct += 1
    return (correct / len(gold_answers)) * 100
```

## Common pitfalls

- Multiple-choice question (MCQ) formats significantly overstate model performance compared to open-ended QA due to lexical cues and guessing opportunities.
- ASR transcripts cannot replace raw audio for tasks requiring deep acoustic comprehension (e.g., music or ambient sound understanding).
- Frame sampling density heavily influences results; sparse sampling misses key temporal events, especially in videos longer than 5 minutes.

## Evidence (verbatim from paper)

> To investigate whether the multiple-choice question (MCQ) format overstates model performance, we additionally evaluated several representative models on open-ended question-answering (QA) tasks, where no predefined answer options are provided. In this setting, models must directly generate textual responses, eliminating both the possibility of random guessing and any lexical cues potentially present in candidate options. In Table[4], the accuracy of all models drops significantly compared to their performance on multiple-choice questions.

## Citation

```bibtex
@misc{li2025omnivideobench,
  title={OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2510.10689}
}
```

- arXiv: 2510.10689