Mvbench Eval

---
name: mvbench-eval
description: Evaluates multi-modal large language models' ability to understand video content, with a strong focus on temporal perception and static-to-dynamic task transformation across 20 diverse categories ranging from basic perception to complex reasoning. Use when the user wants to benchmark on MVBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2311.17005
  bibtex_key: li2023mvbench
  confidence: high
---

# mvbench-eval

> MVBench: A Comprehensive Multi-modal Video Understanding Benchmark — Kunchang Li et al. (2023) (arXiv:2311.17005, 2023)

## What this evaluates

Evaluates multi-modal large language models' ability to understand video content, with a strong focus on temporal perception and static-to-dynamic task transformation across 20 diverse categories ranging from basic perception to complex reasoning.

## Datasets

- **MVBench** — total ?; splits: test (-1); repo https://github.com/OpenGVLab/Ask-Anything

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted answers across all test instances. Calculated as (number of correct predictions / total number of predictions) * 100.

## Input / output format

**Input**: 16-frame video clips paired with multiple-choice questions and system prompts.

**Output**: Text response containing the selected option (e.g., 'Best option: (A)') or direct answer string.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        if normalize_answer(pred) == normalize_answer(gold):
            correct += 1
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Using 4-frame inputs instead of the specified 16-frame evaluation setting.
- Failing to use the explicit answer prompt ('Best option: (') which significantly boosts option extraction precision.
- Confusing training instruction data sizes (e.g., 1.1M) with the test set size.
- Evaluating on static image tasks without applying the static-to-dynamic transformation methodology.

## Evidence (verbatim from paper)

> For evaluation, we input 16-frame videos with elaborate prompts for better results. VideoChat [42], a top-performing video MLLM, only marginally surpasses VideoChat2_text by 0.8% in average accuracy (35.5% vs. 34.7%), with the latter generating responses from text alone.

## Citation

```bibtex
@misc{li2023mvbench,
  title={MVBench: A Comprehensive Multi-modal Video Understanding Benchmark},
  author={Kunchang Li et al. (2023)},
  year={2023},
  note={arXiv:2311.17005}
}
```

- arXiv: 2311.17005