Mavors Video Image Eval

---
name: mavors-video-image-eval
description: Evaluates multimodal large language models on video and image understanding tasks, including general knowledge QA, long-video QA, event understanding, temporal reasoning, and captioning, as well as image QA, cognitive understanding, and captioning. Use when the user wants to benchmark on MMWorld, PerceptionTest, Video-MME, MLVU, MVBench, EventHallusion, TempCompass, VinoGround, DREAM-1K, MMMU, MathVista, AI2D, CapsBench, or asks about evaluating this task. Reports score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.10068
  bibtex_key: shi2025mavors
  confidence: high
---

# mavors-video-image-eval

> Mavors: Multi-granularity Video Representation for Multimodal Large Language Model — Shi et al. (2025) (arXiv:2504.10068, 2025)

## What this evaluates

Evaluates multimodal large language models on video and image understanding tasks, including general knowledge QA, long-video QA, event understanding, temporal reasoning, and captioning, as well as image QA, cognitive understanding, and captioning.

## Datasets

- **MMWorld** — total ?; splits: test (-1)
- **PerceptionTest** — total ?; splits: test (-1)
- **Video-MME** — total ?; splits: test (-1)
- **MLVU** — total ?; splits: test (-1)
- **MVBench** — total ?; splits: test (-1)
- **EventHallusion** — total ?; splits: test (-1)
- **TempCompass** — total ?; splits: test (-1)
- **VinoGround** — total ?; splits: test (-1)
- **DREAM-1K** — total ?; splits: test (-1)
- **MMMU** — total ?; splits: test (-1)
- **MathVista** — total ?; splits: test (-1)
- **AI2D** — total ?; splits: test (-1)
- **CapsBench** — total ?; splits: test (-1)

## Metrics

- `score` **(primary)** — range: percent
  - Average of task-specific scores (typically accuracy or normalized performance) across all tasks within a benchmark. Reported as a percentage.

## Input / output format

**Input**: Video frames (processed in chunks of 16) or single images paired with text prompts/questions.

**Output**: Text responses (multiple-choice answers or descriptive captions).

## Scoring recipe

```python
def compute_score(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if is_mcq(gold):
            correct += 1 if normalize(pred) == normalize(gold) else 0
        else:
            correct += caption_metric(pred, gold)
    return (correct / len(golds)) * 100
```

## Common pitfalls

- Scores for most baseline models are taken directly from their original papers rather than reproduced, which may introduce inconsistencies due to different evaluation settings or versions.
- No benchmark-specific hyperparameters (e.g., frame sampling rate, resolution) were tuned during evaluation, meaning models are compared under a uniform but potentially suboptimal setting for each specific dataset.
- Video benchmarks cover diverse capabilities (long-video, event, temporal) that require different evaluation focuses, so a single average score may mask strengths/weaknesses in specific temporal reasoning tasks.

## Evidence (verbatim from paper)

> Video understanding capabilities are assessed across general knowledge QA (MMWorld, PerceptionTest), long-video QA (Video-MME, MLVU), event understanding QA (MVBench, EventHallusion), temporal understanding QA (TempCompass, VinoGround), and captioning (DREAM-1K). Image understanding evaluation includes comprehensive capabilities (MMMU), cognitive understanding (MathVista, AI2D), and captioning (CapsBench). Crucially, aside from prompt modifications, no benchmark-specific hyperparameters (e.g., frame sampling, resolution) were tuned during evaluation for any model, including Mavors.

## Citation

```bibtex
@misc{shi2025mavors,
  title={Mavors: Multi-granularity Video Representation for Multimodal Large Language Model},
  author={Shi et al. (2025)},
  year={2025},
  note={arXiv:2504.10068}
}
```

- arXiv: 2504.10068