Mm Vet V2 Eval

---
name: mm-vet-v2-eval
description: This benchmark evaluates large multimodal models on integrated vision-language capabilities, with a specific focus on sequential image-text understanding, spatial reasoning, knowledge retrieval, and long-form generation. It probes how well models can process interleaved visual and textual inputs to answer complex, real-world questions. Use when the user wants to benchmark on MM-Vet v2, or asks about evaluating this task. Reports MM-Vet-v2 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.00765
  bibtex_key: yu2024mmvetv2
  confidence: high
---

# mm-vet-v2-eval

> MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities — Weihao Yu et al. (arXiv:2408.00765, 2024)

## What this evaluates

This benchmark evaluates large multimodal models on integrated vision-language capabilities, with a specific focus on sequential image-text understanding, spatial reasoning, knowledge retrieval, and long-form generation. It probes how well models can process interleaved visual and textual inputs to answer complex, real-world questions.

## Datasets

- **MM-Vet v2** — total 517; splits: test (517); repo https://github.com/yuweihao/MM-Vet

## Metrics

- `MM-Vet-v2 score` **(primary)** — range: percent
  - An LLM-as-a-judge metric where GPT-4 scores model outputs on a 0–1 scale based on a prompt containing the question, ground truth, and model prediction. The final score is the average of 5 independent GPT-4 evaluations, reported as a percentage out of 100.

## Input / output format

**Input**: Interleaved image-text sequences paired with open-ended questions or prompts requiring multimodal reasoning.

**Output**: Free-form text responses generated by the LMM to answer the provided questions.

## Scoring recipe

```python
scores = []
for _ in range(5):
    judge_prompt = build_prompt(question, ground_truth, model_output)
    raw_score = gpt4_judge(judge_prompt) # returns float in [0, 1]
    scores.append(raw_score)
final_score = (sum(scores) / len(scores)) * 100
```

## Common pitfalls

- LLM-as-a-judge scoring exhibits inherent variance even at temperature 0, necessitating multiple runs (5) and reporting of standard deviation.
- The judge prompt includes the ground truth, which may bias GPT-4 toward leniency or specific phrasing rather than evaluating semantic equivalence.
- Sequential image-text understanding requires models to handle interleaved modalities, which standard single-image VLMs often fail at without explicit architectural support.

## Evidence (verbatim from paper)

> As illustrated in [1], for each sample, we complete the prompt template with its question, ground truth, and the output from a specific LMM. When this filled prompt is input into GPT-4, it generates a score ranging from 0 to 1 for each sample. Although the temperature is set to 0, we observe some variance in GPT-4’s outputs. To address this, we evaluate the outputs of the LLMs using GPT-4 five times. Due to space constraints, we present the average scores for capabilities and for some capability integrations, and include both the average and variance for the total score.

## Citation

```bibtex
@misc{yu2024mmvetv2,
  title={MM-Vet v2: A Challenging Benchmark to Evaluate Large Multimodal Models for Integrated Capabilities},
  author={Weihao Yu et al.},
  year={2024},
  note={arXiv:2408.00765}
}
```

- arXiv: 2408.00765