Mmstar Eval

---
name: mmstar-eval
description: Evaluates Large Vision-Language Models (LVLMs) on six core capabilities (coarse perception, fine-grained perception, instance reasoning, logical reasoning, science & technology, and mathematics) using a human-curated benchmark designed to enforce strict visual dependency and minimize data leakage. Use when the user wants to benchmark on MMStar, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.20330
  bibtex_key: chen2024mmstar
  confidence: high
---

# mmstar-eval

> Are We on the Right Way for Evaluating Large Vision-Language Models? — Lin Chen et al. (2024) (arXiv:2403.20330, 2024)

## What this evaluates

Evaluates Large Vision-Language Models (LVLMs) on six core capabilities (coarse perception, fine-grained perception, instance reasoning, logical reasoning, science & technology, and mathematics) using a human-curated benchmark designed to enforce strict visual dependency and minimize data leakage.

## Datasets

- **MMStar** — total 1500; splits: test (1500)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions, determined via heuristic matching rules against the gold option.
- `multi-modal gain (MG)` — range: percent
  - Measures the actual performance improvement attributable to vision, calculated as the difference between accuracy with image and accuracy without image.
- `multi-modal leakage (ML)` — range: percent
  - Quantifies data leakage by measuring model accuracy when evaluated in a text-only setting (LVLM-text), indicating reliance on memorized text rather than visual reasoning.

## Input / output format

**Input**: Multiple-choice question paired with an image (for visual evaluation) or text-only (for leakage evaluation). Models receive the question and options.

**Output**: Selected option letter/text from the multiple-choice answers.

## Scoring recipe

```python
# Per instance
pred = heuristic_match(model_output, options)
acc = 1.0 if pred == gold else 0.0

# Aggregation
acc_with_img = mean(acc_with_image)
acc_without_img = mean(acc_without_image) # LVLM-text setting
MG = acc_with_img - acc_without_img
ML = acc_without_img
```

## Common pitfalls

- LLMs often refuse to answer visual questions without images; using 2-shot prompting for LLMs on MMStar is required to prevent refusal bias from skewing results.
- Some LVLMs (e.g., GeminiPro-Vision, CogVLM) crash or behave incorrectly if image tokens are simply removed; they require replacing images with pure grey images to properly evaluate the text-only setting.
- Questions must be strictly visually dependent; answers derivable from text alone or world knowledge invalidate the visual dependency control.

## Evidence (verbatim from paper)

> In this section, we present the results of our proposed multi-modal gain (MG) and multi-modal leakage (ML) metrics of 16 LVLMs with varying sizes and architectures on 6 popular benchmarks and our MMStar benchmark. Given that all questions are ensured to be converted into a multiple-choice format, we develop some heuristic matching rules to calculate accuracy, avoiding the cumbersome process of re-invoking GPT4 for answer extraction.

## Citation

```bibtex
@misc{chen2024mmstar,
  title={Are We on the Right Way for Evaluating Large Vision-Language Models?},
  author={Lin Chen et al. (2024)},
  year={2024},
  note={arXiv:2403.20330}
}
```

- arXiv: 2403.20330