Mme Mmvet Spatial Eval

---
name: mme-mmvet-spatial-eval
description: Evaluates multi-modal large language models' spatial awareness and general visual perception capabilities. It probes position reasoning, object detection, and scene understanding through both binary QA pairs and open-ended prompts. Use when the user wants to benchmark on MME, MM-Vet, or asks about evaluating this task. Reports accuracy+accuracy+, GPT-4 score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.20357
  bibtex_key: zhao2023enhancing
  confidence: high
---

# mme-mmvet-spatial-eval

> Enhancing the Spatial Awareness Capability of Multi-Modal Large Language Model — Yongqiang Zhao et al. (2023) (arXiv:2310.20357, 2023)

## What this evaluates

Evaluates multi-modal large language models' spatial awareness and general visual perception capabilities. It probes position reasoning, object detection, and scene understanding through both binary QA pairs and open-ended prompts.

## Datasets

- **MME** — total 1914; splits: test (1914)
- **MM-Vet** — total 218; splits: test (218)

## Metrics

- `accuracy+accuracy+` **(primary)** — range: other
  - Sum of binary accuracy (yes/no correctness) and accuracy+ (fine-grained correctness) across all perception tasks.
- `GPT-4 score` **(primary)** — range: other
  - GPT-4 evaluates model outputs against ground truth answers and prompts, assigning a score based on alignment and correctness.

## Input / output format

**Input**: Image paired with a question (binary yes/no format for MME, open-ended format for MM-Vet).

**Output**: For MME: strictly 'yes' or 'no'. For MM-Vet: open-ended text response.

## Scoring recipe

```python
# MME scoring
mme_total = 0
for qa in mme_dataset:
    pred = model.predict(qa.image, qa.question)
    if pred == "yes":
        mme_total += accuracy_score(qa)
        if is_fine_grained_correct(pred, qa):
            mme_total += accuracy_plus_score(qa)

# MM-Vet scoring
mmvet_total = 0
for qa in mmvet_dataset:
    pred = model.predict(qa.image, qa.question)
    score = gpt4_evaluate(pred, qa.ground_truth, qa.question)
    mmvet_total += score
```

## Common pitfalls

- MME restricts outputs to 'yes'/'no', so models must be constrained to avoid free-text generation that breaks the metric.
- MM-Vet relies on GPT-4 for scoring, which introduces non-determinism and requires careful prompt engineering for the scorer.
- Spatial awareness is only a subset of tasks in both benchmarks, so overall scores may mask spatial-specific performance.

## Evidence (verbatim from paper)

> Specifically, for the MME benchmark, the model's output is limited to two types ("yes" or "no"), making it convenient to measure accuracy and accuracy+ metrics. We choose to use the sum of accuracy and accuracy+ to calculate the task score. In the case of the MM-Vet benchmark, based on existing scoring instances and the model's output under the input question and real answer conditions for each sample, GPT-4 provides specific scores to evaluate the model's performance.

## Citation

```bibtex
@misc{zhao2023enhancing,
  title={Enhancing the Spatial Awareness Capability of Multi-Modal Large Language Model},
  author={Yongqiang Zhao et al. (2023)},
  year={2023},
  note={arXiv:2310.20357}
}
```

- arXiv: 2310.20357