Mmsi Bench Eval

---
name: mmsi-bench-eval
description: This benchmark evaluates multimodal large language models' ability to perform multi-image spatial reasoning. It probes capabilities such as tracking object and camera motion, reconstructing scenes from multiple views, and inferring spatial logic across image sequences. Use when the user wants to benchmark on MMSI-Bench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.23764
  bibtex_key: yang2025mmsibench
  confidence: high
---

# mmsi-bench-eval

> MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence — Yang et al. (2025) (arXiv:2505.23764, 2025)

## What this evaluates

This benchmark evaluates multimodal large language models' ability to perform multi-image spatial reasoning. It probes capabilities such as tracking object and camera motion, reconstructing scenes from multiple views, and inferring spatial logic across image sequences.

## Datasets

- **MMSI-Bench** — total 1000; splits: test (1000)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Exact match between answers extracted from model outputs and ground-truth answers for multiple-choice questions. An LLM-based fallback strategy extracts the intended response if the model fails to generate the answer in the required format.

## Input / output format

**Input**: A set of multiple images accompanied by a multiple-choice question testing spatial relationships, motion tracking, or scene reconstruction.

**Output**: A single character/option corresponding to the correct choice from the provided multiple-choice options.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        extracted = extract_answer(pred)  # Uses LLM fallback if format fails
        if extracted == gold:
            correct += 1
    return (correct / len(gold_answers)) * 100
```

## Common pitfalls

- Blind GPT-4o scores near random guessing, proving tasks require genuine visual-spatial reasoning rather than language priors or commonsense knowledge.
- Chain-of-Thought prompting often degrades performance, as models lack foundational spatial understanding to benefit from step-by-step reasoning.
- Increasing model parameters yields only marginal accuracy gains, indicating data quality/diversity is a bigger bottleneck than scale.

## Evidence (verbatim from paper)

> We report accuracy (%) using exact match between answers extracted from model outputs and ground-truth answers for our multiple-choice questions. If a model fails to generate an answer in the required format, we adopt the LLM-based fallback strategy from VLMEvalKit*(Duan et al., [2024])* to extract the intended response.

## Citation

```bibtex
@misc{yang2025mmsibench,
  title={MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence},
  author={Yang et al. (2025)},
  year={2025},
  note={arXiv:2505.23764}
}
```

- arXiv: 2505.23764