Vcbench Eval

---
name: vcbench-eval
description: Evaluates multimodal mathematical reasoning capabilities of vision-language models, specifically focusing on vision-centric elementary math problems that require explicit visual dependencies across multiple images. It probes spatial, temporal, geometric, logical, and pattern recognition skills to measure how well models integrate cross-modal information for compositional reasoning. Use when the user wants to benchmark on VCBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.18589
  bibtex_key: wang2025vcbench
  confidence: high
---

# vcbench-eval

> Benchmarking Multimodal Mathematical Reasoning with Explicit Visual Dependency — Wang et al. (2025) (arXiv:2504.18589, 2025)

## What this evaluates

Evaluates multimodal mathematical reasoning capabilities of vision-language models, specifically focusing on vision-centric elementary math problems that require explicit visual dependencies across multiple images. It probes spatial, temporal, geometric, logical, and pattern recognition skills to measure how well models integrate cross-modal information for compositional reasoning.

## Datasets

- **VCBench** — total 1720; splits: test (1720)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered multiple-choice questions. Calculated as the number of correct predictions divided by the total number of questions, averaged across all 17 subtasks.

## Input / output format

**Input**: A set of multiple images (average 3.9 per question) accompanied by a textual question. Questions are presented in a multiple-choice format.

**Output**: A single selected option from the provided multiple-choice answers.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Models often show significantly higher accuracy in single-image settings compared to multi-image settings, indicating a bias toward single-image optimization rather than true compositional reasoning.
- Math-specialist models (e.g., G-LLaVA, MathLlava) underperform general-purpose VLMs, contrary to expectations for math benchmarks.
- Performance varies drastically across difficulty levels (Easy/Medium/Hard), so reporting only average accuracy masks model weaknesses on complex reasoning tasks.

## Evidence (verbatim from paper)

> Table 5: Accuracy comparison of various models on questions categorized by difficulty along with their average performance.

## Citation

```bibtex
@misc{wang2025vcbench,
  title={Benchmarking Multimodal Mathematical Reasoning with Explicit Visual Dependency},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2504.18589}
}
```

- arXiv: 2504.18589