Visulogic Eval

---
name: visulogic-eval
description: VisuLogic probes vision-centric reasoning in multimodal large language models by presenting problems that require retaining critical visual cues during image description. It eliminates text-based reasoning shortcuts, forcing models to perform genuine visual inference across categories like spatial relations, quantitative shifts, and stylistic details. Use when the user wants to benchmark on VisuLogic, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.15279
  bibtex_key: xu2025visulogic
  confidence: high
---

# visulogic-eval

> VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models — Weiye Xu et al. (2025) (arXiv:2504.15279, 2025)

## What this evaluates

VisuLogic probes vision-centric reasoning in multimodal large language models by presenting problems that require retaining critical visual cues during image description. It eliminates text-based reasoning shortcuts, forcing models to perform genuine visual inference across categories like spatial relations, quantitative shifts, and stylistic details.

## Datasets

- **VisuLogic** — total 1000; splits: test (1000)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Calculated as the percentage of correctly answered multiple-choice questions out of the total. For human baselines, it aggregates results from 100 participants each solving 10 randomly sampled questions. For models, it measures the proportion of correct selections among four options.

## Input / output format

**Input**: For MLLMs: an image paired with a multiple-choice question. For LLMs: a GPT-4o-generated image description, the question, and four options. Three prompting modes are used: Non-CoT (direct answer), CoT (step-by-step reasoning), and Hint (CoT with solution-derived hints).

**Output**: Answer: \boxed{$LETTER} where $LETTER corresponds to one of the four provided options.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip() == g.strip())
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Using text-only LLMs with external image captions fails to capture essential visual details, leading to hallucinations or irrelevant responses.
- Chain-of-thought (CoT) prompting provides minimal performance gains for visual reasoning tasks compared to pure-text tasks, as current CoT training relies heavily on text corpora.
- Models struggle disproportionately with stylistic reasoning (e.g., overlays, contours) and spatial reasoning, often resorting to superficial cues like object counts.

## Evidence (verbatim from paper)

> We invite 100 graduate students majoring in science and engineering to solve 10 randomly sampled VisuLogic questions each, allowing 2–5 minutes per question. The aggregate accuracy over all participants constitutes the human benchmark. We apply three distinct prompting paradigms to investigate model reasoning capabilities: 1)Non-CoT prompt evaluation: Models receive a concise instruction: “Answer the question using a single word or phrase, following this format: Answer: \boxed{$LETTER}”. 2)CoT prompt evaluation: We prompt models to articulate intermediate reasoning steps: “Solve the complex visual logical reasoning problem through step-by-step reasoning. Think about the reasoning process first and answer the question following this format: Answer: \boxed{$LETTER}”.

## Citation

```bibtex
@misc{xu2025visulogic,
  title={VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models},
  author={Weiye Xu et al. (2025)},
  year={2025},
  note={arXiv:2504.15279}
}
```

- arXiv: 2504.15279