Visuriddles Eval

---
name: visuriddles-eval
description: Evaluates multimodal large language models' ability to perform abstract visual reasoning across five fine-grained perceptual dimensions (numerosity, attributes, style, position, spatial relations) and two high-level reasoning tasks (analogical pattern matching and constraint-based logic). Use when the user wants to benchmark on VisuRiddles, or asks about evaluating this task. Reports exact match.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.02537
  bibtex_key: yan2025visuriddles
  confidence: high
---

# visuriddles-eval

> VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning — Yan et al. (2025) (arXiv:2506.02537, 2025)

## What this evaluates

Evaluates multimodal large language models' ability to perform abstract visual reasoning across five fine-grained perceptual dimensions (numerosity, attributes, style, position, spatial relations) and two high-level reasoning tasks (analogical pattern matching and constraint-based logic).

## Datasets

- **VisuRiddles** — total 1000; splits: test (1000); repo https://github.com/yh-hust/VisuRiddles

## Metrics

- `exact match` **(primary)** — range: [0, 1]
  - Fraction of correctly predicted answers. For single-choice tasks, correct if the predicted option matches the gold option. For high-level tasks, correct if the generated symbolic output exactly matches the gold output.

## Input / output format

**Input**: Image grid(s) (e.g., 3x3 matrix or progression) accompanied by a text question in English.

**Output**: For basic categories: a single letter (A, B, C, or D). For high-level categories: exact symbolic output (constraint-based grid or pattern match index).

## Scoring recipe

```python
def compute_metric(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if len(gold) > 1:  # high-level symbolic
            correct += (pred.strip() == gold.strip())
        else:  # single-choice
            correct += (pred.strip().upper() == gold.strip().upper())
    return correct / len(golds)
```

## Common pitfalls

- Assuming all tasks use multiple-choice format; high-level RAVEN and Sudoku tasks require exact symbolic generation, not letter selection.
- Overestimating real-world difficulty; the dataset is heavily synthesized via the PRS framework with deliberately tractable complexity to focus on perceptual supervision.

## Evidence (verbatim from paper)

> In the high-level categories, RAVEN Reasoning and Sudoku Reasoning each account for 10% of the total data, and require models to generate exact symbolic outputs to be considered correct.

## Citation

```bibtex
@misc{yan2025visuriddles,
  title={VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning},
  author={Yan et al. (2025)},
  year={2025},
  note={arXiv:2506.02537}
}
```

- arXiv: 2506.02537