Visnumbench Eval

---
name: visnumbench-eval
description: Evaluates the intuitive number sense of Multimodal Large Language Models (MLLMs) by testing their ability to estimate and reason about seven visual numerical attributes (angle, scale, length, quantity, depth, area, volume) across four estimation tasks (range estimation, value estimation, value comparison, multiplicative estimation). Use when the user wants to benchmark on VisNumBench, or asks about evaluating this task. Reports Accuracy (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.14939
  bibtex_key: weng2025visnumbench
  confidence: high
---

# visnumbench-eval

> VisNumBench: Evaluating Number Sense of Multimodal Large Language Models — Weng et al. (2025) (arXiv:2503.14939, 2025)

## What this evaluates

Evaluates the intuitive number sense of Multimodal Large Language Models (MLLMs) by testing their ability to estimate and reason about seven visual numerical attributes (angle, scale, length, quantity, depth, area, volume) across four estimation tasks (range estimation, value estimation, value comparison, multiplicative estimation).

## Datasets

- **VisNumBench** — total 600; splits: synthetic (300), real (300)

## Metrics

- `Accuracy (%)` **(primary)** — range: percent
  - Percentage of correctly answered questions out of the total number of questions. Calculated as (number of correct predictions / total number of questions) * 100.

## Input / output format

**Input**: Image (synthetic or real) paired with a question targeting one of seven visual numerical attributes (angle, scale, length, quantity, depth, area, volume) across four estimation tasks.

**Output**: Text response to the question.

## Scoring recipe

```python
correct = 0
total = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
    total += 1
accuracy = (correct / total) * 100
```

## Common pitfalls

- Accuracy is averaged across all tasks and attributes, but task difficulty varies significantly (e.g., quantity tasks are easiest, angle tasks are hardest).
- Performance on synthetic images does not reliably transfer to real-world images due to increased complexity and variability in the latter.
- Applying multimodal CoT or math-specialized fine-tuning does not guarantee accuracy gains and can sometimes degrade performance on real-world data.

## Evidence (verbatim from paper)

> We randomly selected $600$ samples ($50$ QA pairs from each numerical attribute), with $300$ sourced from VisNumBench-Synthetic and $300$ from VisNumBench-Real. Human evaluators independently answered each question and provided assessments. Accuracy (%) is reported for all experimental results, and all the results are provided in Tables[2] and [3].

## Citation

```bibtex
@misc{weng2025visnumbench,
  title={VisNumBench: Evaluating Number Sense of Multimodal Large Language Models},
  author={Weng et al. (2025)},
  year={2025},
  note={arXiv:2503.14939}
}
```

- arXiv: 2503.14939