Visonlyqa Eval

---
name: visonlyqa-eval
description: This benchmark probes a model's ability to accurately perceive basic geometric information—such as shape, angle, length, area, and intersections—in scientific figures and diagrams. It isolates visual perception from higher-level reasoning or domain knowledge by using direct, low-reasoning questions on synthetic and real-world images. Use when the user wants to benchmark on VisOnlyQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.00947
  bibtex_key: kamoi2024visonlyqa
  confidence: high
---

# visonlyqa-eval

> VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information — Kamoi et al. (2024) (arXiv:2412.00947, 2024)

## What this evaluates

This benchmark probes a model's ability to accurately perceive basic geometric information—such as shape, angle, length, area, and intersections—in scientific figures and diagrams. It isolates visual perception from higher-level reasoning or domain knowledge by using direct, low-reasoning questions on synthetic and real-world images.

## Datasets

- **VisOnlyQA** — total ?; splits: train (10000), Eval-Real (-1), Eval-Synthetic (-1); repo https://github.com/psunlpgroup/VisOnlyQA

## Metrics

- `accuracy` **(primary)** — range: percent
  - The percentage of correctly answered questions out of the total number of instances. Calculated as (number of exact matches between model prediction and ground truth label) / (total number of predictions) * 100.

## Input / output format

**Input**: A single image (scientific figure, geometric shape, chart, or 3D diagram) paired with a natural language question asking about its geometric properties.

**Output**: A text response containing the model's answer. Models may optionally generate chain-of-thought reasoning before the final answer.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        if normalize_answer(pred) == normalize_answer(gold):
            correct += 1
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Assuming chain-of-thought prompting improves performance; the paper shows CoT does not consistently help because reasoning is not the bottleneck.
- Misattributing low scores to reasoning or knowledge deficits; error analysis confirms that almost all mistakes stem from visual perception errors rather than logical flaws.

## Evidence (verbatim from paper)

> Table 5 shows the accuracy of LVLMs on Eval-Real and Eval-Synthetic (with no chain-of-thought). The performance of LVLMs is far from perfect on all tasks, with the best average accuracies of  $79.0\%$  and  $55.4\%$  by Gemini 2.5 Pro on the Real and Synthetic splits, while human performance is nearly perfect  $(93.5\%$  and  $95.0%)$ .

## Citation

```bibtex
@misc{kamoi2024visonlyqa,
  title={VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information},
  author={Kamoi et al. (2024)},
  year={2024},
  note={arXiv:2412.00947}
}
```

- arXiv: 2412.00947