Sphere Eval

---
name: sphere-eval
description: Evaluates spatial reasoning and visual understanding in vision-language models across a hierarchy of tasks, including single-skill perception (position, counting, distance, size), multi-skill integration, and complex physical-world reasoning (object occlusion and manipulation). It specifically probes egocentric vs. allocentric perspective taking and susceptibility to object hallucination. Use when the user wants to benchmark on SPHERE, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.12693
  bibtex_key: zhang2024sphere
  confidence: high
---

# sphere-eval

> SPHERE: Unveiling Spatial Blind Spots in Vision-Language Models Through Hierarchical Evaluation — Zhang et al. (2024) (arXiv:2412.12693, 2024)

## What this evaluates

Evaluates spatial reasoning and visual understanding in vision-language models across a hierarchy of tasks, including single-skill perception (position, counting, distance, size), multi-skill integration, and complex physical-world reasoning (object occlusion and manipulation). It specifically probes egocentric vs. allocentric perspective taking and susceptibility to object hallucination.

## Datasets

- **SPHERE** — total 2285; splits: test (2285); repo https://github.com/zwenyu/SPHERE-VLM

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered instances. For multiple-choice questions, it is the proportion of correct selections. For open-ended counting tasks, it is exact-match accuracy on the integer count.

## Input / output format

**Input**: An image (sourced from MS COCO-2017 test split) paired with a natural language question. Questions are either multiple-choice (2-3 options) or open-ended numerical prompts.

**Output**: A natural language response: either the selected option text for MCQs or an integer for counting tasks.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if str(pred).strip().lower() == str(gold).strip().lower():
            correct += 1
    return (correct / len(golds)) * 100
```

## Common pitfalls

- Confusing egocentric (entity-relative) and allocentric (camera-relative) viewpoints when evaluating position tasks.
- Overlooking 'trick' counting questions where the ground truth is zero, which specifically tests model hallucination rather than mere counting ability.
- Failing to separate intermediate (perception-based) and final (reasoning-based) questions in occlusion/manipulation tasks, as they measure different cognitive stages.

## Evidence (verbatim from paper)

> We annotate a total of 2,285 question-answer pairs. ... Counting-related tasks require open-ended numerical responses, while other tasks use multiple-choice questions (MCQs) with two options by default and three options when additional clarity is needed. ... Table 3: Average accuracy (%) of VLMs on SPHERE tasks.

## Citation

```bibtex
@misc{zhang2024sphere,
  title={SPHERE: Unveiling Spatial Blind Spots in Vision-Language Models Through Hierarchical Evaluation},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2412.12693}
}
```

- arXiv: 2412.12693