Sphinx Eval

---
name: sphinx-eval
description: Probes visual perception and reasoning capabilities of vision-language models across 25 distinct task types, including symmetry, spatial transformations, chart interpretation, and sequence prediction. Uses a synthetic environment with verifiable ground truth to measure model accuracy against human baselines. Use when the user wants to benchmark on Sphinx, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.20814
  bibtex_key: alam2025sphinx
  confidence: high
---

# sphinx-eval

> SPHINX: A Synthetic Environment for Visual Perception and Reasoning — Alam et al. (2025) (arXiv:2511.20814, 2025)

## What this evaluates

Probes visual perception and reasoning capabilities of vision-language models across 25 distinct task types, including symmetry, spatial transformations, chart interpretation, and sequence prediction. Uses a synthetic environment with verifiable ground truth to measure model accuracy against human baselines.

## Datasets

- **Sphinx** — total 2500; splits: test (2500); repo https://github.com/xashru/sphinx

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Percentage of correctly answered questions out of the total number of questions. Answers are extracted using the mathruler library, with GPT-5-mini as a fallback for ambiguous extractions.

## Input / output format

**Input**: An image and a natural language question describing a visual reasoning task (e.g., symmetry, spatial transformation, chart interpretation, or sequence prediction).

**Output**: A natural language answer or selected option, processed through the mathruler library for extraction and verification.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_answers):
    extracted = mathruler_extract(pred)
    if extracted == gold:
        correct += 1
accuracy = correct / len(gold_answers)
```

## Common pitfalls

- Answer extraction relies on an external library (mathruler) with an LLM fallback, which may introduce extraction bias or inconsistency across models.
- Models are evaluated using their 'default reasoning settings', which are not standardized or optimized, making cross-model comparisons sensitive to inference hyperparameters.
- Human performance is reported as a single aggregate gap (24.3%) without detailing the human evaluation protocol or task breakdown.

## Evidence (verbatim from paper)

> We curated the Sphinx benchmark to include 2,500 questions, with 100 instances per task. We evaluated three proprietary variants of GPT-5 (regular, mini, and nano) using their default reasoning settings*. In addition, we evaluated nine open-source vision-language models, including the Qwen2.5-VL family*, Llama 3.2*, InternVL3*, and the Qwen3-VL family*, covering parameter scales from 3B to 38B. The evaluation prompt is provided in the Appendix[G]. We use the mathruler library* for answer extraction and verification, with GPT-5-mini serving as a fallback when extraction is ambiguous. For open-source models, we use the VLMEvalKit framework for inference*. Results are summarized in Table[1]. Overall, GPT-5 achieves the best performance with an average accuracy of 51.1% on all tasks, although it still falls short of human accuracy by 24.3%.

## Citation

```bibtex
@misc{alam2025sphinx,
  title={SPHINX: A Synthetic Environment for Visual Perception and Reasoning},
  author={Alam et al. (2025)},
  year={2025},
  note={arXiv:2511.20814}
}
```

- arXiv: 2511.20814