Sparrta Eval

---
name: sparrta-eval
description: This benchmark evaluates the spatial reasoning capabilities of Visual Foundation Models (VFMs) by testing their ability to recognize spatial relations between object triples in synthetic images. It specifically probes both egocentric (camera-perspective) and allocentric (world-perspective) spatial understanding across diverse semantic objects and environments. Use when the user wants to benchmark on SpaRRTa, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.11729
  bibtex_key: kargin2026sparrta
  confidence: high
---

# sparrta-eval

> SpaRRTa: A Synthetic Benchmark for Evaluating Spatial Intelligence in Visual Foundation Models — Kargin et al. (2026) (arXiv:2601.11729, 2026)

## What this evaluates

This benchmark evaluates the spatial reasoning capabilities of Visual Foundation Models (VFMs) by testing their ability to recognize spatial relations between object triples in synthetic images. It specifically probes both egocentric (camera-perspective) and allocentric (world-perspective) spatial understanding across diverse semantic objects and environments.

## Datasets

- **SpaRRTa** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted spatial relations (source-target-viewpoint triples) on the test set. Probes are trained per object triple and environment, and results are averaged across 2 random seeds and 3 distinct object triples per environment.

## Input / output format

**Input**: A single RGB image (224×224) depicting a layout of three objects (source, target, viewpoint), processed by a frozen VFM backbone to extract patch tokens or global features.

**Output**: A categorical label indicating the spatial relation between the specified source, target, and viewpoint objects.

## Scoring recipe

```python
correct = 0
total = 0
for image, true_relation in test_set:
    features = frozen_vfm(image)
    probe_output = probe_head(features)
    predicted_relation = argmax(probe_output)
    if predicted_relation == true_relation:
        correct += 1
    total += 1
accuracy = (correct / total) * 100
```

## Common pitfalls

- Using global average pooling (Linear probing) discards crucial patch-level spatial information, leading to artificially low performance compared to attention-based or multi-query pooling methods.
- Assuming 3D-supervised models (like VGGT) will universally outperform 2D self-supervised models; in reality, 3D supervision primarily enriches patch-level structure rather than global representations, making advanced probing necessary to observe gains.
- Confusing egocentric and allocentric tasks; allocentric recognition is consistently more challenging and yields systematically lower accuracy across all models and probing strategies.

## Evidence (verbatim from paper)

> For a given VFM, we evaluate the SpaRRTa performance in terms of the accuracy of probes trained for recognizing the spatial relations of given source, target, viewpoint object triples. Performance on SpaRRTa across images belonging to a given semantic triple therefore reflects the ability to resolve spatial relations for those particular object types, while averaging over diverse semantic triples provides an aggregate measure of spatial relation recognition. For each such triple of objects, we curate a dataset of images depicting different object layouts, and split it into train, validation, and test folds in 80/10/10 proportions. We train triple-specific probes for the number of epochs specified in Table 2, using the validation set to select the best-performing probe parameters. We report the test accuracies of probes, and repeat this procedure with 2 random seeds and a diverse selection of 3 distinct object triples per each of the 5 environments (see Section 3.3) to achieve a robust evaluation.

## Citation

```bibtex
@misc{kargin2026sparrta,
  title={SpaRRTa: A Synthetic Benchmark for Evaluating Spatial Intelligence in Visual Foundation Models},
  author={Kargin et al. (2026)},
  year={2026},
  note={arXiv:2601.11729}
}
```

- arXiv: 2601.11729