ESpatial Eval

---
name: eSpatial-eval
description: Probes multimodal models' ability to perform complex, long-horizon spatial reasoning and physical consistency checks in dynamic, embodied scenarios. It evaluates object attribute recognition, relational understanding, and robotic manipulation planning across static images and real-world assembly tasks. Use when the user wants to benchmark on eSpatial-Benchmark, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.11089
  bibtex_key: zhang2025embodiedvsr
  confidence: high
---

# eSpatial-eval

> EmbodiedVSR: Dynamic Scene Graph-Guided Chain-of-Thought Reasoning for Visual Spatial Tasks — Yi Zhang et al. (2025) (arXiv:2503.11089, 2025)

## What this evaluates

Probes multimodal models' ability to perform complex, long-horizon spatial reasoning and physical consistency checks in dynamic, embodied scenarios. It evaluates object attribute recognition, relational understanding, and robotic manipulation planning across static images and real-world assembly tasks.

## Datasets

- **eSpatial-Benchmark** — total 1005; splits: test (1005)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted answers or attributes out of the total number of instances. Calculated as (correct predictions / total instances) * 100.
- `success_rate` — range: percent
  - Percentage of real-world robotic assembly tasks completed successfully out of the total number of randomized tests.

## Input / output format

**Input**: RGB or RGB-D images of spatial scenes or LEGO structures, accompanied by textual prompts or questions regarding spatial relationships, object attributes, or manipulation instructions.

**Output**: Textual chain-of-thought reasoning followed by a final answer (e.g., spatial relationship label, attribute value, or structured command for robot assembly).

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p.strip().lower() == g.strip().lower())
    return (correct / len(gold_labels)) * 100

def compute_success_rate(successes, total_tests):
    return (successes / total_tests) * 100
```

## Common pitfalls

- Models often struggle with depth perception and precise object localization when relying solely on prompting, leading to VLM confusion where the model misinterprets its own outputs.
- Evaluating real-world robotic assembly requires accounting for occlusion challenges and physical constraints not fully captured in static image benchmarks.
- Accuracy metrics may mask failures in multi-step chain-of-thought reasoning if only the final answer is checked without verifying intermediate reasoning steps.

## Evidence (verbatim from paper)

> In 20 real-world randomized block assembly tests, EmbodiedVSR achieved a 100% accuracy in describing the block assembly, while the robot’s operational success rate was 80%.

## Citation

```bibtex
@misc{zhang2025embodiedvsr,
  title={EmbodiedVSR: Dynamic Scene Graph-Guided Chain-of-Thought Reasoning for Visual Spatial Tasks},
  author={Yi Zhang et al. (2025)},
  year={2025},
  note={arXiv:2503.11089}
}
```

- arXiv: 2503.11089