Embspatial Bench Eval

---
name: embspatial-bench-eval
description: Evaluates large vision-language models' ability to reason about egocentric spatial relations (e.g., above, below, left, right, close, far) within 3D embodied environments. It probes whether models can accurately localize objects and identify spatial configurations from a first-person perspective. Use when the user wants to benchmark on Embspacial-Bench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.05756
  bibtex_key: du2024embspatialbench
  confidence: high
---

# embspatial-bench-eval

> Embspacial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Models — Du et al. (2024) (arXiv:2406.05756, 2024)

## What this evaluates

Evaluates large vision-language models' ability to reason about egocentric spatial relations (e.g., above, below, left, right, close, far) within 3D embodied environments. It probes whether models can accurately localize objects and identify spatial configurations from a first-person perspective.

## Datasets

- **Embspacial-Bench** — total ?; splits: test (-1); repo https://github.com/mengfeidu/EmbSpatial-Bench

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted options out of the total number of queries. Calculated separately for generation-based and likelihood-based evaluation strategies.

## Input / output format

**Input**: Egocentric visual scene (image or 3D representation) paired with a natural language query asking about spatial relations between objects.

**Output**: For generation strategy: a textual prediction of the correct option. For likelihood strategy: token probabilities for each candidate option.

## Scoring recipe

```python
correct = 0
total = 0
for pred, gold in zip(predictions, gold_labels):
    if pred == gold:
        correct += 1
    total += 1
accuracy = (correct / total) * 100
```

## Common pitfalls

- Generation-based accuracy is heavily penalized by instruction-following failures rather than pure spatial reasoning errors.
- Egocentric spatial relations are viewpoint-dependent; models often confuse left/right or close/far when the camera orientation changes.
- Likelihood strategy requires access to internal token probabilities, which is not available for all closed-source models.

## Evidence (verbatim from paper)

> Based on EmbSpatial-Bench, we conduct zero-shot evaluation of current LVLMs, using accuracy as the metric. Two evaluation strategies are employed. The first one is the generation-based strategy, which directly uses predicted options from the textual outputs of models. Considering the insufficient instruction-following ability of some LVLMs, we also employed a likelihood strategy, using the option with the highest probability generated by the model

## Citation

```bibtex
@misc{du2024embspatialbench,
  title={Embspacial-Bench: Benchmarking Spatial Understanding for Embodied Tasks with Large Vision-Language Models},
  author={Du et al. (2024)},
  year={2024},
  note={arXiv:2406.05756}
}
```

- arXiv: 2406.05756