Vlaser Embodied Reasoning Eval

---
name: vlaser-embodied-reasoning-eval
description: This evaluation probes a model's embodied reasoning capabilities, including spatial understanding, visual grounding, task planning, and closed-loop robotic control. It measures how well vision-language models transfer general multimodal knowledge to robot-specific manipulation tasks and identifies the domain gap between internet-scale pretraining and real-world embodiment. Use when the user wants to benchmark on ERQA, Ego-Plan2, Where2place, Pointarena, Paco-Lavis, Pixmo-Points, VSI-Bench, RefSpatial-Bench, MMSI-Bench, VLABench, EmbodiedBench, SimplerEnv, Robotwin, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.11027
  bibtex_key: yang2025vlaser
  confidence: high
---

# vlaser-embodied-reasoning-eval

> Vlaser: Vision-Language-Action Model with Synergistic Embodied Reasoning — Yang et al. (2025) (arXiv:2510.11027, 2025)

## What this evaluates

This evaluation probes a model's embodied reasoning capabilities, including spatial understanding, visual grounding, task planning, and closed-loop robotic control. It measures how well vision-language models transfer general multimodal knowledge to robot-specific manipulation tasks and identifies the domain gap between internet-scale pretraining and real-world embodiment.

## Datasets

- **ERQA** — total ?; splits: test (-1)
- **Ego-Plan2** — total ?; splits: test (-1)
- **Where2place** — total ?; splits: test (-1)
- **Pointarena** — total ?; splits: test (-1)
- **Paco-Lavis** — total ?; splits: test (-1)
- **Pixmo-Points** — total ?; splits: test (-1)
- **VSI-Bench** — total ?; splits: test (-1)
- **RefSpatial-Bench** — total ?; splits: test (-1)
- **MMSI-Bench** — total ?; splits: test (-1)
- **VLABench** — total ?; splits: test (-1)
- **EmbodiedBench** — total ?; splits: test (-1)
- **SimplerEnv** — total ?; splits: test (-1)
- **Robotwin** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions or correctly grounded points across all instances. Computed as (correct predictions / total instances) * 100.
- `success rate` — range: percent
  - Percentage of successfully completed robotic manipulation episodes. Computed as (successful episodes / total episodes) * 100.
- `normalized average accuracy` — range: percent
  - Arithmetic mean of the accuracy/success rates across all 12 embodied reasoning benchmarks and simulation environments.

## Input / output format

**Input**: RGB images (single or multi-view) paired with natural language questions, spatial queries, or task instructions for manipulation.

**Output**: Natural language answers (e.g., object names, coordinates, step-by-step plans) or discrete action tokens for robot control.

## Scoring recipe

```python
def compute_score(predictions, golds, task_type):
    if task_type in ['reasoning', 'grounding', 'planning']:
        correct = sum(1 for p, g in zip(predictions, golds) if normalize(p) == normalize(g))
        return (correct / len(golds)) * 100
    elif task_type == 'robot_control':
        success = sum(1 for p, g in zip(predictions, golds) if p == g)
        return (success / len(golds)) * 100
    return 0.0
```

## Common pitfalls

- Normalizing averages across benchmarks with different scales can obscure performance drops on specific sub-tasks like spatial intelligence.
- High scores on internet-sourced reasoning benchmarks do not guarantee high success rates on closed-loop robot control due to domain shift between pretraining data and real embodiments.
- Model scaling is non-monotonic; smaller models (2B) can outperform larger ones (8B) on simple grounding tasks requiring direct answers.

## Evidence (verbatim from paper)

> Table 2: SimplerEnv Evaluation on WidowX Robot Tasks. Avg indicates the average success rate among the four tasks. Model sizes are indicated within parentheses. The result of RT-1-X, Octo-Base, OpenVLA, RoboVLM and SpatialVLA are from Qu et al. while the results of π0 is from Zren.

## Citation

```bibtex
@misc{yang2025vlaser,
  title={Vlaser: Vision-Language-Action Model with Synergistic Embodied Reasoning},
  author={Yang et al. (2025)},
  year={2025},
  note={arXiv:2510.11027}
}
```

- arXiv: 2510.11027