Simplerenv Eval

---
name: simplerenv_eval
description: Evaluates a robot's ability to execute manipulation tasks from visual inputs and textual instructions, testing both atomic skill execution and high-level instruction generalization in simulated and real-world settings. Use when the user wants to benchmark on SimplerEnv, SimplerEnv-Instruct, or asks about evaluating this task. Reports visual matching (VM).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.17520
  bibtex_key: yang2025instructvla
  confidence: high
---

# simplerenv_eval

> InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulation — Yang et al. (2025) (arXiv:2507.17520, 2025)

## What this evaluates

Evaluates a robot's ability to execute manipulation tasks from visual inputs and textual instructions, testing both atomic skill execution and high-level instruction generalization in simulated and real-world settings.

## Datasets

- **SimplerEnv** — total ?; splits: test (-1)
- **SimplerEnv-Instruct** — total ?; splits: test (-1)

## Metrics

- `visual matching (VM)` **(primary)** — range: percent
  - Success rate measured by visual matching of the final state to the goal.
- `variance aggregation (VA)` — range: percent
  - Success rate measured by variance aggregation across multiple trials or states.

## Input / output format

**Input**: Robot camera images, robot state observations, and textual instructions.

**Output**: Action tokens (low-level control commands) and optionally textual reasoning.

## Scoring recipe

```python
predictions, gold = load_data(simplerenv)
success = sum(1 for p in predictions if p == gold) / len(predictions) * 100
return success
```

## Common pitfalls

- VM and VA are two distinct evaluation settings; results are reported separately and should not be averaged without clarification.
- SimplerEnv-Instruct evaluates high-level instruction generalization, which requires reasoning over novel objects/tasks not seen during training.

## Evidence (verbatim from paper)

> SimplerEnv [37] features real-to-sim evaluation on large-scale manipulation datasets [34, 33] with visual matching and variance aggregation settings to evaluate generalization ability.

## Citation

```bibtex
@misc{yang2025instructvla,
  title={InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulation},
  author={Yang et al. (2025)},
  year={2025},
  note={arXiv:2507.17520}
}
```

- arXiv: 2507.17520