Vlabench Eval

---
name: vlabench-eval
description: Evaluates the generalization, long-horizon reasoning, and language-conditioned manipulation capabilities of Vision-Language-Action (VLA) models, workflow frameworks, and Vision-Language Models (VLMs) in simulated robotic environments. It probes performance across seen/unseen objects, semantic instruction understanding, and composite task decomposition. Use when the user wants to benchmark on VLABench, or asks about evaluating this task. Reports task_progress_score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.18194
  bibtex_key: [REDACTED]
  confidence: high
---

# vlabench-eval

> VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks — Zhang et al. (2024) (arXiv:2412.18194, 2024)

## What this evaluates

Evaluates the generalization, long-horizon reasoning, and language-conditioned manipulation capabilities of Vision-Language-Action (VLA) models, workflow frameworks, and Vision-Language Models (VLMs) in simulated robotic environments. It probes performance across seen/unseen objects, semantic instruction understanding, and composite task decomposition.

## Datasets

- **VLABench** — total 1600; splits: seen (-1), unseen (-1)

## Metrics

- `task_progress_score` **(primary)** — range: percent
  - A percentage-based score (0-100) measuring task completion rate or success rate across primitive and composite tasks. For VLMs, it is a weighted aggregation of four DAG-matching metrics (skill, parameter, dependency, and completeness) comparing predicted skill sequences to reference ones.

## Input / output format

**Input**: For VLAs: multi-view visual observations and natural language instructions. For VLMs: two four-view images (one annotated with masks/labels, one reference) paired with a natural language instruction.

**Output**: For VLAs: continuous or discretized robot action sequences/trajectories. For VLMs: a Domain-Specific Language (DSL) output consisting of a sequence of skills, each with a name and parameters, conforming to predefined patterns.

## Scoring recipe

```python
def score_vlabench(predictions, gold, mode="interactive"):
    if mode == "interactive":
        env = VLABench_Env()
        return env.run_and_track_progress(predictions)
    else:
        pred_dag = build_dag(predictions)
        gold_dag = build_dag(gold)
        m1, m2, m3, m4 = compute_dag_metrics(pred_dag, gold_dag)
        return weighted_aggregate([m1, m2, m3, m4])
```

## Common pitfalls

- Confusing interactive evaluation (environment simulation tracking task progress) with non-interactive evaluation (DAG matching of DSL outputs).
- Assuming the 'four metrics' for VLM scoring are standard NLP metrics; they specifically measure structural alignment between predicted and reference skill DAGs.
- Overlooking that 'seen' vs 'unseen' splits are category-level, meaning models may memorize specific object textures rather than generalize to new instances.

## Evidence (verbatim from paper)

> Interactive evaluation computes a task progress score based on the interaction with the environment. VLABench provides a controller that parses the DSL action sequences output by the VLM into executable actions, which are then applied in a simulation environment to interact with real-world objects.

## Citation

```bibtex
@misc{zhang2024vlabench,
  title={VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks},
  author={Zhang et al. (2024)},
  year={2024},
  note={arXiv:2412.18194}
}
```

- arXiv: 2412.18194