Cebench Eval

---
name: cebench-eval
description: Evaluates vision-language-action (VLA) models on cross-embodiment robotic manipulation tasks, including single-arm, bimanual, and mobile manipulation. It probes spatial reasoning, visual generalization under domain randomization, and the ability to unify navigation and manipulation in a single policy. Use when the user wants to benchmark on CEBench, or asks about evaluating this task. Reports success_rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.22663
  bibtex_key: song2026cebench
  confidence: high
---

# cebench-eval

> Rethinking the Practicality of Vision-language-action Model: A Comprehensive Benchmark and An Improved Baseline — Song et al. (2026) (arXiv:2602.22663, 2026)

## What this evaluates

Evaluates vision-language-action (VLA) models on cross-embodiment robotic manipulation tasks, including single-arm, bimanual, and mobile manipulation. It probes spatial reasoning, visual generalization under domain randomization, and the ability to unify navigation and manipulation in a single policy.

## Datasets

- **CEBench** — total ?; splits: test (-1), seen (-1), domain_randomization (-1)

## Metrics

- `success_rate` **(primary)** — range: percent
  - Binary metric indicating whether a task episode is completed successfully. Reported as a percentage across multiple trials (e.g., 100 or 1000 runs per task).
- `avg_completed_trajectory_length` — range: other
  - Average number of steps or subtasks successfully completed per trajectory across all trials.

## Input / output format

**Input**: Multi-view camera images, language instructions, and proprioceptive state observations.

**Output**: Unified action commands for navigation and manipulation (e.g., joint velocities or end-effector poses).

## Scoring recipe

```python
def compute_metrics(predictions, gold, episodes):
    success_count = 0
    total_steps = 0
    for ep in episodes:
        if check_success(ep, predictions[ep]):
            success_count += 1
            total_steps += len(ep.completed_trajectory)
    success_rate = (success_count / len(episodes)) * 100
    avg_len = total_steps / success_count if success_count > 0 else 0
    return {'success_rate': success_rate, 'avg_completed_trajectory_length': avg_len}
```

## Common pitfalls

- Domain randomization (DR) settings cause dramatic success rate drops for baseline models, so evaluating only on 'seen' tasks overestimates generalization.
- Mobile manipulation success depends heavily on navigation accuracy; poor navigation causes manipulation success to drop to near zero regardless of the manipulation policy.
- Robotics evaluations are highly stochastic; the protocol requires 100 to 1000 trials per task to ensure statistical reliability.

## Evidence (verbatim from paper)

> We report the average completed trajectory length (Avg. Len.) across all five subtasks as well as success rates on each subtask. Following the official ABC→D settings, the evaluation is conducted in an unseen scene. To ensure reliable evaluation, we test each method 1000 times.

## Citation

```bibtex
@misc{song2026cebench,
  title={Rethinking the Practicality of Vision-language-action Model: A Comprehensive Benchmark and An Improved Baseline},
  author={Song et al. (2026)},
  year={2026},
  note={arXiv:2602.22663}
}
```

- arXiv: 2602.22663