Vla Cross Embodiment Eval

---
name: vla-cross-embodiment-eval
description: Evaluates a vision-language-action model's ability to generalize across diverse robotic embodiments, simulation environments, and real-world platforms. It probes cross-embodiment adaptation, parameter-efficient fine-tuning capabilities, and dexterous manipulation performance. Use when the user wants to benchmark on Libero, Simpler, Calvin, VLABench, RoboTwin-2.0, NAVSIM, BridgeData-v2, Soft-Fold, or asks about evaluating this task. Reports success_rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.10274
  bibtex_key: zheng2025xvla
  confidence: high
---

# vla-cross-embodiment-eval

> X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Model — Zheng et al. (2025) (arXiv:2510.10274, 2025)

## What this evaluates

Evaluates a vision-language-action model's ability to generalize across diverse robotic embodiments, simulation environments, and real-world platforms. It probes cross-embodiment adaptation, parameter-efficient fine-tuning capabilities, and dexterous manipulation performance.

## Datasets

- **Libero** — total ?; splits: test (-1)
- **Simpler** — total ?; splits: test (-1)
- **Calvin** — total ?; splits: test (-1)
- **VLABench** — total ?; splits: test (-1)
- **RoboTwin-2.0** — total ?; splits: test (-1)
- **NAVSIM** — total ?; splits: test (-1)
- **BridgeData-v2** — total ?; splits: test (-1)
- **Soft-Fold** — total 1200; splits: test (1200)

## Metrics

- `success_rate` **(primary)** — range: percent
  - Percentage of episodes where the robot successfully completes the specified task within the environment's constraints or step limit.
- `l1_action_error` — range: other
  - Mean absolute error between predicted actions (after flow-matching denoising) and ground-truth actions on held-out validation sets.

## Input / output format

**Input**: Multi-view RGB images concatenated with natural language task instructions.

**Output**: Continuous action vectors (e.g., end-effector poses/velocities) generated via flow-matching denoising.

## Scoring recipe

```python
def compute_success_rate(predictions, gold, env_config):
    successes = 0
    for pred, gold_inst in zip(predictions, gold):
        if env_config.check_success(pred, gold_inst):
            successes += 1
    return (successes / len(gold)) * 100
```

## Common pitfalls

- Different benchmarks report success rates on different sub-tasks (e.g., Libero-Spatial vs. Libero-Long); results must specify the exact variant.
- Real-world success rates depend heavily on contact-based or goal-reaching criteria that vary per environment; mixing simulation and real-world metrics without clarification causes confusion.
- PEFT results use only 1% of parameters (9M); comparing them directly to fully fine-tuned baselines without noting parameter count is misleading.

## Evidence (verbatim from paper)

> achieving 93% and 54% success rates on Libero and Simpler-WidowX benchmarks, respectively.

## Citation

```bibtex
@misc{zheng2025xvla,
  title={X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Model},
  author={Zheng et al. (2025)},
  year={2025},
  note={arXiv:2510.10274}
}
```

- arXiv: 2510.10274