Starvla Alpha Generalist Eval

---
name: starvla-alpha-generalist-eval
description: Evaluates whether a single Vision-Language-Action model can generalize across diverse robotic manipulation benchmarks without task-specific fine-tuning. It probes the model's cross-embodiment generalization and robustness to varying action spaces and task distributions. Use when the user wants to benchmark on LIBERO, SimplerEnv, RoboTwin 2.0, RoboCasa-GR1, RoboChallenge, or asks about evaluating this task. Reports success_rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.11757
  bibtex_key: ye2026starvla
  confidence: high
---

# starvla-alpha-generalist-eval

> StarVLA-$\alpha$: Reducing Complexity in Vision-Language-Action Systems — Ye et al. (2026) (arXiv:2604.11757, 2026)

## What this evaluates

Evaluates whether a single Vision-Language-Action model can generalize across diverse robotic manipulation benchmarks without task-specific fine-tuning. It probes the model's cross-embodiment generalization and robustness to varying action spaces and task distributions.

## Datasets

- **LIBERO** — total ?; splits: train (-1), test (-1)
- **SimplerEnv** — total ?; splits: train (-1), test (-1)
- **RoboTwin 2.0** — total ?; splits: train (-1), test (-1)
- **RoboCasa-GR1** — total ?; splits: train (-1), test (-1)
- **RoboChallenge** — total ?; splits: test (-1)

## Metrics

- `success_rate` **(primary)** — range: percent
  - Calculated as the percentage of successful task completions out of the total number of evaluation trials.
- `progress_score` — range: other
  - A continuous progress metric averaged across trials for each task, reported alongside success rate.

## Input / output format

**Input**: Visual observations, natural language task instructions, and robot proprioceptive/state observations.

**Output**: 32-dimensional action vector (uniformly padded across robots with varying degrees of freedom).

## Scoring recipe

```python
def compute_metrics(predictions, gold, total_trials):
    # gold: binary flags for task completion per trial
    success_rate = sum(gold) / total_trials * 100
    # progress_score: average of continuous progress values per trial
    progress_score = sum(progress_values) / total_trials
    return success_rate, progress_score
```

## Common pitfalls

- Fine-tuning the model on individual benchmark datasets before evaluation, which violates the generalist protocol.
- Using complex, robot-specific action parameterizations (e.g., RDT Action, Multi-Action Head) instead of the specified simple padding strategy.
- Ignoring the uniform 32-dimensional action space requirement, leading to incompatible evaluation across different robot embodiments.

## Evidence (verbatim from paper)

> In this setting, we utilize all datasets to train a single model jointly and directly evaluate it on multiple benchmarks, without any additional fine-tuning on benchmark-specific datasets. Specifically, we select LIBERO, SimplerEnv, RoboTwin 2.0, and RoboCasa-GR1 as the unified benchmark suite and train the model on the combined training sets of these benchmarks. SR represents success rate, and score represents progress score.

## Citation

```bibtex
@misc{ye2026starvla,
  title={StarVLA-$\alpha$: Reducing Complexity in Vision-Language-Action Systems},
  author={Ye et al. (2026)},
  year={2026},
  note={arXiv:2604.11757}
}
```

- arXiv: 2604.11757