Smolvla Robotics Eval

---
name: smolvla-robotics-eval
description: Evaluates a vision-language-action model's ability to perform robotic manipulation tasks in both simulated and real-world environments. It probes visuomotor policy generalization, fine-grained task decomposition handling, and the impact of pretraining and inference modes on success rates. Use when the user wants to benchmark on LIBERO, Meta-World, SO100 Real-World Tasks, SO101 Real-World Tasks, or asks about evaluating this task. Reports Success Rate (SR).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.01844
  bibtex_key: shukor2025smolvla
  confidence: high
---

# smolvla-robotics-eval

> SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics — Mustafa Shukor et al. (2025) (arXiv:2506.01844, 2025)

## What this evaluates

Evaluates a vision-language-action model's ability to perform robotic manipulation tasks in both simulated and real-world environments. It probes visuomotor policy generalization, fine-grained task decomposition handling, and the impact of pretraining and inference modes on success rates.

## Datasets

- **LIBERO** — total 1693; splits: eval (-1)
- **Meta-World** — total 2500; splits: eval (-1)
- **SO100 Real-World Tasks** — total 150; splits: eval (-1)
- **SO101 Real-World Tasks** — total 50; splits: eval (-1)

## Metrics

- `Success Rate (SR)` **(primary)** — range: percent
  - Average score over 10 trials per task. In simulation, binary: 1 if fully completed, 0 otherwise. In real-world, fine-grained subtask scoring (e.g., 0.5 for grasp + 0.5 for place; max 1.0). Reported as percentage.

## Input / output format

**Input**: RGB images (resized to 512×512), robot sensorimotor states, and natural language instructions.

**Output**: Action chunks of n=50 continuous actions predicted via flow matching.

## Scoring recipe

```python
def compute_sr(trials, mode):
    scores = []
    for trial in trials:
        if mode == 'simulation':
            scores.append(1.0 if trial.completed else 0.0)
        else:
            score = 0.0
            if trial.grasp: score += 0.5
            if trial.place: score += 0.5
            # Sorting uses 0.25 per subtask (grasp, match1, match2, grasp2)
            scores.append(min(score, 1.0))
    return sum(scores) / len(scores) * 100
```

## Common pitfalls

- Assuming real-world success rates use binary completion; they actually use fine-grained subtask scoring that must be summed to a maximum of 1.0.
- Overlooking that the SO101 benchmark is strictly out-of-distribution for pretraining, as the model was explicitly not pretrained on any SO101 data.
- Confusing synchronous inference (standard robotics eval) with asynchronous inference, which trades some success rate for significantly lower latency and higher task throughput.

## Evidence (verbatim from paper)

> Evaluation metrics. We report success rate (SR) as the primary metric across all benchmarks. For simulation-based evaluations, SR is binary-set to 1 if the task is successfully completed, and 0 otherwise. For real-world evaluations, we adopt a more fine-grained scoring approach by decomposing each task into subtasks. For example, in the Pick-and-Place task, we assign a score of 0.5 for successfully picking the cube and an additional 0.5 for correctly placing it into the target container.

## Citation

```bibtex
@misc{shukor2025smolvla,
  title={SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics},
  author={Mustafa Shukor et al. (2025)},
  year={2025},
  note={arXiv:2506.01844}
}
```

- arXiv: 2506.01844