Libero Plus Eval

---
name: libero-plus-eval
description: Evaluates the robustness of vision-language-action (VLA) models under realistic perturbations across seven dimensions (camera, robot, language, light, background, noise, layout). It probes visual shift tolerance, kinematic reasoning, and linguistic robustness by measuring success rates on a curated set of non-trivial tasks. Use when the user wants to benchmark on LIBERO-Plus, or asks about evaluating this task. Reports success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.13626
  bibtex_key: [REDACTED]
  confidence: high
---

# libero-plus-eval

> LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models — Fei et al. (2025) (arXiv:2510.13626, 2025)

## What this evaluates

Evaluates the robustness of vision-language-action (VLA) models under realistic perturbations across seven dimensions (camera, robot, language, light, background, noise, layout). It probes visual shift tolerance, kinematic reasoning, and linguistic robustness by measuring success rates on a curated set of non-trivial tasks.

## Datasets

- **LIBERO-Plus** — total 10030; splits: test (10030)

## Metrics

- `success rate` **(primary)** — range: [0, 1]
  - Calculated as the fraction of tasks successfully completed by the model. Tasks are stratified into five difficulty levels (L1–L5) based on how many of four baseline models solved them.

## Input / output format

**Input**: Perturbed visual observations (varying camera viewpoint, lighting, background, sensor noise, and layout), language instructions, and robot initial state.

**Output**: Robot action sequence or policy output.

## Scoring recipe

```python
def compute_success_rate(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold) if len(gold) > 0 else 0.0
```

## Common pitfalls

- Tasks are pre-filtered to remove ceiling effects, so performance reflects only non-trivial difficulty.
- Difficulty levels (L1-L5) are relative to four specific baseline models, not absolute ground truth.
- Language perturbations yield negligible performance drops, suggesting models rely on visual cues rather than true language understanding.

## Evidence (verbatim from paper)

> Figure [8] shows the success rates of each model across the five difficulty levels.

## Citation

```bibtex
@misc{fei2025liberoplus,
  title={LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models},
  author={Fei et al. (2025)},
  year={2025},
  note={arXiv:2510.13626}
}
```

- arXiv: 2510.13626