Rovi Manipulation Eval

---
name: rovi-manipulation-eval
description: This benchmark probes a model's ability to comprehend hand-drawn, object-centric visual instructions (arrows, circles, colors) and translate them into precise spatiotemporal action plans for robotic manipulation. It evaluates both high-level task reasoning and low-level execution accuracy in cluttered, unseen environments. Use when the user wants to benchmark on RoVI Book dataset, SIMPLER, or asks about evaluating this task. Reports action success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.00693
  bibtex_key: li2025rovi
  confidence: high
---

# rovi-manipulation-eval

> Robotic Visual Instruction — Yanbang Li et al. (2025) (arXiv:2505.00693, 2025)

## What this evaluates

This benchmark probes a model's ability to comprehend hand-drawn, object-centric visual instructions (arrows, circles, colors) and translate them into precise spatiotemporal action plans for robotic manipulation. It evaluates both high-level task reasoning and low-level execution accuracy in cluttered, unseen environments.

## Datasets

- **RoVI Book dataset** — total 15000; splits: train (-1)
- **SIMPLER** — total ?; splits: test (3)

## Metrics

- `action success rate` **(primary)** — range: percent
  - Percentage of tasks that successfully meet the defined semantic goals over 10 trials per task.
- `spatiotemporal alignment` — range: other
  - A 6-point Likert scale rating the consistency of movement trajectories and the alignment of an object’s final spatial state with semantic goals.
- `task and planning success rate` — range: percent
  - Human-annotated success rate measuring the accuracy of VLM language responses in comprehending task definitions and decomposing complex tasks into sequential sub-goals.
- `Euclidean distance error` — range: other
  - Mean pixel distance between predicted keypoint locations and ground-truth locations.
- `mAP` — range: [0, 1]
  - Mean Average Precision calculated at a 50-pixel localization threshold for keypoint extraction.

## Input / output format

**Input**: RGB-D camera observations of the environment paired with hand-drawn RoVI sketches containing symbolic elements (arrows, circles, colors, numbers) to encode spatial and temporal constraints.

**Output**: Language responses describing task comprehension and sequential sub-goal planning, or Python functions for low-level robotic control.

## Scoring recipe

```python
def score_action_success(predictions, gold, trials=10):
    success_count = 0
    for trial in range(trials):
        if predictions[trial].meets_goal(gold.semantic_goal):
            success_count += 1
    return (success_count / trials) * 100

def score_alignment(predictions, gold):
    return likert_scale(predictions.trajectory, predictions.final_state, gold)

def score_task_planning(predictions, gold):
    return human_eval(predictions.language_response, gold.plan)
```

## Common pitfalls

- Assuming standard language instructions are sufficient; RoVI requires pixel-level precision via symbolic drawing, which open-vocabulary detectors often miss.
- Evaluating only end-state accuracy without considering trajectory consistency; the protocol explicitly requires spatiotemporal alignment scoring via Likert scale.
- Using models without fine-tuning on the RoVI Book dataset for multi-step tasks; small models fail to comprehend RoVI without LoRA fine-tuning.

## Evidence (verbatim from paper)

> We report two metrics for assessing manipulation execution: action success rate, measuring the percentage of tasks that meet defined goals, and spatiotemporal alignment, evaluating the consistency of movement trajectories and the alignment of an object’s final spatial state with semantic goals. A 6-point Likert scale is used for assessment (details in the appendix). Each task is evaluated over 10 trials.

## Citation

```bibtex
@misc{li2025rovi,
  title={Robotic Visual Instruction},
  author={Yanbang Li et al. (2025)},
  year={2025},
  note={arXiv:2505.00693}
}
```

- arXiv: 2505.00693