Robocoin Eval

---
name: robocoin-eval
description: Evaluates the effectiveness of a large-scale bimanual manipulation dataset and a hierarchical annotation framework on Vision-Language-Action models across different robotic platforms. It probes the models' ability to generalize across task complexities, leverage multi-resolution annotations, and benefit from trajectory quality filtering. Use when the user wants to benchmark on RoboCOIN, or asks about evaluating this task. Reports success_rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.17441
  bibtex_key: wu2025robocoin
  confidence: high
---

# robocoin-eval

> RoboCOIN: An Open-Sourced Bimanual Robotic Data COllection for INtegrated Manipulation — Wu et al. (2025) (arXiv:2511.17441, 2025)

## What this evaluates

Evaluates the effectiveness of a large-scale bimanual manipulation dataset and a hierarchical annotation framework on Vision-Language-Action models across different robotic platforms. It probes the models' ability to generalize across task complexities, leverage multi-resolution annotations, and benefit from trajectory quality filtering.

## Datasets

- **RoboCOIN** — total 180000; splits: test (-1); repo https://github.com/FlagOpen/CoRobot

## Metrics

- `success_rate` **(primary)** — range: percent
  - Calculated as the ratio of successfully completed trials to the total number of trials, reported as a percentage. Success is determined by task completion criteria defined per task (e.g., object placed correctly, drawer closed).
- `trajectory_filtering_rate` — range: percent
  - Percentage of trajectories removed by the RTML framework based on global, phase-wise, or metric-wise constraints.

## Input / output format

**Input**: Multi-view camera images, natural language task instructions, and hierarchical annotations (trajectory-level scene context, segment-level subtask instructions, frame-level state descriptions) fed into VLA models.

**Output**: Continuous motor commands / action sequences for robotic arms and grippers.

## Scoring recipe

```python
def compute_success_rate(successful_trials, total_trials):
    return (successful_trials / total_trials) * 100.0

def compute_filtering_rate(removed_trajectories, total_trajectories):
    return (removed_trajectories / total_trajectories) * 100.0
```

## Common pitfalls

- RTML filtering relies on empirically set thresholds that may not generalize across all scenarios or robot morphologies.
- Success rates vary significantly with task complexity and embodiment, so comparing results across different platforms or task grids requires careful normalization.
- The evaluation does not include mixed-embodiment training or cross-embodiment policy transfer, limiting claims about generalization.

## Evidence (verbatim from paper)

> As shown in Figure [8](c), the four configurations exhibited a clear progressive improvement in average success rates. GR00T-Coarse showed a modest 3% gain over the GR00T-Raw baseline, indicating that basic global filtering provides limited benefits. In contrast, GR00T-Fine achieved a more substantial 16% improvement, underscoring the significant impact of phase-level constraints on trajectory quality.

## Citation

```bibtex
@misc{wu2025robocoin,
  title={RoboCOIN: An Open-Sourced Bimanual Robotic Data COllection for INtegrated Manipulation},
  author={Wu et al. (2025)},
  year={2025},
  note={arXiv:2511.17441}
}
```

- arXiv: 2511.17441