Claw Machine Bin Clearing Eval

---
name: claw-machine-bin-clearing-eval
description: Measures end-to-end robotic manipulation performance and grasp robustness by clearing a bin of soft objects using a learned policy. It evaluates the ability to predict optimal grasping poses from RGB-D inputs and execute them across different hardware platforms. Use when the user wants to benchmark on Soft toy bin-clearing set, or asks about evaluating this task. Reports r_success.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2005.02588
  bibtex_key: wan2020deepclaw
  confidence: high
---

# claw-machine-bin-clearing-eval

> DeepClaw: A Robotic Hardware Benchmarking Platform for Learning Object Manipulation — Wan et al. (2020) (arXiv:2005.02588, 2020)

## What this evaluates

Measures end-to-end robotic manipulation performance and grasp robustness by clearing a bin of soft objects using a learned policy. It evaluates the ability to predict optimal grasping poses from RGB-D inputs and execute them across different hardware platforms.

## Datasets

- **Soft toy bin-clearing set** — total 8; splits: (unstated)

## Metrics

- `r_success` **(primary)** — range: [0, 1]
  - Total picking success rate, calculated as 8 divided by the total number of grasp attempts required to clear the bin.
- `t_pick` — range: seconds
  - Robot arm execution time for a single pick task, excluding gripper closing time. Averaged over 10 runs.

## Input / output format

**Input**: RGB image and depth map of the bin containing soft toys.

**Output**: Grasp pose (x, y, z) and orientation (theta) for vertical grasps.

## Scoring recipe

```python
def score_claw(predictions, gold):
    attempts = count_grasp_attempts(predictions)
    success_rate = 8.0 / attempts
    pick_times = [measure_time(p) - gripper_close_time for p in predictions]
    avg_time = sum(pick_times) / len(pick_times)
    return success_rate, avg_time
```

## Common pitfalls

- Gripper closing time varies significantly across hardware (e.g., 0.80s vs 2.0s) and must be subtracted from t_pick.
- Grasps are strictly restricted to vertical orientation with only z-axis rotation allowed; non-compliant poses invalidate the attempt.

## Evidence (verbatim from paper)

> For performance assessment, we report robot arm execution time $t_{pick}$ for a single pick task, and the total picking success rate $r_{success}$ , which equals to eight over the number of total grasp attempts before clearing the bin. We performed the bin-clearing task on three robot cells and ten times on each robot cell.

## Citation

```bibtex
@misc{wan2020deepclaw,
  title={DeepClaw: A Robotic Hardware Benchmarking Platform for Learning Object Manipulation},
  author={Wan et al. (2020)},
  year={2020},
  note={arXiv:2005.02588}
}
```

- arXiv: 2005.02588