Graspclutter6d Eval

---
name: graspclutter6d-eval
description: Evaluates robotic perception and manipulation capabilities in highly cluttered, real-world environments. It benchmarks instance segmentation, 6D object pose estimation, and 6-DoF grasp detection under varying levels of occlusion and scene complexity. Use when the user wants to benchmark on GraspClutter6D, or asks about evaluating this task. Reports Grasp Success Rate (GSR).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.06866
  bibtex_key: back2025graspclutter6d
  confidence: high
---

# graspclutter6d-eval

> GraspClutter6D: A Large-scale Real-world Dataset for Robust Perception and Grasping in Cluttered Scenes — Back et al. (2025) (arXiv:2504.06866, 2025)

## What this evaluates

Evaluates robotic perception and manipulation capabilities in highly cluttered, real-world environments. It benchmarks instance segmentation, 6D object pose estimation, and 6-DoF grasp detection under varying levels of occlusion and scene complexity.

## Datasets

- **GraspClutter6D** — total 1000; splits: train (413), test (235)

## Metrics

- `Grasp Success Rate (GSR)` **(primary)** — range: percent
  - Ratio of successful lifts without slippage to total attempts.
- `Declutter Rate (DR)` — range: percent
  - Average ratio of objects cleared per round.
- `COCO AP` — range: [0, 1]
  - Average Precision for instance segmentation, computed using standard COCO evaluation metrics.
- `ADD-S` — range: [0, 1]
  - Area under the accuracy-threshold curve up to 10 cm for 6D pose estimation.
- `APμ` — range: [0, 1]
  - Average success rate of the top 50 predicted grasps across friction coefficients μ from 0.2 to 1.2 at 0.2 intervals, using the force-closure metric.

## Input / output format

**Input**: RGB-D images or point clouds of cluttered scenes (table, shelf, bin) with varying object counts (5-15) and occlusion levels.

**Output**: Predicted 6-DoF grasps, segmentation masks, or 6D object poses.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    successful = sum(1 for g in predictions if g.lift_success)
    cleared = sum(1 for obj in gold if obj.cleared)
    GSR = successful / len(predictions)
    DR = cleared / len(gold)
    AP = coco_compute_ap(predictions.masks, gold.masks)
    ADD_S = compute_auc(predictions.poses, gold.poses, threshold=0.1)
    AP_mu = mean([compute_grasp_ap(predictions.grasps, mu) for mu in np.arange(0.2, 1.21, 0.2)])
    return GSR, DR, AP, ADD_S, AP_mu
```

## Common pitfalls

- Models are evaluated only on foreground grasps (≤5cm from target objects) to remove background-induced ambiguity.
- Occlusion levels are explicitly stratified (low: v≥0.9, medium: 0.9>v≥0.6, high: 0.6>v) and must be reported separately.
- Grasp detection AP is averaged across friction coefficients (μ=0.2 to 1.2), not just a single value.

## Evidence (verbatim from paper)

> We compute the standard average precision (APμ), which measures the average success rate of the top 50 predicted grasps at friction coefficient μ using the force-closure metric. We report AP as a primary metric, averaging APμ across friction coefficients from 0.2 to 1.2 at 0.2 intervals.

## Citation

```bibtex
@misc{back2025graspclutter6d,
  title={GraspClutter6D: A Large-scale Real-world Dataset for Robust Perception and Grasping in Cluttered Scenes},
  author={Back et al. (2025)},
  year={2025},
  note={arXiv:2504.06866}
}
```

- arXiv: 2504.06866