Jigsaw Puzzle Tiling Eval

---
name: jigsaw-puzzle-tiling-eval
description: Evaluates object detection, recognition, and precise tiling assembly capabilities for modular robotic manipulation. It tests the pipeline's ability to segment oddly shaped pieces, recognize their identity, and assemble them with high spatial accuracy. Use when the user wants to benchmark on Jigsaw puzzle set, or asks about evaluating this task. Reports score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2005.02588
  bibtex_key: wan2020deepclaw
  confidence: high
---

# jigsaw-puzzle-tiling-eval

> DeepClaw: A Robotic Hardware Benchmarking Platform for Learning Object Manipulation — Wan et al. (2020) (arXiv:2005.02588, 2020)

## What this evaluates

Evaluates object detection, recognition, and precise tiling assembly capabilities for modular robotic manipulation. It tests the pipeline's ability to segment oddly shaped pieces, recognize their identity, and assemble them with high spatial accuracy.

## Datasets

- **Jigsaw puzzle set** — total 4; splits: (unstated)

## Metrics

- `score` **(primary)** — range: [0, 1]
  - Area rate defined as A_standard / A_actual, where A_standard is the area of perfectly aligned pieces and A_actual is the area of the minimum bounding box of the assembled pieces.
- `IoU` — range: [0, 1]
  - Intersection over Union for segmentation accuracy.
- `AP` — range: [0, 1]
  - Average Precision for object recognition accuracy.

## Input / output format

**Input**: RGB image and depth map of randomly placed jigsaw pieces on a table.

**Output**: Bounding box, piece identity, grasp orientation, and placement coordinates.

## Scoring recipe

```python
def score_jigsaw(predictions, gold):
    A_standard = 10.2 * 10.2
    A_actual = compute_min_bounding_box_area(predictions)
    score = A_standard / A_actual
    iou = compute_iou(predictions, gold)
    ap = compute_ap(predictions, gold)
    return score, iou, ap
```

## Common pitfalls

- Uses a suction cup instead of a standard parallel gripper due to the 5mm thickness of the pieces.
- Training data for segmentation was small (~80 images) and collected on a specific camera (D435), causing potential domain gaps on other sensors.

## Evidence (verbatim from paper)

> For performance assessment of the overall task, we report a 2D measurement of the area rate defined by score = A_standard / A_actual to evaluate the task completion... we report the IoU for segmentation, average precision (AP) for recognition, the success rate of picking and time cost of a pick and place sub-task

## Citation

```bibtex
@misc{wan2020deepclaw,
  title={DeepClaw: A Robotic Hardware Benchmarking Platform for Learning Object Manipulation},
  author={Wan et al. (2020)},
  year={2020},
  note={arXiv:2005.02588}
}
```

- arXiv: 2005.02588