Phocal Eval

---
name: phocal-eval
description: Evaluates category-level 6D object pose estimation on photometrically challenging objects (reflective, transparent, occluded). It tests both in-distribution generalization (seen objects) and out-of-distribution generalization (novel objects within the same category), comparing RGB-D and monocular approaches. Use when the user wants to benchmark on PhoCaL, or asks about evaluating this task. Reports 3D IoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2205.08811
  bibtex_key: wang2022phocal
  confidence: high
---

# phocal-eval

> PhoCaL: A Multi-Modal Dataset for Category-Level Object Pose Estimation with Photometrically Challenging Objects — Wang et al. (2022) (arXiv:2205.08811, 2022)

## What this evaluates

Evaluates category-level 6D object pose estimation on photometrically challenging objects (reflective, transparent, occluded). It tests both in-distribution generalization (seen objects) and out-of-distribution generalization (novel objects within the same category), comparing RGB-D and monocular approaches.

## Datasets

- **PhoCaL** — total 24; splits: train (-1), test (-1)

## Metrics

- `3D IoU` **(primary)** — range: percent
  - Intersection over Union (IoU) between predicted and ground-truth 3D bounding boxes. Mean Average Precision (mAP) is computed at IoU thresholds of 25% and 50% across all categories and images.

## Input / output format

**Input**: RGB or RGB-D image sequences containing household objects under varying illumination, occlusion, and partial visibility.

**Output**: 6D object pose (rotation and translation), scale, and 3D bounding box or point cloud shape.

## Scoring recipe

```python
def compute_3d_iou(pred_box, gt_box):
    intersection = volume_of_intersection(pred_box, gt_box)
    union = volume_of_union(pred_box, gt_box)
    return intersection / union if union > 0 else 0.0

def compute_mAP(predictions, ground_truths, iou_thresh=0.25):
    matches = []
    for pred, gt in zip(predictions, ground_truths):
        if compute_3d_iou(pred, gt) >= iou_thresh:
            matches.append(1.0)
        else:
            matches.append(0.0)
    return sum(matches) / len(matches) * 100
```

## Common pitfalls

- Depth sensor artifacts on reflective/transparent objects cause significant performance drops for RGB-D methods, independent of the pose algorithm itself.
- Monocular methods suffer severe performance degradation on novel objects without extensive synthetic pretraining.
- Deformable objects (e.g., empty boxes) are difficult to annotate accurately and are not well-represented in the benchmark.

## Evidence (verbatim from paper)

> The evaluation metric is the intersection over union (IoU) result with a threshold of 25% and 50%. The mean average precision (mAP) for 3D IoU at 25% is 43.34%.

## Citation

```bibtex
@misc{wang2022phocal,
  title={PhoCaL: A Multi-Modal Dataset for Category-Level Object Pose Estimation with Photometrically Challenging Objects},
  author={Wang et al. (2022)},
  year={2022},
  note={arXiv:2205.08811}
}
```

- arXiv: 2205.08811