Bop 6d Pose Refinement Eval

---
name: bop-6d-pose-refinement-eval
description: Evaluates the ability to refine 6D object poses in cluttered real-world scenes using RGB or RGB-D inputs. It probes generalization to novel objects by measuring pose accuracy against ground truth under symmetry-aware error metrics. Use when the user wants to benchmark on LM-O, T-LESS, TUD-L, IC-BIN, ITODD, HomebrewdDB, YCB-V, or asks about evaluating this task. Reports Average Recall (AR).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.11510
  bibtex_key: moon2024genflow
  confidence: high
---

# bop-6d-pose-refinement-eval

> GenFlow: Generalizable Recurrent Flow for 6D Pose Refinement of Novel Objects — Moon et al. (2024) (arXiv:2403.11510, 2024)

## What this evaluates

Evaluates the ability to refine 6D object poses in cluttered real-world scenes using RGB or RGB-D inputs. It probes generalization to novel objects by measuring pose accuracy against ground truth under symmetry-aware error metrics.

## Datasets

- **LM-O** — total ?; splits: test (-1)
- **T-LESS** — total ?; splits: test (-1)
- **TUD-L** — total ?; splits: test (-1)
- **IC-BIN** — total ?; splits: test (-1)
- **ITODD** — total ?; splits: test (-1)
- **HomebrewdDB** — total ?; splits: test (-1)
- **YCB-V** — total ?; splits: test (-1)

## Metrics

- `Average Recall (AR)` **(primary)** — range: percent
  - AR is computed by averaging recall over multiple correctness thresholds for three pose-error functions: Visible Surface Discrepancy (VSD), Maximum Symmetry-Aware Surface Distance (MSSD), and Maximum Symmetry-Aware Projection Distance (MSPD). A pose is considered correct if any error function falls below its threshold.

## Input / output format

**Input**: RGB or RGB-D images of cluttered real-world scenes containing multiple objects, along with 2D bounding box detections and an initial 6D pose hypothesis for each object.

**Output**: Refined 6D pose (rotation matrix and translation vector) for each detected object instance.

## Scoring recipe

```python
def compute_ar(predictions, ground_truths):
    correct = 0
    for pred, gt in zip(predictions, ground_truths):
        vsd_err = compute_vsd(pred.pose, gt.pose)
        mssd_err = compute_mssd(pred.pose, gt.pose)
        mspd_err = compute_mspd(pred.pose, gt.pose)
        if (vsd_err < VSD_TOL or mssd_err < MSSD_THRESH or mspd_err < MSPD_THRESH):
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Failing to match the multi-hypothesis strategy settings (n=10 hypotheses, 5 outer updates, 8 inner updates) when comparing against baselines like MegaPose.
- Confusing RGB-only and RGB-D input settings, which require different evaluation splits and yield different performance baselines.
- Not adhering to the standard BOP challenge evaluation protocol for pose error functions and threshold averaging.

## Evidence (verbatim from paper)

> Following the evaluation methodology of BOP challenge [[28]], we report the average recall (AR) considering three pose-error functions: Visible Surface Discrepancy (VSD), Maximum Symmetry-Aware Surface Distance (MSSD), and Maximum Symmetry-Aware Projection Distance (MSPD). In consideration of the global symmetry information, MSSD computes the maximum distance between the estimated camera-space coordinates and corresponding ground truth. In contrast, MSPD computes the maximum distance between the estimated image-space coordinates and the corresponding ground truth. Concerning each pose-error function, the AR is obtained by averaging recall calculated for multiple settings of the correctness thresholds (and misalignment tolerances for VSD).

## Citation

```bibtex
@misc{moon2024genflow,
  title={GenFlow: Generalizable Recurrent Flow for 6D Pose Refinement of Novel Objects},
  author={Moon et al. (2024)},
  year={2024},
  note={arXiv:2403.11510}
}
```

- arXiv: 2403.11510