Rio 3rscan Eval

---
name: rio-3rscan-eval
description: Evaluates a model's ability to match 3D object patches and re-localize object instances in dynamically changing indoor environments. It measures feature matching robustness and 6DoF pose estimation accuracy under partial observations and contextual shifts. Use when the user wants to benchmark on 3RScan, or asks about evaluating this task. Reports Recall <0.1m, 10°.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1908.06109
  bibtex_key: wald2019rio
  confidence: high
---

# rio-3rscan-eval

> RIO: 3D Object Instance Re-Localization in Changing Indoor Environments — Wald et al. (2019) (arXiv:1908.06109, 2019)

## What this evaluates

Evaluates a model's ability to match 3D object patches and re-localize object instances in dynamically changing indoor environments. It measures feature matching robustness and 6DoF pose estimation accuracy under partial observations and contextual shifts.

## Datasets

- **3RScan** — total ?; splits: test (-1)

## Metrics

- `Recall <0.1m, 10°` **(primary)** — range: percent
  - Percentage of object instances successfully aligned where translation error ≤ 0.1 m and rotation error ≤ 10°.
- `Recall <0.2m, 20°` — range: percent
  - Percentage of object instances successfully aligned where translation error ≤ 0.2 m and rotation error ≤ 20°.
- `MRE [deg]` — range: other
  - Median Rotation Error computed from the axis-angle representation of the predicted vs ground-truth rotation matrices.
- `MTE [m]` — range: other
  - Median Translation Error computed as the Euclidean distance between predicted and ground-truth translation vectors.

## Input / output format

**Input**: RGB-D patches (TSDF) around annotated keypoints on object instances in changing indoor scenes.

**Output**: Predicted 6DoF pose (rotation matrix R_p and translation vector t_p) for each object instance.

## Scoring recipe

```python
def compute_recall_and_errors(predictions, ground_truth, t_thresh=0.1, r_thresh=10.0):
    correct = 0
    r_errors = []
    t_errors = []
    for pred, gt in zip(predictions, ground_truth):
        t_delta = np.linalg.norm(pred['t'] - gt['t'])
        R_delta = np.dot(pred['R'].T, gt['R'])
        r_delta = np.arccos(np.clip((np.trace(R_delta) - 1) / 2, -1, 1)) * 180 / np.pi
        r_errors.append(r_delta)
        t_errors.append(t_delta)
        if t_delta <= t_thresh and r_delta <= r_thresh:
            correct += 1
    recall = correct / len(predictions)
    mre = np.median(r_errors)
    mte = np.median(t_errors)
    return recall, mre, mte
```

## Common pitfalls

- Symmetries of object instances are explicitly considered in the error computation, which must be accounted for when calculating rotation/translation deltas.
- The top-1 matching metric uses 50 randomly chosen negative patches per positive keypoint, differing from standard 1:1 matching.
- Evaluation is performed on the test set of 3RScan, which contains dynamically changing environments, not static scenes.

## Evidence (verbatim from paper)

> We evaluate the predicted rotation R_p and translation t_p against the ground truth annotation R_GT and t_GT according to equation [5] and [4]. An instance has successfully been aligned if the alignment error for the translation t_Δ and rotation R_Δ are lower than t≤0.1m, r≤10° or t≤0.2m, r≤20°. Please note that respective symmetry are considered in the error computation:

## Citation

```bibtex
@misc{wald2019rio,
  title={RIO: 3D Object Instance Re-Localization in Changing Indoor Environments},
  author={Wald et al. (2019)},
  year={2019},
  note={arXiv:1908.06109}
}
```

- arXiv: 1908.06109