Stanford Orb Eval

---
name: stanford-orb-eval
description: Evaluates the ability of models to recover 3D geometry and surface material from images, and to synthesize novel views or relight objects in unseen real-world environments. It probes inverse rendering capabilities under natural, uncontrolled lighting conditions where ground-truth material is unavailable. Use when the user wants to benchmark on Stanford-ORB, or asks about evaluating this task. Reports Bidirectional Chamfer Distance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.16044
  bibtex_key: [REDACTED]
  confidence: high
---

# stanford-orb-eval

> Stanford-ORB: A Real-World 3D Object Inverse Rendering Benchmark — Kuang et al. (2023) (arXiv:2310.16044, 2023)

## What this evaluates

Evaluates the ability of models to recover 3D geometry and surface material from images, and to synthesize novel views or relight objects in unseen real-world environments. It probes inverse rendering capabilities under natural, uncontrolled lighting conditions where ground-truth material is unavailable.

## Datasets

- **Stanford-ORB** — total 2940; splits: train (60), test (10); repo https://github.com/StanfordORB/Stanford-ORB

## Metrics

- `Bidirectional Chamfer Distance` **(primary)** — range: other (meters)
  - Computes the average minimum squared distance between points in the predicted mesh and the ground-truth scan, and vice versa. Measures geometric fidelity of the recovered 3D shape.
- `Depth/Normal Map Error` — range: [0, 1]
  - Compares predicted depth and normal maps against those generated from ground-truth 3D scans using L1 or angular error metrics.
- `Rendered Image Reconstruction Error` — range: [0, 1]
  - Renders the predicted geometry and material under ground-truth lighting (for relighting) or input lighting (for novel view), then compares the output image to the captured ground-truth image using standard image metrics (e.g., LPIPS, PSNR, SSIM).

## Input / output format

**Input**: Multi-view masked images (or single image) of a real-world object, optionally accompanied by environment lighting maps and camera poses.

**Output**: Predicted 3D mesh (or depth/normal maps) and surface material properties (e.g., BRDF parameters).

## Scoring recipe

```python
def score_geometry(pred_mesh, gt_mesh):
    cd = bidirectional_chamfer_distance(pred_mesh, gt_mesh)
    depth_err = l1_error(pred_depth, gt_depth)
    normal_err = 1 - cosine_similarity(pred_normal, gt_normal)
    return cd, depth_err, normal_err

def score_image(pred_mesh, pred_mat, gt_light, gt_img, cam_pose):
    rendered = render(pred_mesh, pred_mat, gt_light, cam_pose)
    img_err = l1_or_lpips(rendered, gt_img)
    return img_err
```

## Common pitfalls

- Material ground truth is unavailable in-the-wild, so evaluation relies on relighting quality as a proxy rather than direct material comparison.
- Real-world lighting changes during capture, requiring per-view environment maps solved from chrome ball reflections rather than a single static map.
- Automated segmentation masks (SAM/GrabCut) tend to underestimate object boundaries, requiring manual refinement for accurate evaluation.

## Evidence (verbatim from paper)

> We measure the quality of the geometry estimated from different methods by comparing the predicted depth maps and normal maps to those generated from the ground-truth scans, as well as directly computing the bidirectional Chamfer Distance between the predicted meshes and the scans. To evaluate the performance of a model, we render the object using the predicted geometry and materials and the ground-truth lighting of a new scene, and compare the rendered image with the captured ground-truth image from a given viewpoint.

## Citation

```bibtex
@misc{kuang2023stanfordorb,
  title={Stanford-ORB: A Real-World 3D Object Inverse Rendering Benchmark},
  author={Kuang et al. (2023)},
  year={2023},
  note={arXiv:2310.16044}
}
```

- arXiv: 2310.16044