Siu3r Scanet Eval

---
name: siu3r-scanet-eval
description: Evaluates simultaneous 3D scene reconstruction and multi-task scene understanding on sparse multi-view images. It probes geometric accuracy, novel view synthesis quality, and cross-view consistent segmentation across semantic, instance, panoptic, and text-referred tasks. Use when the user wants to benchmark on ScanNet, or asks about evaluating this task. Reports mIoU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.02705
  bibtex_key: xu2025siu3r
  confidence: high
---

# siu3r-scanet-eval

> SIU3R: Simultaneous Scene Understanding and 3D Reconstruction Beyond Feature Alignment — Qi Xu et al. (2025) (arXiv:2507.02705, 2025)

## What this evaluates

Evaluates simultaneous 3D scene reconstruction and multi-task scene understanding on sparse multi-view images. It probes geometric accuracy, novel view synthesis quality, and cross-view consistent segmentation across semantic, instance, panoptic, and text-referred tasks.

## Datasets

- **ScanNet** — total ?; splits: train (-1), val (-1)

## Metrics

- `AbsRel` — range: other
  - Mean absolute relative difference between predicted and ground truth depth: mean(|pred - gt| / gt).
- `RMSE` — range: other
  - Root mean square error of depth predictions: sqrt(mean((pred - gt)^2)).
- `PSNR` — range: other
  - Peak signal-to-noise ratio between rendered and ground truth images, computed from mean squared error.
- `SSIM` — range: [0, 1]
  - Structural similarity index measuring luminance, contrast, and structure between images.
- `LPIPS` — range: [0, 1]
  - Learned perceptual image patch similarity using deep feature distances.
- `mIoU` **(primary)** — range: [0, 1]
  - Mean intersection over union across classes. Computed with global IDs as ground truths to penalize cross-view inconsistencies.
- `mAP` — range: [0, 1]
  - Mean average precision across instance categories, computed with global IDs.
- `PQ` — range: [0, 1]
  - Panoptic quality combining segmentation quality and detection performance, computed with global IDs.

## Input / output format

**Input**: Sparse multi-view RGB images of a scene (context views).

**Output**: Reconstructed 3D representation (e.g., 3D Gaussians), rendered novel-view images, and 2D/3D segmentation masks (semantic, instance, panoptic, text-referred).

## Scoring recipe

```python
def compute_metrics(pred_depth, gt_depth, pred_img, gt_img, pred_masks, gt_masks):
    absrel = np.mean(np.abs(pred_depth - gt_depth) / gt_depth)
    rmse = np.sqrt(np.mean((pred_depth - gt_depth)**2))
    mse = np.mean((pred_img - gt_img)**2)
    psnr = 10 * np.log10(1.0 / mse)
    ssim = compute_ssim(pred_img, gt_img)
    lpips = compute_lpips(pred_img, gt_img)
    miou = compute_mIoU(pred_masks, gt_masks, use_global_ids=True)
    map_ = compute_mAP(pred_masks, gt_masks, use_global_ids=True)
    pq = compute_PQ(pred_masks, gt_masks, use_global_ids=True)
    return {'AbsRel': absrel, 'RMSE': rmse, 'PSNR': psnr, 'SSIM': ssim, 'LPIPS': lpips, 'mIoU': miou, 'mAP': map_, 'PQ': pq}
```

## Common pitfalls

- Confusing the 2D-only evaluation (on context views) with the 3D-aware evaluation (projecting 3D masks to novel views).
- Using local instance IDs instead of global IDs for mAP/PQ, which fails to penalize cross-view segmentation inconsistencies.
- Assuming ground-truth 3D segmentation labels exist; novel views are used instead due to method-dependent 3D structures.

## Evidence (verbatim from paper)

> For 3D reconstruction, we evaluate the performance from two aspects: depth estimation and novel view synthesis, using depth accuracy metrics (i.e., AbsRel and RMSE) and image quality metrics (i.e., PSNR, SSIM and LPIPS), respectively. For scene understanding, we employ distinct evaluation protocols for 2D-based and 3D-based approaches... As for different understanding tasks, we employ mIoU for semantic and text-referred segmentation, mAP for instance segmentation, and PQ for panoptic segmentation, where all metrics are computed with global IDs as ground truths to penalize inconsistencies across views.

## Citation

```bibtex
@misc{xu2025siu3r,
  title={SIU3R: Simultaneous Scene Understanding and 3D Reconstruction Beyond Feature Alignment},
  author={Qi Xu et al. (2025)},
  year={2025},
  note={arXiv:2507.02705}
}
```

- arXiv: 2507.02705