Panoptic Radiance Field Eval

---
name: panoptic-radiance-field-eval
description: Evaluates a NeRF-based method's ability to jointly reconstruct 3D scene geometry, appearance, and panoptic segmentation (semantic + instance) from multi-view images. It probes 3D consistency, boundary handling across indoor/outdoor scales, and robustness to pseudo-label noise via perceptual priors. Use when the user wants to benchmark on Replica, HyperSim, ScanNet, KITTI-360, or asks about evaluating this task. Reports mIOU.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.04529
  bibtex_key: li2024inplacepanoptic
  confidence: high
---

# panoptic-radiance-field-eval

> In-Place Panoptic Radiance Field Segmentation with Perceptual Prior for 3D Scene Understanding — Li et al. (2024) (arXiv:2410.04529, 2024)

## What this evaluates

Evaluates a NeRF-based method's ability to jointly reconstruct 3D scene geometry, appearance, and panoptic segmentation (semantic + instance) from multi-view images. It probes 3D consistency, boundary handling across indoor/outdoor scales, and robustness to pseudo-label noise via perceptual priors.

## Datasets

- **Replica** — total ?; splits: test (-1)
- **HyperSim** — total ?; splits: test (-1)
- **ScanNet** — total ?; splits: test (-1)
- **KITTI-360** — total ?; splits: test (-1)

## Metrics

- `PSNR` — range: dB
  - Quantifies reconstructed luminance quality by measuring the difference between the rendered color image and the ground truth image.
- `mIOU` **(primary)** — range: percent
  - Evaluates semantic segmentation accuracy by calculating the intersection over union between the rendered semantic map and the ground truth semantic map, averaged across classes.
- `PQ${}^{	ext{scene}}$` — range: percent
  - Assesses panoptic segmentation quality within the target scene by comparing the degree of alignment between the rendered semantic and instance maps with the supervised semantic and instance maps.
- `SQ${}^{	ext{scene}}$` — range: percent
  - Measures segmentation accuracy of panoptic segmentation by evaluating the differences in segmentation between the rendered semantic and instance maps and the supervised semantic and instance maps.
- `RQ${}^{	ext{scene}}$` — range: percent
  - Determines retrieval effectiveness of panoptic segmentation by comparing the retrieval discrepancies between the rendered semantic and instance maps and the supervised semantic and instance maps.

## Input / output format

**Input**: Multi-view RGB images with corresponding camera poses, used to render 2D semantic and instance maps from a 3D implicit scene representation.

**Output**: Rendered 2D semantic map and 2D instance map per viewpoint, aligned with ground truth supervision.

## Scoring recipe

```python
def compute_miou(pred_semantic, gt_semantic, num_classes):
    ious = []
    for c in range(num_classes):
        pred_c = (pred_semantic == c)
        gt_c = (gt_semantic == c)
        intersection = np.sum(pred_c & gt_c)
        union = np.sum(pred_c | gt_c)
        iou = intersection / union if union > 0 else 1.0
        ious.append(iou)
    return np.mean(ious) * 100
```

## Common pitfalls

- Resolution varies across datasets (512x512 for Replica/HyperSim, 256x256 for ScanNet, 1408x376 for KITTI-360), which directly impacts PSNR and mIOU scores.
- The 'Void' category is explicitly included in the semantic class count (22 indoor, 21 outdoor) and must be accounted for in IoU calculations.
- Scene-level metrics (PQ_scene, SQ_scene, RQ_scene) evaluate alignment between rendered and supervised maps per scene, not per-instance, which differs from standard panoptic benchmarks.

## Evidence (verbatim from paper)

> The proposed method is primarily assessed using the following evaluation metrics, where an upward arrow ($\uparrow$) signifies that higher values denote better performance, and vice versa: Peak Signal-to-Noise Ratio (PSNR$\uparrow$): This metric quantifies the quality of the reconstructed luminance by measuring the difference between the rendered color image and the ground truth image. Mean Intersection over Union (mIOU$\uparrow$): This metric evaluates the accuracy of semantic segmentation by calculating the intersection over union between the rendered semantic map and the ground truth semantic map.

## Citation

```bibtex
@misc{li2024inplacepanoptic,
  title={In-Place Panoptic Radiance Field Segmentation with Perceptual Prior for 3D Scene Understanding},
  author={Li et al. (2024)},
  year={2024},
  note={arXiv:2410.04529}
}
```

- arXiv: 2410.04529