Omni Recon Downstream Eval

---
name: omni-recon-downstream-eval
description: Evaluates a general-purpose NeRF framework on downstream 3D tasks, including real-time novel view synthesis, parameter-efficient 3D scene understanding, and text-guided 3D editing. It probes the model's ability to generalize to unseen scenes and adapt to various geometric and appearance tasks with minimal fine-tuning. Use when the user wants to benchmark on DTU, ScanNet, or asks about evaluating this task. Reports PSNR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2403.11131
  bibtex_key: fu2024omnirecon
  confidence: high
---

# omni-recon-downstream-eval

> Omni-Recon: Harnessing Image-based Rendering for General-Purpose Neural Radiance Fields — Yonggan Fu et al. (arXiv:2403.11131, 2024)

## What this evaluates

Evaluates a general-purpose NeRF framework on downstream 3D tasks, including real-time novel view synthesis, parameter-efficient 3D scene understanding, and text-guided 3D editing. It probes the model's ability to generalize to unseen scenes and adapt to various geometric and appearance tasks with minimal fine-tuning.

## Datasets

- **DTU** — total ?; splits: test (-1)
- **ScanNet** — total ?; splits: train (-1), test (-1)

## Metrics

- `PSNR` **(primary)** — range: other
  - Peak Signal-to-Noise Ratio computed between rendered novel-view images and ground truth images. Higher is better.
- `FPS` — range: other
  - Frames Per Second measured on an NVIDIA A5000 GPU to assess rendering speed.
- `Sem. mIoU` — range: percent
  - Mean Intersection over Union for semantic segmentation predictions on 3D scenes.
- `Sem. Total Acc` — range: percent
  - Overall accuracy of semantic segmentation predictions.
- `Edge error` — range: other
  - Error metric for edge detection (lower is better).
- `Keypoint error` — range: other
  - Error metric for 2D and 3D keypoint detection (lower is better).

## Input / output format

**Input**: For rendering: novel view camera poses and source views to extract features and bake into a mesh. For understanding: 3D scene data (point clouds/meshes) from ScanNet with semantic labels, edges, and keypoints.

**Output**: Rendered novel view images and FPS; semantic segmentation maps, edge maps, and 3D keypoint coordinates.

## Scoring recipe

```python
def score(predictions, gold):
    psnr = compute_psnr(predictions.rendered_images, gold.images)
    fps = measure_fps(predictions.rendered_images, gold.cameras)
    mIoU = compute_mIoU(predictions.semantic_maps, gold.labels)
    acc = compute_accuracy(predictions.semantic_maps, gold.labels)
    edge_err = compute_error(predictions.edge_maps, gold.edges)
    kp_err = compute_error(predictions.keypoints, gold.keypoints)
    return {'PSNR': psnr, 'FPS': fps, 'Sem. mIoU': mIoU, 'Sem. Total Acc': acc, 'Edge error': edge_err, 'Keypoint error': kp_err}
```

## Common pitfalls

- Finetuning time is capped at 5 minutes in the main table, but quality improves with longer finetuning (up to 5 min tested).
- FPS calculation assumes one-time feature extraction; actual throughput may be lower if feature extraction is included per frame.
- Evaluation on ScanNet follows a zero-shot generalizable setting on non-overlapping test scenes, not standard in-domain evaluation.

## Evidence (verbatim from paper)

> Our rendering pipeline can be swiftly finetuned to significantly boost the PSNR. For instance, with just a 10-second finetuning, our rendering pipeline can match the average rendering quality of the strongest baseline, ReTR [34]; With 1-minute/2-minute finetuning, it can surpass ReTR by a +2.75 and +3.36 PSNR improvement on average, respectively;

## Citation

```bibtex
@misc{fu2024omnirecon,
  title={Omni-Recon: Harnessing Image-based Rendering for General-Purpose Neural Radiance Fields},
  author={Yonggan Fu et al.},
  year={2024},
  note={arXiv:2403.11131}
}
```

- arXiv: 2403.11131