Ommo Eval

---
name: ommo-eval
description: Evaluates the novel view synthesis and implicit scene reconstruction capabilities of NeRF-based methods on large-scale outdoor environments. It probes how well models handle diverse camera trajectories, varying lighting conditions, and different scene scales (e.g., buildings vs. cities). Use when the user wants to benchmark on OMMO, or asks about evaluating this task. Reports PSNR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2301.06782
  bibtex_key: lu2023ommo
  confidence: high
---

# ommo-eval

> A Large-Scale Outdoor Multi-modal Dataset and Benchmark for Novel View Synthesis and Implicit Scene Reconstruction — Lu et al. (2023) (arXiv:2301.06782, 2023)

## What this evaluates

Evaluates the novel view synthesis and implicit scene reconstruction capabilities of NeRF-based methods on large-scale outdoor environments. It probes how well models handle diverse camera trajectories, varying lighting conditions, and different scene scales (e.g., buildings vs. cities).

## Datasets

- **OMMO** — total 14700; splits: train (-1), test (-1)

## Metrics

- `PSNR` **(primary)** — range: dB
  - Peak Signal-to-Noise Ratio. Measures pixel-level reconstruction error in decibels. Higher values indicate better fidelity.
- `SSIM` — range: [0, 1]
  - Structural Similarity Index. Measures perceived structural similarity between predicted and ground truth images. Ranges from 0 to 1, where 1 is identical.
- `LPIPS` — range: [0, 1]
  - Learned Perceptual Image Patch Similarity. Measures perceptual difference using deep network features. Lower values indicate higher perceptual similarity.

## Input / output format

**Input**: Calibrated RGB images and camera poses for training views; novel camera poses for testing views.

**Output**: Synthesized RGB images for the requested novel test views.

## Scoring recipe

```python
def compute_metrics(pred_imgs, gt_imgs):
    mse = np.mean((pred_imgs - gt_imgs) ** 2)
    psnr = 10 * np.log10(255**2 / mse)
    ssim = compute_ssim(pred_imgs, gt_imgs)
    lpips = compute_lpips(pred_imgs, gt_imgs)
    return psnr, ssim, lpips
```

## Common pitfalls

- NeRF-based methods frequently fail on scenes with irregular camera trajectories, performing significantly better on circular or line-based paths.
- Low-light/night scenes can artificially inflate PSNR/SSIM because networks output black areas to erase details, masking poor depth reconstruction.
- Large-scale scenes (e.g., cities) consistently yield lower metrics than small-scale subsets (e.g., buildings) due to limitations in implicit scene representation.

## Evidence (verbatim from paper)

> Evaluation Metrics. To evaluate the performance of each method, we use three common metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity (SSIM)[[38]], and LPIPS[[45]] on novel view synthesis. Higher PSNR and SSIM mean better performancence, while a lower LPIPS means better.

## Citation

```bibtex
@misc{lu2023ommo,
  title={A Large-Scale Outdoor Multi-modal Dataset and Benchmark for Novel View Synthesis and Implicit Scene Reconstruction},
  author={Lu et al. (2023)},
  year={2023},
  note={arXiv:2301.06782}
}
```

- arXiv: 2301.06782