Aria Nerf Eval

---
name: aria-nerf-eval
description: Evaluates NeRF-based models on their ability to synthesize novel views from egocentric, multimodal sensor data captured in dynamic real-world environments. It probes how well current neural rendering methods handle temporal dynamics, lens distortion, and non-visual cues like IMU and gaze. Use when the user wants to benchmark on Aria-NeRF Dataset, or asks about evaluating this task. Reports PSNR.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2311.06455
  bibtex_key: sun2023arianerf
  confidence: medium
---

# aria-nerf-eval

> Aria-NeRF: Multimodal Egocentric View Synthesis — Sun et al. (2023) (arXiv:2311.06455, 2023)

## What this evaluates

Evaluates NeRF-based models on their ability to synthesize novel views from egocentric, multimodal sensor data captured in dynamic real-world environments. It probes how well current neural rendering methods handle temporal dynamics, lens distortion, and non-visual cues like IMU and gaze.

## Datasets

- **Aria-NeRF Dataset** — total ?; splits: test (-1)

## Metrics

- `PSNR` **(primary)** — range: other
  - Peak Signal-to-Noise Ratio computed in decibels between rendered and ground-truth images. Higher values indicate better reconstruction fidelity.
- `SSIM` — range: [0, 1]
  - Structural Similarity Index measuring perceived structural changes between rendered and ground-truth images. Values range from -1 to 1, with 1 indicating identical structure.
- `LPIPS` — range: [0, 1]
  - Learned Perceptual Image Patch Similarity using deep features to measure perceptual distance between images. Lower values indicate higher perceptual similarity.

## Input / output format

**Input**: Multimodal sensor streams from Aria Glasses including RGB frames, IMU, eye-tracking, audio, GPS, Wi-Fi, Bluetooth, and environmental data.

**Output**: Rendered novel-view images (either de-distorted or fisheye depending on the baseline model) generated by the NeRF representation.

## Scoring recipe

```python
def evaluate(rendered_img, gt_img):
    mse = np.mean((rendered_img - gt_img) ** 2)
    psnr = 10 * np.log10(255**2 / mse)
    ssim = compute_ssim(rendered_img, gt_img)
    lpips = compute_lpips(rendered_img, gt_img)
    return {'PSNR': psnr, 'SSIM': ssim, 'LPIPS': lpips}
```

## Common pitfalls

- Nerfacto outputs de-distorted images while NeuralDiff outputs fisheye images, requiring careful alignment or distortion-aware comparison for fair metric calculation.
- Static image metrics (PSNR/SSIM) may not capture temporal consistency or background/foreground disentanglement quality in dynamic egocentric scenes.

## Evidence (verbatim from paper)

> In terms of PSNR, SSIM, and LPIPS metrics, NeuralDiff generally surpasses Nerfacto across various scenarios.

## Citation

```bibtex
@misc{sun2023arianerf,
  title={Aria-NeRF: Multimodal Egocentric View Synthesis},
  author={Sun et al. (2023)},
  year={2023},
  note={arXiv:2311.06455}
}
```

- arXiv: 2311.06455