Oxford Spires Eval

---
name: oxford-spires-eval
description: Evaluates large-scale outdoor localization, 3D reconstruction, and novel-view synthesis using synchronized LiDAR, visual, and IMU data against millimetre-accurate TLS ground truth. Use when the user wants to benchmark on Oxford Spires Dataset, or asks about evaluating this task. Reports metric ground truth.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2411.10546
  bibtex_key: tao2024oxfordspires
  confidence: medium
---

# oxford-spires-eval

> The Oxford Spires Dataset: Benchmarking Large-Scale LiDAR-Visual Localisation, Reconstruction and Radiance Field Methods — Tao et al. (2024) (arXiv:2411.10546, 2024)

## What this evaluates

Evaluates large-scale outdoor localization, 3D reconstruction, and novel-view synthesis using synchronized LiDAR, visual, and IMU data against millimetre-accurate TLS ground truth.

## Datasets

- **Oxford Spires Dataset** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `metric ground truth` **(primary)** — range: meters
  - Computed by aligning predicted trajectories to the LiDAR/COLMAP trajectory using Umeyama's Sim(3) transformation, then measuring positional/orientational deviation against the TLS-derived ground truth (TUM format).
- `Reconstruction accuracy` — range: other
  - Compares predicted 3D point clouds or radiance fields against the registered TLS ground truth maps (1 cm resolution) to assess geometric fidelity.
- `Novel view rendering/depth estimation` — range: other
  - Evaluates rendered images or depth maps against ground truth depth images rendered from the TLS map using ground truth sensor trajectories.

## Input / output format

**Input**: Synchronized RGB fisheye images (3 cameras at 20Hz), 3D LiDAR point clouds (10Hz), IMU measurements, COLMAP SfM outputs (camera parameters, poses, 3D points), and TLS ground truth maps.

**Output**: Estimated SE(3) trajectories, reconstructed 3D point clouds or radiance fields, and rendered depth/novel-view images.

## Scoring recipe

```python
# Load predicted poses and ground truth poses (TUM format)
# Estimate Sim(3) scale/alignment using Umeyama's method
scale_align = umeyama_alignment(pred_poses, gt_poses)
# Compute localization error on aligned poses
localization_error = compute_ate_rpe(pred_poses * scale_align, gt_poses)
# For reconstruction/rendering, compare predicted outputs against TLS ground truth
reconstruction_error = compute_chamfer_or_psnr(pred_reconstruction, tls_ground_truth)
```

## Common pitfalls

- Vision-only methods (NeRF, 3DGS) often overfit to training poses and fail to generalize to out-of-sequence viewpoints.
- Metric scale ambiguity requires explicit Sim(3) alignment (Umeyama's method) before comparing to LiDAR/TLS ground truth.
- High-frequency image capture (20Hz) creates redundancy; methods must handle downsampled/aligned subsets (~1Hz) for SfM input.

## Evidence (verbatim from paper)

> Correcting the metric scale of vision-based 3D reconstructions produced by MVS and radiance field methods is necessary to enable comparison to the metric ground truth. To estimate the scale, we used Umeyama’s method to estimate a Sim(3) transformation between the LiDAR trajectory and a COLMAP trajectory, and the results are saved in evo_align_results.json.

## Citation

```bibtex
@misc{tao2024oxfordspires,
  title={The Oxford Spires Dataset: Benchmarking Large-Scale LiDAR-Visual Localisation, Reconstruction and Radiance Field Methods},
  author={Tao et al. (2024)},
  year={2024},
  note={arXiv:2411.10546}
}
```

- arXiv: 2411.10546