Rover Eval

---
name: rover-eval
description: Evaluates the accuracy and robustness of visual-inertial SLAM systems across diverse outdoor environments, seasons, and lighting conditions. It probes long-term trajectory consistency, scale estimation, and environmental adaptability under challenging visual degradation. Use when the user wants to benchmark on ROVER, or asks about evaluating this task. Reports mATE.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.02506
  bibtex_key: schmidt2024rover
  confidence: high
---

# rover-eval

> ROVER: A Multi-Season Dataset for Visual SLAM — Schmidt et al. (2024) (arXiv:2412.02506, 2024)

## What this evaluates

Evaluates the accuracy and robustness of visual-inertial SLAM systems across diverse outdoor environments, seasons, and lighting conditions. It probes long-term trajectory consistency, scale estimation, and environmental adaptability under challenging visual degradation.

## Datasets

- **ROVER** — total 39; splits: test (39)

## Metrics

- `mATE` **(primary)** — range: meters
  - Mean Absolute Trajectory Error. Computed as the average SE(3) alignment error between estimated and ground-truth poses across all frames in a sequence, then averaged over all locations and seasons.
- `mRPE` — range: meters
  - Mean Relative Pose Error. Computed as the average translation/rotation error between relative poses over a fixed interval, averaged over all locations and seasons.
- `SR` — range: percent
  - Success Rate. The percentage of sequences where the SLAM system maintains tracking without failure or excessive drift.

## Input / output format

**Input**: Synchronized RGB images (mono, stereo, or RGBD) and IMU data from 39 sequences across 5 outdoor locations, 4 seasons, and 4 lighting conditions.

**Output**: Time-stamped camera pose trajectory (6-DoF) over the sequence duration.

## Scoring recipe

```python
def compute_rover_metrics(trajectories, ground_truths):
    ate_scores, rpe_scores, success_count = [], [], 0
    for est, gt in zip(trajectories, ground_truths):
        if not is_successful(est): success_count += 1
        ate_scores.append(compute_ate(est, gt))
        rpe_scores.append(compute_rpe(est, gt))
    return {
        'mATE': sum(ate_scores) / len(ate_scores),
        'mRPE': sum(rpe_scores) / len(rpe_scores),
        'SR': (success_count / len(trajectories)) * 100
    }
```

## Common pitfalls

- Monocular SLAM requires scale alignment (e.g., Sim(3)) before ATE computation, otherwise errors are meaningless.
- SR is highly sensitive to tracking failures in low-light or dense vegetation, which can artificially deflate mATE if failed sequences are excluded.
- Internal vs. external IMU mounting significantly alters performance, so configurations must be reported explicitly.

## Evidence (verbatim from paper)

> For each SLAM configuration, we compute the ATE and RPE for every location (averaged across the four seasons) and then determine the overall mean ATE (mATE), mean RPE (mRPE), and SR across all locations.

## Citation

```bibtex
@misc{schmidt2024rover,
  title={ROVER: A Multi-Season Dataset for Visual SLAM},
  author={Schmidt et al. (2024)},
  year={2024},
  note={arXiv:2412.02506}
}
```

- arXiv: 2412.02506