Mapless Navigation Eval

---
name: mapless-navigation-eval
description: Evaluates a robot's ability to navigate to a goal in unknown environments using only local sensor data without a pre-built map. It probes the policy's generalization across varying obstacle densities, passage widths, and real-world conditions, as well as its energy efficiency on neuromorphic hardware. Use when the user wants to benchmark on Gazebo Training Environments, Gazebo Test Environment, Real-world Office Environment, or asks about evaluating this task. Reports success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2003.01157
  bibtex_key: tang2020sddpg
  confidence: high
---

# mapless-navigation-eval

> Reinforcement co-Learning of Deep and Spiking Neural Networks for Energy-Efficient Mapless Navigation with Neuromorphic Hardware — Tang et al. (2020) (arXiv:2003.01157, 2020)

## What this evaluates

Evaluates a robot's ability to navigate to a goal in unknown environments using only local sensor data without a pre-built map. It probes the policy's generalization across varying obstacle densities, passage widths, and real-world conditions, as well as its energy efficiency on neuromorphic hardware.

## Datasets

- **Gazebo Training Environments** — total ?; splits: train (-1)
- **Gazebo Test Environment** — total 200; splits: test (200)
- **Real-world Office Environment** — total 15; splits: test (15)

## Metrics

- `success rate` **(primary)** — range: percent
  - Percentage of test episodes that result in the robot successfully reaching the goal, as opposed to colliding with an obstacle or timing out after 1000 steps.
- `route distance` — range: meters
  - Average Euclidean distance traveled by the robot along successful trajectories from start to goal.
- `route speed` — range: m/s
  - Average speed maintained by the robot along successful trajectories.
- `energy cost per inference` — range: other
  - Average power consumption (W) divided by inference speed (Inf/s) to yield energy cost per forward pass in microjoules.

## Input / output format

**Input**: 18 laser range measurements (0.2-40 m) with 10-degree resolution, representing a front-facing 180-degree field of view.

**Output**: Continuous wheel speed commands constrained between 0.05 and 0.5 m/s.

## Scoring recipe

```python
outcomes = []
for start, goal in test_locations:
    traj = run_episode(agent, start, goal, max_steps=1000)
    outcomes.append(traj.outcome)
success_rate = sum(1 for o in outcomes if o == 'success') / len(outcomes)
route_distance = mean(len(traj) for traj in outcomes if traj.outcome == 'success')
route_speed = mean(traj.speed for traj in outcomes if traj.outcome == 'success')
energy_per_inf = avg_power_W / inference_speed_Inf_per_s * 1e6
```

## Common pitfalls

- Evaluating on the same environments used for training instead of the held-out 20m x 20m test environment with different obstacle shapes and narrower passages.
- Comparing energy efficiency without accounting for hardware-specific power modes (e.g., TX2 MAXQ vs MAXN) or using idle power instead of dynamic inference power.
- Ignoring timeout episodes (1000 steps) when calculating success rates, which underestimates failure rates in complex environments.

## Evidence (verbatim from paper)

> We first compared the methods based on the rate of the three possible outcomes–success, collision, and timeout... we computed the average distance and speed corresponding to the successful routes taken by each method... The energy cost per inference was obtained by dividing the power consumed over 1 second with the number of inferences performed in that second.

## Citation

```bibtex
@misc{tang2020sddpg,
  title={Reinforcement co-Learning of Deep and Spiking Neural Networks for Energy-Efficient Mapless Navigation with Neuromorphic Hardware},
  author={Tang et al. (2020)},
  year={2020},
  note={arXiv:2003.01157}
}
```

- arXiv: 2003.01157