Local Planner Benchmarking Eval

---
name: local-planner-benchmarking-eval
description: Compares local trajectory planners (DWB and TEB) for mobile manipulators by evaluating path smoothness, end-effector stability, trajectory deviation from a global path, and navigation accuracy/time in static and dynamic simulated environments. Use when the user wants to benchmark on Simulated Environments (Playground, Office, Warehouse), or asks about evaluating this task. Reports $\mathbf{p}_{e}$ (end-effector stability).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2211.01812
  bibtex_key: tafnakaji2022localplannerbenchmarking
  confidence: high
---

# local-planner-benchmarking-eval

> Benchmarking local motion planners for navigation of mobile manipulators — Tafnakaji et al. (2022) (arXiv:2211.01812, 2022)

## What this evaluates

Compares local trajectory planners (DWB and TEB) for mobile manipulators by evaluating path smoothness, end-effector stability, trajectory deviation from a global path, and navigation accuracy/time in static and dynamic simulated environments.

## Datasets

- **Simulated Environments (Playground, Office, Warehouse)** — total ?; splits: static (-1), dynamic (-1); repo https://github.com/sevag-tafnakaji/local_planner_benchmarking

## Metrics

- `$\mathbf{p}_{e}$ (end-effector stability)` **(primary)** — range: other
  - Quantifies the difference between the expected trajectory of the end-effector and its real trajectory during navigation.
- `$A_{\text{between}}$ (path deviation)` — range: other
  - Measures the area between the global path and the local trajectory, indicating how much the robot deviates from the planned route.
- `Smoothness` — range: radians
  - Calculated as the angle between the current direction and the new direction at each point along the travelled path.
- `$p_{\text{acc}}$ (accuracy)` — range: other
  - Final position accuracy metric comparing the robot's end pose to the target.
- `$T_{\text{taken}}$ (time)` — range: seconds
  - Total time taken to complete the navigation task.

## Input / output format

**Input**: Simulated environment map containing static and dynamic obstacles, a pre-computed global path, and robot kinematic constraints.

**Output**: A sequence of local trajectory waypoints (poses and velocities) generated by the planner to navigate from start to goal while avoiding obstacles.

## Scoring recipe

```python
def compute_metrics(traj_log, global_path, expected_ee_traj):
    angles = [angle_between(traj[i].dir, traj[i+1].dir) for i in range(len(traj)-1)]
    smoothness = mean(angles)
    deviation = integrate_area_between_paths(global_path, traj)
    ee_stability = mean(abs(expected_ee_traj - actual_ee_traj))
    accuracy = distance(traj[-1].pos, goal_pos)
    time_taken = traj[-1].timestamp - traj[0].timestamp
    return smoothness, deviation, ee_stability, accuracy, time_taken
```

## Common pitfalls

- Dynamic obstacles can completely block DWB's small sampling window, causing total path planning failure.
- Measurement timing mismatches between base and arm sensors can artificially inflate z-axis stability metrics.
- High smoothness penalties in narrow pathways may reflect necessary navigation maneuvers rather than planner inefficiency.

## Evidence (verbatim from paper)

> The metric that quantifies the load’s stability ($\mathbf{p}_{e}$) can be seen in Fig. 3. Data on the average performance of algorithms can be found in Table I. We can see that, on average, DWB leads to more stable paths at the end-effector, suggesting that it provides more stable load carrying capabilities.

## Citation

```bibtex
@misc{tafnakaji2022localplannerbenchmarking,
  title={Benchmarking local motion planners for navigation of mobile manipulators},
  author={Tafnakaji et al. (2022)},
  year={2022},
  note={arXiv:2211.01812}
}
```

- arXiv: 2211.01812