Robosense Eval

---
name: robosense-eval
description: Evaluates egocentric robot perception and navigation in crowded, unstructured environments. It probes multi-view 3D detection, 3D multi-object tracking, motion prediction, and 3D/BEV occupancy prediction using synchronized camera, LiDAR, and ultrasonic sensor data. Use when the user wants to benchmark on RoboSense, or asks about evaluating this task. Reports average precision.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.15503
  bibtex_key: su2024robosense
  confidence: high
---

# robosense-eval

> RoboSense: Large-scale Dataset and Benchmark for Egocentric Robot Perception and Navigation in Crowded and Unstructured Environments — Su et al. (2024) (arXiv:2408.15503, 2024)

## What this evaluates

Evaluates egocentric robot perception and navigation in crowded, unstructured environments. It probes multi-view 3D detection, 3D multi-object tracking, motion prediction, and 3D/BEV occupancy prediction using synchronized camera, LiDAR, and ultrasonic sensor data.

## Datasets

- **RoboSense** — total 133000; splits: train (-1), test (-1), val (-1); repo https://github.com/suhaisheng/RoboSense

## Metrics

- `average precision` **(primary)** — range: percent
  - Average Precision computed over recall thresholds. Predictions are matched to ground truth using either Center-Point (CP) distance or the proposed Closest-Collision-Point (CCP) distance, with a relative proportion threshold p (5% for LiDAR, 10% for images).
- `sAMOTA` — range: percent
  - simplified Average Multi-Object Tracking Accuracy, measuring tracking consistency and identity switches over time.
- `minADE` — range: meters
  - Minimum Average Displacement Error, the lowest L2 distance between predicted and ground truth future trajectories across multiple sampled hypotheses.
- `mIoU-3D` — range: percent
  - Mean Intersection over Union calculated in 3D voxel space for occupancy prediction, excluding ground voxels from the calculation.

## Input / output format

**Input**: Synchronized multi-sensor data at 10 FPS: RGB camera frames, fisheye camera frames, LiDAR point clouds, ultrasonic readings, and GPS/IMU localization. Inputs vary by task (e.g., image/point cloud sequences for detection/tracking, history trajectories or sensor data for prediction).

**Output**: Per instance: predicted 3D bounding boxes with class labels and orientations; track IDs for multi-object tracking; future trajectory waypoints for motion prediction; or 3D/BEV voxel occupancy grids.

## Scoring recipe

```python
def compute_3d_ap(pred_boxes, gt_boxes, criterion='CCP', p=0.05):
    matches = []
    for pred in pred_boxes:
        best_dist = float('inf')
        best_gt = None
        for gt in gt_boxes:
            if pred.class != gt.class: continue
            dist = ccp_distance(pred, gt) if criterion=='CCP' else center_distance(pred, gt)
            if dist < best_dist:
                best_dist, best_gt = dist, gt
        if best_dist < p * gt.length:
            matches.append((pred, best_gt))
    return average_precision_over_recall(matches)
```

## Common pitfalls

- Using standard Center Distance or IoU matching instead of the proposed CCP criterion significantly overestimates near-field detection performance.
- The test set is closed (no ground truth provided); models must be submitted to an online benchmark for evaluation.
- Occupancy mIoU scores are artificially lowered because ground voxels are explicitly excluded from the metric calculation.

## Evidence (verbatim from paper)

> For practical usages, we report performance using our proposed Closest-Collision Distance Proportion (CCDP) as matching criterion. Comparisons of different matching functions on average precision are shown in Fig.[4].

## Citation

```bibtex
@misc{su2024robosense,
  title={RoboSense: Large-scale Dataset and Benchmark for Egocentric Robot Perception and Navigation in Crowded and Unstructured Environments},
  author={Su et al. (2024)},
  year={2024},
  note={arXiv:2408.15503}
}
```

- arXiv: 2408.15503