Nuscenes Hdmap Eval

---
name: nuscenes-hdmap-eval
description: This benchmark evaluates the accuracy of end-to-end vectorized high-definition map construction from multi-camera images. It probes a model's ability to precisely predict instance-level road elements (lane dividers, pedestrian crossings, road boundaries) as continuous curves rather than rasterized masks or polylines. Use when the user wants to benchmark on NuScenes, or asks about evaluating this task. Reports mAP.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.09700
  bibtex_key: qiao2023endtoend
  confidence: high
---

# nuscenes-hdmap-eval

> End-to-End Vectorized HD-map Construction with Piecewise Bezier Curve — Limeng Qiao et al. (2023) (arXiv:2306.09700, 2023)

## What this evaluates

This benchmark evaluates the accuracy of end-to-end vectorized high-definition map construction from multi-camera images. It probes a model's ability to precisely predict instance-level road elements (lane dividers, pedestrian crossings, road boundaries) as continuous curves rather than rasterized masks or polylines.

## Datasets

- **NuScenes** — total 28130; splits: train (700), val (150)

## Metrics

- `mAP` **(primary)** — range: percent
  - Instance-level Average Precision (AP) computed using Chamfer Distance between ground-truth and predicted curve instances. A prediction is a true positive if the Chamfer Distance is below a specified threshold. The overall mAP is the average of AP scores across three thresholds ([0.2, 0.5, 1.0] m) and three map categories (lane-divider, ped-crossing, road-boundary).

## Input / output format

**Input**: Six surrounding camera images (360° FOV) per driving scene, resized to 896×512 pixels, covering a perception range of [30, 30, 15, 15] meters relative to the ego-vehicle.

**Output**: A sparse set of vectorized map elements, each represented as a piecewise Bézier curve defined by control point coordinates and a segmentation class (lane-divider, ped-crossing, or road-boundary).

## Scoring recipe

```python
def compute_mAP(predictions, ground_truth, thresholds=[0.2, 0.5, 1.0]):
    categories = ['lane-divider', 'ped-crossing', 'road-boundary']
    ap_scores = []
    for cat in categories:
        gt = [g for g in ground_truth if g.category == cat]
        preds = [p for p in predictions if p.category == cat]
        cat_ap = 0.0
        for thresh in thresholds:
            tp = sum(1 for p in preds if min(chamfer_distance(p, g) for g in gt) < thresh)
            cat_ap += tp / max(len(gt), 1)
        ap_scores.append(cat_ap / len(thresholds))
    return sum(ap_scores) / len(ap_scores) * 100
```

## Common pitfalls

- The evaluation uses Chamfer Distance rather than point-wise IoU or Hausdorff distance, making results sensitive to curve parameterization and control point density.
- The perception range is strictly fixed to [30, 30, 15, 15]m with a resolution of 0.15 m/pixel; deviating from these bounds or resolutions breaks comparability.
- A secondary 'simpler' protocol uses thresholds [0.5, 1.0, 1.5]m, which significantly inflates AP scores compared to the standard [0.2, 0.5, 1.0]m protocol and should not be mixed in reporting.

## Evidence (verbatim from paper)

> We utilize the exact same evaluation protocol as [[24]] of average precision (AP) to access the map construction quality over the instance-level. To be concrete, given a pair of instances from ground-truth and predictions respectively, this protocol computes the Chamfer Distance between them and considers the prediction as true-positive only if the distance is less than a specified threshold, which is set to [0.2,0.5,1.0]m in our experiment. Note the overall AP metric is obtained by averaging across three thresholds.

## Citation

```bibtex
@misc{qiao2023endtoend,
  title={End-to-End Vectorized HD-map Construction with Piecewise Bezier Curve},
  author={Limeng Qiao et al. (2023)},
  year={2023},
  note={arXiv:2306.09700}
}
```

- arXiv: 2306.09700