Collaborative 3d Detection Eval

---
name: collaborative-3d-detection-eval
description: Evaluates collaborative 3D object detection performance and communication efficiency under various conditions including homogeneous/heterogeneous sensor setups, bandwidth constraints, communication latency, and pose errors. Use when the user wants to benchmark on DAIR-V2X, V2V4Real, TUMTraf-V2X, OPV2V, V2X-SIM2.0, or asks about evaluating this task. Reports Average Precision (AP) at IoU 0.30/0.50, Mean Average Precision (mAP) in BEV.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2404.09496
  bibtex_key: liu2024collaborative
  confidence: high
---

# collaborative-3d-detection-eval

> Towards Collaborative Autonomous Driving: Simulation Platform and End-to-End System — Liu et al. (2024) (arXiv:2404.09496, 2024)

## What this evaluates

Evaluates collaborative 3D object detection performance and communication efficiency under various conditions including homogeneous/heterogeneous sensor setups, bandwidth constraints, communication latency, and pose errors.

## Datasets

- **DAIR-V2X** — total ?; splits: test (-1)
- **V2V4Real** — total 20000; splits: test (-1)
- **TUMTraf-V2X** — total ?; splits: test (-1)
- **OPV2V** — total 12000; splits: test (-1)
- **V2X-SIM2.0** — total 47200; splits: test (-1)

## Metrics

- `Average Precision (AP) at IoU 0.30/0.50` **(primary)** — range: [0, 1]
  - Standard object detection metric computing precision-recall curve area at specified IoU thresholds (0.30 and 0.50) between predicted and ground-truth 3D bounding boxes.
- `Mean Average Precision (mAP) in BEV` **(primary)** — range: [0, 1]
  - Average of AP scores across all object classes, computed in the Bird's Eye View (BEV) perspective using center distance for matching.
- `Communication cost` — range: bits
  - Calculated as log2(H × W × ||M||1 × C × 32 / 8) bits, where H and W are feature map dimensions, M is the selection matrix for transmitted features, and C is the number of channels.

## Input / output format

**Input**: Multi-agent sensor data (LiDAR point clouds and/or RGB images) with corresponding poses, targeting a predefined spatial detection area.

**Output**: 3D bounding boxes with class labels and confidence scores for detected objects within the detection area.

## Scoring recipe

```python
def compute_ap_mAP(predictions, ground_truth, iou_thresh=0.5):
    # 1. Match predictions to ground truth boxes using IoU > iou_thresh
    # 2. Sort matches by confidence score descending
    # 3. Compute precision and recall at each threshold
    # 4. Interpolate precision-recall curve to get AP
    # 5. Average AP across all classes for mAP
    return ap, mAP
```

## Common pitfalls

- Focusing solely on detection accuracy while ignoring the communication bandwidth trade-off.
- Evaluating only homogeneous sensor setups, neglecting heterogeneous configurations (e.g., LiDAR-only vs. camera-only agents).
- Assuming perfect synchronization and pose alignment, failing to test robustness against realistic communication latency and pose errors.

## Evidence (verbatim from paper)

> Detection performance. Following the collaborative perception methods [10], [11], [20], [44], the detection results are evaluated by 1) Average Precision (AP) at Intersection-over-Union (IoU) thresholds of 0.30, 0.50. 2) Mean average precision (mAP) in BEV perspective, considering the BEV center distance.

## Citation

```bibtex
@misc{liu2024collaborative,
  title={Towards Collaborative Autonomous Driving: Simulation Platform and End-to-End System},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2404.09496}
}
```

- arXiv: 2404.09496