M3cad Eval

---
name: m3cad-eval
description: Evaluates cooperative autonomous driving capabilities across perception, mapping, motion forecasting, occupancy prediction, and path planning. It probes whether multi-vehicle cooperation and realistic, non-straight trajectories improve ego-vehicle performance compared to single-vehicle baselines. Use when the user wants to benchmark on M3CAD, or asks about evaluating this task. Reports AMOTA.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.06746
  bibtex_key: zhu2025m3cad
  confidence: high
---

# m3cad-eval

> M3CAD: Towards Generic Cooperative Autonomous Driving Benchmark — Zhu et al. (2025) (arXiv:2505.06746, 2025)

## What this evaluates

Evaluates cooperative autonomous driving capabilities across perception, mapping, motion forecasting, occupancy prediction, and path planning. It probes whether multi-vehicle cooperation and realistic, non-straight trajectories improve ego-vehicle performance compared to single-vehicle baselines.

## Datasets

- **M3CAD** — total 204; splits: test (-1); repo https://github.com/zhumorui/M3CAD

## Metrics

- `AMOTA` **(primary)** — range: [0, 1]
  - Average Multi-Object Tracking Accuracy. Computed using the same protocol as UniAD, measuring tracking consistency and detection quality over sequences.
- `minADE` — range: meters
  - Minimum Average Displacement Error. The lowest average L2 distance between predicted and ground truth future trajectories across multiple sampled hypotheses.
- `L2` — range: meters
  - Average L2 distance between the predicted ego-vehicle trajectory and the ground truth trajectory over the planning horizon.
- `Col.` — range: percent
  - Average collision probability. Measures the likelihood of the planned trajectory intersecting with predicted agent occupancy or static obstacles.

## Input / output format

**Input**: Multi-view camera images, LiDAR point clouds, GPS/IMU data, and optionally shared Bird's Eye View (BEV) features from cooperating vehicles.

**Output**: Per-frame object detections (bounding boxes), tracking IDs, lane/road segmentation masks, future trajectories for surrounding agents, 3D occupancy grids, and ego-vehicle planned trajectory.

## Scoring recipe

```python
def compute_metrics(preds, gold):
    # Detection/Tracking (per UniAD protocol)
    mAP = compute_map(preds.dets, gold.dets)
    AMOTA = compute_amota(preds.tracks, gold.tracks)
    # Motion Forecasting
    minADE = min(np.mean(np.linalg.norm(preds.trajectories - gold.trajectories, axis=-1), axis=0))
    # Planning
    L2 = np.mean(np.linalg.norm(preds.ego_traj - gold.ego_traj, axis=1))
    Col = np.mean(preds.collision_mask)
    return {'mAP': mAP, 'AMOTA': AMOTA, 'minADE': minADE, 'L2': L2, 'Col': Col}
```

## Common pitfalls

- Training cooperative models with only local ground truth instead of global ground truth leads to incomplete supervision and significant performance drops.
- Assuming straight-line ego trajectories (common in NuScenes) generalize to complex cooperative scenarios; M3CAD's realistic non-straight paths heavily depend on perception data rather than just ego state.
- Failing to transform shared BEV features into the ego vehicle's coordinate frame before fusion, causing misaligned perception inputs.

## Evidence (verbatim from paper)

> Table 3 shows single-vehicle (S.) and cooperative (C.) tracking and mapping performance, using the same metrics defined in UniAD. ... Specifically, M3CAD yields a 2.4x improvement in minADE (minimum Average Displacement Error), likely due to its inclusion of multi-vehicle interactions, which enhance the model's understanding of object dynamics.

## Citation

```bibtex
@misc{zhu2025m3cad,
  title={M3CAD: Towards Generic Cooperative Autonomous Driving Benchmark},
  author={Zhu et al. (2025)},
  year={2025},
  note={arXiv:2505.06746}
}
```

- arXiv: 2505.06746