Mapdr Eval

---
name: mapdr-eval
description: Probes autonomous driving models' ability to extract lane-level traffic regulations from visual inputs and map them to vectorized HD map centerlines. It evaluates both rule extraction from image sequences and bipartite graph construction for rule-lane correspondence reasoning. Use when the user wants to benchmark on MapDR, or asks about evaluating this task. Reports correspondence status.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2410.23780
  bibtex_key: chang2024mapdr
  confidence: medium
---

# mapdr-eval

> Driving by the Rules: A Benchmark for Integrating Traffic Sign Regulations into Vectorized HD Map — Chang et al. (2024) (arXiv:2410.23780, 2024)

## What this evaluates

Probes autonomous driving models' ability to extract lane-level traffic regulations from visual inputs and map them to vectorized HD map centerlines. It evaluates both rule extraction from image sequences and bipartite graph construction for rule-lane correspondence reasoning.

## Datasets

- **MapDR** — total 18000; splits: unspecified (-1); repo https://github.com/MIV-XJTU/MapDR

## Metrics

- `correspondence status` **(primary)** — range: [0, 1]
  - Accuracy of the predicted binary adjacency matrix E_pred against the ground truth E_gold, where each element indicates the correspondence status between a rule and a lane centerline.

## Input / output format

**Input**: Image sequences X={x_i}_{i=1}^n and local vectorized HD map centerlines L={l_i}_{i=1}^k.

**Output**: For rule extraction: a set of rules R={r_i}_{i=1}^m in key:value pairs. For correspondence: a bipartite graph G=(R∪L,E) with a binary adjacency matrix E⊆{0,1}^{m×k}.

## Scoring recipe

```python
def score(predictions, gold):
    # predictions: list of rules, adjacency matrix E_pred (m x k)
    # gold: list of rules, adjacency matrix E_gold (m x k)
    # Align rule ordering if necessary, then compare adjacency matrices
    correct = (E_pred == E_gold).sum()
    total = E_pred.size
    return correct / total
```

## Common pitfalls

- Confusing the two sub-tasks: rule extraction (visual-to-text) and correspondence reasoning (text-to-graph).
- Assuming a fixed number of rules (m) or lanes (k) per instance, as both vary dynamically.
- Treating the bipartite adjacency matrix as directed or weighted, whereas the protocol specifies a strictly binary correspondence status.

## Evidence (verbatim from paper)

> The final correspondence forms a bipartite graph G=(R∪L,E), where E⊆{0,1}^{m×k} and the element E_ij in the i-th row and j-th column of matrix E represents the corresponding status between r_i and l_j.

## Citation

```bibtex
@misc{chang2024mapdr,
  title={Driving by the Rules: A Benchmark for Integrating Traffic Sign Regulations into Vectorized HD Map},
  author={Chang et al. (2024)},
  year={2024},
  note={arXiv:2410.23780}
}
```

- arXiv: 2410.23780