Maptrace Eval

---
name: maptrace-eval
description: Evaluates fine-grained spatial reasoning and pixel-accurate route tracing on commercial map images. Models must generate precise path coordinates or masks corresponding to text-based navigation queries. Use when the user wants to benchmark on MapTrace, Map-Bench, or asks about evaluating this task. Reports NDTW.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.19609
  bibtex_key: panagopoulou2025maptrace
  confidence: medium
---

# maptrace-eval

> MapTrace: Scalable Data Generation for Route Tracing on Maps — Panagopoulou et al. (2025) (arXiv:2512.19609, 2025)

## What this evaluates

Evaluates fine-grained spatial reasoning and pixel-accurate route tracing on commercial map images. Models must generate precise path coordinates or masks corresponding to text-based navigation queries.

## Datasets

- **MapTrace** — total 23000; splits: train (20000), val (3000)
- **Map-Bench** — total 1573; splits: test (1573)

## Metrics

- `NDTW` **(primary)** — range: [0, 1]
  - Normalized Dynamic Time Warping distance between the predicted path coordinates and the ground-truth path. Lower values indicate better alignment and accuracy.

## Input / output format

**Input**: A map image paired with a text query specifying start/end points or route instructions.

**Output**: A sequence of pixel coordinates or a binary mask representing the traced path.

## Scoring recipe

```python
def compute_ndtw(pred_path, gt_path):
    # pred_path and gt_path are lists of (x, y) coordinates
    dist = dtw_distance(pred_path, gt_path)
    norm_dist = dist / max(len(pred_path), len(gt_path))
    return norm_dist
```

## Common pitfalls

- Pixel-level precision is required; models often output topologically correct but spatially misaligned paths, which heavily penalizes NDTW scores.
- Evaluation queries on Map-Bench rely on human annotations that may vary in granularity, making cross-query comparison noisy.
- Baseline methods like Mask-Method extract paths from color masks, which can fail on maps with low contrast or overlapping features, skewing comparative results.

## Evidence (verbatim from paper)

> Fine-tuning MLLMs on this dataset improves path-tracing robustness (up to +6.4 points on MapBench) and reduces NDTW error, demonstrating that explicit supervision in coordinate-level spatial reasoning can bridge a critical gap in current models.

## Citation

```bibtex
@misc{panagopoulou2025maptrace,
  title={MapTrace: Scalable Data Generation for Route Tracing on Maps},
  author={Panagopoulou et al. (2025)},
  year={2025},
  note={arXiv:2512.19609}
}
```

- arXiv: 2512.19609