Mapeval Eval

---
name: mapeval-eval
description: This benchmark evaluates foundation models' geospatial reasoning capabilities across textual, visual, and API-based interaction modalities. It probes abilities such as place information retrieval, nearby point-of-interest identification, route planning, multi-step trip scheduling, and recognizing unanswerable queries. Use when the user wants to benchmark on MapEval, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.00316
  bibtex_key: dihan2024mapeval
  confidence: high
---

# mapeval-eval

> MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models — Dihan et al. (2024) (arXiv:2501.00316, 2024)

## What this evaluates

This benchmark evaluates foundation models' geospatial reasoning capabilities across textual, visual, and API-based interaction modalities. It probes abilities such as place information retrieval, nearby point-of-interest identification, route planning, multi-step trip scheduling, and recognizing unanswerable queries.

## Datasets

- **MapEval** — total 1000; splits: test (1000); repo https://github.com/MapEval/MapEval-API

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correct choices selected by the model. Calculated as (number of correct predictions / total number of questions) * 100.

## Input / output format

**Input**: Context, question, answer format guidelines, and multiple-choice options. For MapEval-API, tool usage documentation is also provided. MapEval-Visual additionally includes map images.

**Output**: Multiple-choice selection (A, B, C, etc.) for Textual and Visual tasks. For API tasks, ReACT-style tool usage sequences followed by a final answer choice.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
    return (correct / len(gold_labels)) * 100
```

## Common pitfalls

- Models frequently fail on 'Trip' planning due to difficulties in multi-step reasoning and aggregating spatio-temporal constraints.
- VLMs struggle significantly with map-specific tasks at higher zoom levels (beyond level 14) due to limited exposure to detailed map symbols and demarcations.
- API-based agents often make incorrect parameter usage errors or enter infinite request loops when no valid results are found.

## Evidence (verbatim from paper)

> We evaluate all tasks using the accuracy metric, defined as the percentage of correct choices selected by the model. We prompt models with the respective context, question, tool usage documentations (only for MapEval-API), answer format guidelines, and choices.

## Citation

```bibtex
@misc{dihan2024mapeval,
  title={MapEval: A Map-Based Evaluation of Geo-Spatial Reasoning in Foundation Models},
  author={Dihan et al. (2024)},
  year={2024},
  note={arXiv:2501.00316}
}
```

- arXiv: 2501.00316