Mobilitybench Eval

---
name: mobilitybench-eval
description: Evaluates LLM-based route-planning agents on real-world mobility queries, probing their ability to handle multi-waypoint itineraries, preference-constrained routing, and multimodal travel. It measures how well agents understand instructions, decompose tasks, select tools, and produce valid, constraint-satisfying routes. Use when the user wants to benchmark on MobilityBench, or asks about evaluating this task. Reports Final Pass Rate (FPR).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.22638
  bibtex_key: [REDACTED]
  confidence: medium
---

# mobilitybench-eval

> MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios — Zhiheng Song et al. (arXiv:2602.22638, 2026)

## What this evaluates

Evaluates LLM-based route-planning agents on real-world mobility queries, probing their ability to handle multi-waypoint itineraries, preference-constrained routing, and multimodal travel. It measures how well agents understand instructions, decompose tasks, select tools, and produce valid, constraint-satisfying routes.

## Datasets

- **MobilityBench** — total ?; splits: test (-1); repo https://github.com/AMAP-ML/MobilityBench

## Metrics

- `Final Pass Rate (FPR)` **(primary)** — range: percent
  - Percentage of tasks where the agent's generated route fully satisfies all user constraints and passes validation.
- `Delivery Rate (DR)` — range: percent
  - Percentage of tasks where the agent successfully generates and delivers a route, regardless of full constraint satisfaction.

## Input / output format

**Input**: Natural language user queries representing real-world mobility intents (e.g., multi-waypoint itineraries, preference-constrained routing, multimodal travel) from Amap.

**Output**: Structured route plans or sequences of tool calls and observations generated by the agent, expected to comply with the mapping API schema and satisfy task constraints.

## Scoring recipe

```python
def compute_metrics(predictions, golds):
    dr = sum(1 for p in predictions if p.is_delivered) / len(predictions)
    fpr = sum(1 for p in predictions if p.is_delivered and p.passes_constraints(golds)) / len(predictions)
    return {'DR': dr * 100, 'FPR': fpr * 100}
```

## Common pitfalls

- Live mapping APIs are non-deterministic; evaluations must use a deterministic API-replay sandbox to ensure reproducibility.
- Preference-constrained routing tasks frequently cause hallucinations and trajectory deviations, especially under Plan-and-Execute frameworks.
- ReAct framework agents accumulate observation history, leading to significantly higher input token counts and inference latency compared to Plan-and-Execute.

## Evidence (verbatim from paper)

> Under the Plan-and-Execute framework, Claude-Opus-4.5 stands out as the strongest performer, achieved a Delivery Rate of 83.53% and a Final Pass Rate of 65.77%, both the highest among all evaluated models in this setting.

## Citation

```bibtex
@misc{song2026mobilitybench,
  title={MobilityBench: A Benchmark for Evaluating Route-Planning Agents in Real-World Mobility Scenarios},
  author={Zhiheng Song et al.},
  year={2026},
  note={arXiv:2602.22638}
}
```

- arXiv: 2602.22638