Travelplanner Eval

---
name: travelplanner-eval
description: Evaluates language agents' ability to perform long-horizon, multi-constraint real-world travel planning. It probes their capacity for dynamic tool use, constraint tracking, commonsense reasoning, and maintaining task coherence across complex decision-making steps. Use when the user wants to benchmark on TravelPlanner, or asks about evaluating this task. Reports macro pass rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.01622
  bibtex_key: [REDACTED]
  confidence: high
---

# travelplanner-eval

> TravelPlanner: A Benchmark for Real-World Planning with Language Agents — Xie et al. (2024) (arXiv:2402.01622, 2024)

## What this evaluates

Evaluates language agents' ability to perform long-horizon, multi-constraint real-world travel planning. It probes their capacity for dynamic tool use, constraint tracking, commonsense reasoning, and maintaining task coherence across complex decision-making steps.

## Datasets

- **TravelPlanner** — total 1225; splits: test (-1)

## Metrics

- `macro pass rate` **(primary)** — range: percent
  - The percentage of tasks where the generated plan satisfies all hard constraints. Calculated as (number of fully satisfied tasks / total tasks) * 100.
- `micro pass rate` — range: percent
  - The average constraint satisfaction rate across all individual constraints in a task, measuring partial success even when the full plan fails.

## Input / output format

**Input**: Natural language travel planning queries with multiple constraints. In two-stage mode, agents receive the query and access to six real-world tools for information collection. In sole-planning mode, agents are provided with the necessary pre-collected information.

**Output**: A structured travel plan (itinerary) specifying activities, bookings, and routes that must satisfy all specified constraints.

## Scoring recipe

```python
def compute_pass_rate(predictions, gold_constraints):
    satisfied = 0
    for pred, constraints in zip(predictions, gold_constraints):
        if all(check_constraint(pred, c) for c in constraints):
            satisfied += 1
    return (satisfied / len(predictions)) * 100
```

## Common pitfalls

- Agents often fail to track multiple constraints holistically, satisfying some but missing others, leading to low macro pass rates despite decent micro scores.
- The two-stage mode (tool use + planning) causes significant performance degradation compared to sole-planning due to limited cognitive capacity and multitasking overhead.
- Greedy search optimizing only for cost may outperform LLMs on hard constraints, highlighting that LLMs struggle with constraint satisfaction rather than cost minimization.

## Evidence (verbatim from paper)

> In the two-stage mode, GPT-4-Turbo with ReAct achieves only 0.6% in the final pass rate, and none of the other LLMs can pass any of the tasks. ... Agents struggle in obtaining a high macro pass rate. While some agents achieve high micro scores, their macro scores remain low.

## Citation

```bibtex
@misc{xie2024travelplanner,
  title={TravelPlanner: A Benchmark for Real-World Planning with Language Agents},
  author={Xie et al. (2024)},
  year={2024},
  note={arXiv:2402.01622}
}
```

- arXiv: 2402.01622