Tau2 Bench Eval

---
name: tau2-bench-eval
description: Evaluates conversational agents' ability to collaborate with a user simulator in a dual-control environment where both parties share tool access to a dynamic world. It probes coordination, communication under decentralized control, and adherence to domain-specific policies while resolving multi-step tasks. Use when the user wants to benchmark on $\tau^2$-Bench, or asks about evaluating this task. Reports pass^1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.07982
  bibtex_key: barres2025tau2bench
  confidence: high
---

# tau2-bench-eval

> $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment — Barres et al. (2025) (arXiv:2506.07982, 2025)

## What this evaluates

Evaluates conversational agents' ability to collaborate with a user simulator in a dual-control environment where both parties share tool access to a dynamic world. It probes coordination, communication under decentralized control, and adherence to domain-specific policies while resolving multi-step tasks.

## Datasets

- **$\tau^2$-Bench** — total 200; splits: test (200); repo https://github.com/sierra-research/tau2-bench

## Metrics

- `pass^1` **(primary)** — range: [0, 1]
  - The fraction of tasks for which the agent achieves successful completion in a single deterministic run. Calculated as (number of tasks with successful trajectory) / (total number of tasks).

## Input / output format

**Input**: Domain policy document, task-specific instructions, OpenAI-format tool definitions, and conversation history. The agent receives generic guidelines plus domain policies; the user simulator receives generic guidelines plus task-specific instructions.

**Output**: Conversational text responses and structured function/tool calls in OpenAI format.

## Scoring recipe

```python
successes = 0
for task in dataset:
    trajectory = agent.run(task, tools, policy, user_sim)
    if environment.verify_success(trajectory):
        successes += 1
return successes / len(dataset)
```

## Common pitfalls

- LLM temperature must be strictly set to 0 to ensure deterministic outputs; non-zero temperatures invalidate pass^k comparisons.
- Ablation settings (No-User, Oracle Plan) alter tool access and information availability, so their scores cannot be directly compared to the default dual-control setting.
- User simulator errors (critical vs. benign) can confound agent performance; critical errors that prevent task completion must be distinguished from benign ones to avoid unfairly penalizing the agent.

## Evidence (verbatim from paper)

> We computed performance metrics on the verified $	au^2$-bench domains (retail and airline) and on our new telecom domain (see Figure 3). Our findings indicate that the telecom domain presents a greater challenge, exhibiting an overall lower success rate compared to other domains. gpt-4.1 pass^1 drops from 74% / 56% for retail and airline respectively to 34% for telecom.

## Citation

```bibtex
@misc{barres2025tau2bench,
  title={$\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment},
  author={Barres et al. (2025)},
  year={2025},
  note={arXiv:2506.07982}
}
```

- arXiv: 2506.07982