Tau Bench Eval

---
name: tau-bench-eval
description: Evaluates the task completion success rate of LLM agents performing long-horizon, tool-using agentic workflows across customer service and software engineering domains. Probes the agent's ability to execute environment-altering (mutating) actions safely and maintain goal alignment over extended trajectories. Use when the user wants to benchmark on $	au$-Bench Airline, $	au$-Bench Retail, $	au$-Bench-V Air, $	au$-Bench-V Ret, SWE-Bench Verified, or asks about evaluating this task. Reports score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.07850
  bibtex_key: cuadron2025saber
  confidence: high
---

# tau-bench-eval

> SABER: Small Actions, Big Errors - Safeguarding Mutating Steps in LLM Agents — Cuadron et al. (2025) (arXiv:2512.07850, 2025)

## What this evaluates

Evaluates the task completion success rate of LLM agents performing long-horizon, tool-using agentic workflows across customer service and software engineering domains. Probes the agent's ability to execute environment-altering (mutating) actions safely and maintain goal alignment over extended trajectories.

## Datasets

- **$	au$-Bench Airline** — total ?; splits: test (-1)
- **$	au$-Bench Retail** — total ?; splits: test (-1)
- **$	au$-Bench-V Air** — total ?; splits: test (-1)
- **$	au$-Bench-V Ret** — total ?; splits: test (-1)
- **SWE-Bench Verified** — total ?; splits: test (-1)

## Metrics

- `score` **(primary)** — range: percent
  - Number of successfully completed trajectories divided by total evaluated trajectories, multiplied by 100.

## Input / output format

**Input**: Task description and environment state; agent interacts via tool-calling interface over multiple turns.

**Output**: Sequence of tool calls/actions and final response, capped at 30 turns per episode.

## Scoring recipe

```python
success_count = 0
for run in range(3):
    for task in dataset:
        trajectory = run_agent(task, max_turns=30)
        if trajectory.is_successful():
            success_count += 1
return (success_count / (len(dataset) * 3)) * 100
```

## Common pitfalls

- High variance in original $	au$-Bench requires averaging over multiple runs to get stable scores.
- Dataset noise in original benchmarks masks true model capacity, making verified versions necessary for fair comparison.
- Auxiliary model pairing significantly impacts results, so cross-model comparisons must control for main/auxiliary configurations.

## Evidence (verbatim from paper)

> To reduce the variance present in $	au$-Bench, we report the average score over three runs. All evaluations are limited to 30 turns.

## Citation

```bibtex
@misc{cuadron2025saber,
  title={SABER: Small Actions, Big Errors - Safeguarding Mutating Steps in LLM Agents},
  author={Cuadron et al. (2025)},
  year={2025},
  note={arXiv:2512.07850}
}
```

- arXiv: 2512.07850