Toolbench Eval

---
name: toolbench-eval
description: This benchmark evaluates an LLM's ability to understand API documentation, select the correct tool, and generate valid arguments to fulfill a user's goal. It probes tool manipulation capabilities across a diverse set of 8 real-world applications, ranging from simple single-call tasks to complex multi-step reasoning. Use when the user wants to benchmark on ToolBench, or asks about evaluating this task. Reports success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.16504
  bibtex_key: xu2023toolmanipulation
  confidence: high
---

# toolbench-eval

> On the Tool Manipulation Capability of Open-source Large Language Models — Xu et al. (2023) (arXiv:2305.16504, 2023)

## What this evaluates

This benchmark evaluates an LLM's ability to understand API documentation, select the correct tool, and generate valid arguments to fulfill a user's goal. It probes tool manipulation capabilities across a diverse set of 8 real-world applications, ranging from simple single-call tasks to complex multi-step reasoning.

## Datasets

- **ToolBench** — total ?; splits: test (-1); repo https://github.com/sambanova/toolbench

## Metrics

- `success rate` **(primary)** — range: percent
  - The percentage of user goals for which the model successfully generates a valid tool call that completes the task. Calculated as (number of successful goals / total goals) * 100.

## Input / output format

**Input**: User goal/query, API documentation, and optionally a few-shot in-context demonstration examples.

**Output**: A structured tool call specification containing the API name and populated arguments.

## Scoring recipe

```python
successes = 0
for goal in goals:
    call = parse_output(model.generate(goal, docs, demos))
    result = execute(call)
    if result.matches_ground_truth(goal):
        successes += 1
return (successes / len(goals)) * 100
```

## Common pitfalls

- The Tabletop task is evaluated exclusively in a few-shot setting, unlike other tasks which use zero-shot baselines.
- Results are averaged over 3 runs with different random seeds, but variance is often ignored in the main paper.
- Complex reasoning tasks like Google Sheets and WebShop show high variance and low success rates even after enhancement.

## Evidence (verbatim from paper)

> We run each job 3 times with different random seeds and report average accuracy. Table 6 exhibits significant disparities in tool manipulation between the closed GPT-4 API and open-source models in the out-of-the-box zero-shot setting. ... the success rates of the open-source LLMs can improve up to 90%.

## Citation

```bibtex
@misc{xu2023toolmanipulation,
  title={On the Tool Manipulation Capability of Open-source Large Language Models},
  author={Xu et al. (2023)},
  year={2023},
  note={arXiv:2305.16504}
}
```

- arXiv: 2305.16504