Toolsandbox Eval

---
name: toolsandbox-eval
description: Evaluates LLM tool-use capabilities in a stateful, conversational, and interactive setting. It probes the model's ability to handle implicit state dependencies, canonicalize arguments, handle insufficient information, and maintain efficiency across single/multiple tool calls and user turns. Use when the user wants to benchmark on ToolSandbox, or asks about evaluating this task. Reports average similarity score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.04682
  bibtex_key: lu2024toolsandbox
  confidence: medium
---

# toolsandbox-eval

> ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities — Lu et al. (2024) (arXiv:2408.04682, 2024)

## What this evaluates

Evaluates LLM tool-use capabilities in a stateful, conversational, and interactive setting. It probes the model's ability to handle implicit state dependencies, canonicalize arguments, handle insufficient information, and maintain efficiency across single/multiple tool calls and user turns.

## Datasets

- **ToolSandbox** — total ?; splits: test (-1); repo https://github.com/apple/ToolSandbox

## Metrics

- `average similarity score` **(primary)** — range: percent
  - The average similarity score between the model's generated tool calls/responses and the gold/reference trajectory, computed per instance and averaged across all instances or specific scenario categories. Scores are reported on a 0-100 scale.

## Input / output format

**Input**: Conversational user queries presented in an interactive environment with access to a predefined set of tools. The model receives a minimalist system prompt and must generate tool calls or text responses based on the current conversation state and tool availability.

**Output**: Sequential tool calls and conversational responses generated by the LLM within an interactive loop, subject to an execution environment that tracks state and returns tool outputs.

## Scoring recipe

```python
scores = []
for instance in test_set:
    trajectory = run_model_interactively(instance, prompt=minimalist_prompt)
    sim = compute_similarity(trajectory, gold_trajectory)
    scores.append(sim)
avg_score = sum(scores) / len(scores)
# Report avg_score overall and grouped by scenario categories (STC, MTC, SD, etc.) and augmentations (DT, TNS, etc.)
```

## Common pitfalls

- Models may hallucinate tool names or arguments when provided tools are insufficient, artificially lowering the 'Insufficient Information' score.
- Larger models often issue parallel tool calls for state-dependent tasks, causing race conditions in the execution environment and degrading performance.
- Using complex prompt engineering can inflate scores; the benchmark intentionally uses a minimalist prompt to measure innate tool-use capability.

## Evidence (verbatim from paper)

> When evaluating the models, all models use the same minimalist prompt shown in Figure [8] for comparison fairness. We do not include additional prompt engineering for the models, as we consider prompt engineering gains orthogonal to the innate model capability surfaced by simpler prompting. Table [5] shows the average similarity for each of the scenario categories described in Section [3].

## Citation

```bibtex
@misc{lu2024toolsandbox,
  title={ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities},
  author={Lu et al. (2024)},
  year={2024},
  note={arXiv:2408.04682}
}
```

- arXiv: 2408.04682