Anytool Eval

---
name: anytool-eval
description: Evaluates an agent's ability to retrieve and invoke relevant APIs from a large-scale pool to resolve user queries. It probes hierarchical API retrieval, self-reflective error recovery, and the capacity to handle context limits when dealing with thousands of available tools. Use when the user wants to benchmark on ToolBench (filtered), AnyToolBench, or asks about evaluating this task. Reports pass rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.04253
  bibtex_key: du2024anytool
  confidence: high
---

# anytool-eval

> AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls — Du et al. (2024) (arXiv:2402.04253, 2024)

## What this evaluates

Evaluates an agent's ability to retrieve and invoke relevant APIs from a large-scale pool to resolve user queries. It probes hierarchical API retrieval, self-reflective error recovery, and the capacity to handle context limits when dealing with thousands of available tools.

## Datasets

- **ToolBench (filtered)** — total 632; splits: G1-I (115), G1-T (132), G1-C (142), G2-I (107), G2-C (98), G3-I (38)
- **AnyToolBench** — total 400; splits: test (400)

## Metrics

- `pass rate` **(primary)** — range: percent
  - The percentage of user queries for which the agent's generated solution successfully resolves the task. Calculated as (number of resolved queries / total queries) × 100%. Success is determined by a GPT-4-32K judge using the ToolBench evaluation prompt.

## Input / output format

**Input**: User query describing a task, along with access to an API pool (either reference APIs, a candidate pool, or the full 16,000+ API collection).

**Output**: A sequence of API calls and parameters generated by the agent to solve the query.

## Scoring recipe

```python
def compute_pass_rate(predictions, gold_queries):
    resolved_count = 0
    for query, solution in zip(gold_queries, predictions):
        # GPT-4-32K judge evaluates if the solution resolves the query
        if gpt4_judge(solution, query) == 'success':
            resolved_count += 1
    return (resolved_count / len(gold_queries)) * 100
```

## Common pitfalls

- Prior benchmarks artificially inflated pass rates by providing reference APIs or using simpler judges; this protocol requires the agent to retrieve APIs from a pool of 16,000+ without reference lists.
- GPT-4's context window limits prevent feeding all APIs at once; the evaluation relies on a hierarchical retriever to manage context, which must be accounted for when reproducing results.
- The GPT-4 judge alignment with humans is high (96.5%), but using GPT-3.5 drops alignment to 73.9%, so the exact judge model matters.

## Evidence (verbatim from paper)

> We employ the pass rate (as defined in Eq. 2) as our evaluation metric. To assess whether a solution generated by an agent can resolve the query, we use GPT-4-32K. The same prompt utilized in ToolBench is applied when GPT-4 serves as the judge.

## Citation

```bibtex
@misc{du2024anytool,
  title={AnyTool: Self-Reflective, Hierarchical Agents for Large-Scale API Calls},
  author={Du et al. (2024)},
  year={2024},
  note={arXiv:2402.04253}
}
```

- arXiv: 2402.04253