Tool Star Eval

---
name: tool-star-eval
description: Evaluates an LLM's ability to autonomously invoke and coordinate multiple tools (search, code, calculator, etc.) for complex reasoning tasks. It probes both computational reasoning (math) and knowledge-intensive reasoning (open-domain QA) under a multi-tool collaborative setting. Use when the user wants to benchmark on AIME2024, AIME2025, MATH500, MATH, GSM8K, GAIA, HLE, WebWalker, HotpotQA, 2WikiMultihopQA, Musique, Bamboogle, or asks about evaluating this task. Reports LLM-based judging accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.16410
  bibtex_key: dong2025toolstar
  confidence: high
---

# tool-star-eval

> Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning — Dong et al. (2025) (arXiv:2505.16410, 2025)

## What this evaluates

Evaluates an LLM's ability to autonomously invoke and coordinate multiple tools (search, code, calculator, etc.) for complex reasoning tasks. It probes both computational reasoning (math) and knowledge-intensive reasoning (open-domain QA) under a multi-tool collaborative setting.

## Datasets

- **AIME2024** — total ?; splits: test (-1)
- **AIME2025** — total ?; splits: test (-1)
- **MATH500** — total ?; splits: test (-1)
- **MATH** — total ?; splits: test (-1)
- **GSM8K** — total ?; splits: test (-1)
- **GAIA** — total ?; splits: test (-1)
- **HLE** — total ?; splits: test (-1)
- **WebWalker** — total ?; splits: test (-1)
- **HotpotQA** — total ?; splits: test (-1)
- **2WikiMultihopQA** — total ?; splits: test (-1)
- **Musique** — total ?; splits: test (-1)
- **Bamboogle** — total ?; splits: test (-1)

## Metrics

- `LLM-based judging accuracy` **(primary)** — range: percent
  - Correctness is determined by an LLM judge (Qwen2.5-72B-Instruct) rather than exact string matching. The judge evaluates whether the model's final answer matches the ground truth.
- `token-level F1 score` — range: percent
  - Standard token-level F1 score computed between the extracted answer and the ground truth for open-domain QA tasks.
- `Tool-use Efficiency ($T_E$)` — range: percent
  - $T_E = \frac{1}{N}\sum_{i=1}^{N}\frac{S_i}{T^c_i}$, where N is the number of datasets, $S_i$ is the number of correct answers, and $T^c_i$ is the total number of samples for the i-th dataset when using tools.

## Input / output format

**Input**: A reasoning question or prompt requiring multi-step problem solving.

**Output**: Step-by-step reasoning with tool invocations, followed by a final answer enclosed in \box{} tags.

## Scoring recipe

```python
def compute_metrics(predictions, golds, task_type):
    if task_type == 'computational':
        return llm_judge_accuracy(predictions, golds, judge='Qwen2.5-72B-Instruct')
    elif task_type == 'qa':
        return token_f1_score(predictions, golds)
    # Tool-use Efficiency across N datasets
    N = len(datasets)
    efficiency_sum = 0
    for i in range(N):
        S_i = count_correct(predictions[i], golds[i])
        T_c_i = len(golds[i])
        efficiency_sum += S_i / T_c_i
    return efficiency_sum / N
```

## Common pitfalls

- Answers must be enclosed in \box{} tags for extraction; missing tags cause evaluation failure.
- Computational tasks use a specific LLM judge (Qwen2.5-72B-Instruct), not exact match.
- Tool-use Efficiency aggregates accuracy across datasets, which can obscure per-task performance variations.

## Evidence (verbatim from paper)

> For all tasks, we follow previous work*[[31]]* and extract answers from the model output enclosed in \box{}. For computational reasoning tasks, we follow Search-o1*[[31]]* employ LLM-based judging (Qwen2.5-72B\=Instruct) to ensure answer correctness. For open-domain QA tasks, we adopt token-level F1 score as the evaluation metric. To assess tool usage efficiency, we propose the Tool-use Efficiency metric: $T_{E}\=\frac{1}{N}\sum_{i\=1}^{N}\frac{S_{i}}{T^{c}_{i}}$ where $N$ is the number of datasets, $S_{i}$ and $T^{c}_{i}$ denote the number of correct answers and the number of total samples for the $i$-th dataset when using tools.

## Citation

```bibtex
@misc{dong2025toolstar,
  title={Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning},
  author={Dong et al. (2025)},
  year={2025},
  note={arXiv:2505.16410}
}
```

- arXiv: 2505.16410