Toolmind Eval

---
name: toolmind-eval
description: Evaluates large language models' tool-use and function-calling capabilities, specifically probing multi-turn dialogues and agentic workflows such as search and memory retrieval. Use when the user wants to benchmark on BFCL-v4, τ-Bench, τ²-Bench, or asks about evaluating this task. Reports BFCL-v4 Overall.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2511.15718
  bibtex_key: [REDACTED]
  confidence: high
---

# toolmind-eval

> ToolMind Technical Report: A Large-Scale, Reasoning-Enhanced Tool-Use Dataset — Chen Yang et al. (arXiv:2511.15718, 2025)

## What this evaluates

Evaluates large language models' tool-use and function-calling capabilities, specifically probing multi-turn dialogues and agentic workflows such as search and memory retrieval.

## Datasets

- **BFCL-v4** — total ?; splits: test (-1)
- **τ-Bench** — total ?; splits: test (-1)
- **τ²-Bench** — total ?; splits: test (-1)

## Metrics

- `BFCL-v4 Overall` **(primary)** — range: percent
  - Percentage of correctly executed tool calls or environment interactions, averaged across single-turn, multi-turn, and agentic sub-tasks.
- `τ-Bench Avg.` — range: percent
  - Average accuracy across retail and airline domains, measured via environment interaction with a GPT-4o user simulator.
- `τ²-Bench Avg.` — range: percent
  - Average accuracy across retail and airline domains in a multi-turn agentic setting, evaluated with a GPT-4o user simulator.

## Input / output format

**Input**: Multi-turn conversation history containing user queries, prior tool calls, and function schemas.

**Output**: Model-generated reasoning traces, content responses, and structured tool calls.

## Scoring recipe

```python
def compute_metric(predictions, gold):
    correct = 0
    for pred, gold_inst in zip(predictions, gold):
        if execute_tool_call(pred) == gold_inst:
            correct += 1
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Single-turn scores may fluctuate modestly while multi-turn and agentic scores show significant gains.
- BFCL-v4 distinguishes between non-live (static AST) and live (dynamic execution) evaluation, which can yield different results.
- τ-bench and τ²-bench rely on a GPT-4o user simulator, introducing potential variability in interaction dynamics.

## Evidence (verbatim from paper)

> Overall Performance. The performance of our SFT models on τ-Bench [20], and τ²-bench [1], BFCL-v4 [11] is summarized in Table 2 and Table 3. On BFCL-v4, we observe notable improvements in both Qwen3-8B and Qwen3-14B after training. Although their single-turn evaluation scores show modest fluctuations, they remain above most baseline models. Performance gains are most pronounced in multi-turn and agentic evaluations.

## Citation

```bibtex
@misc{yang2025toolmind,
  title={ToolMind Technical Report: A Large-Scale, Reasoning-Enhanced Tool-Use Dataset},
  author={Chen Yang et al.},
  year={2025},
  note={arXiv:2511.15718}
}
```

- arXiv: 2511.15718