Simpletoolhallubench Eval

---
name: simpletoolhallubench-eval
description: Probes an agent's ability to abstain from tool use when no appropriate tools are available, specifically measuring its tendency to hallucinate tool invocations in two controlled scenarios: when no tools are provided and when irrelevant distractor tools are present. Use when the user wants to benchmark on SimpleToolHalluBench, or asks about evaluating this task. Reports R_NTA.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.22977
  bibtex_key: yin2025reasoningtrap
  confidence: high
---

# simpletoolhallubench-eval

> The Reasoning Trap: How Enhancing LLM Reasoning Amplifies Tool Hallucination — Chenlong Yin et al. (2025) (arXiv:2510.22977, 2025)

## What this evaluates

Probes an agent's ability to abstain from tool use when no appropriate tools are available, specifically measuring its tendency to hallucinate tool invocations in two controlled scenarios: when no tools are provided and when irrelevant distractor tools are present.

## Datasets

- **SimpleToolHalluBench** — total 296; splits: test (296)

## Metrics

- `R_NTA` **(primary)** — range: [0, 1]
  - Hallucination rate for the No-Tool-Available task, calculated as the fraction of responses flagged by an LLM-as-judge as hallucinating tool invocations: R_NTA = H_NTA / N_NTA.
- `R_DT` — range: [0, 1]
  - Hallucination rate for the Distractor-Tool task, calculated as the fraction of responses flagged by an LLM-as-judge as hallucinating tool invocations: R_DT = H_DT / N_DT.

## Input / output format

**Input**: User query paired with a system prompt that either provides no tools (NTA) or provides an irrelevant distractor tool (DT).

**Output**: Model's generated response, which may include a tool invocation, a refusal/abstention, or a hallucinated tool call.

## Scoring recipe

```python
# For each task (NTA or DT):
N = total number of samples in the task
H = 0
for response in model_responses:
    if llm_as_judge_flags_hallucination(response):
        H += 1
rate = H / N
return rate
```

## Common pitfalls

- Queries are explicitly designed to be impossible without the specific required tool, so models should not rely on internal knowledge to answer them.
- Scoring relies on an LLM-as-judge to flag hallucinations, which may introduce variability or bias compared to deterministic tool-existence checks.
- The benchmark separates 'NTA' (no tools provided) and 'DT' (irrelevant tool provided) scenarios, so results should not be averaged without noting the distinct failure modes.

## Evidence (verbatim from paper)

> We calculate the hallucination rate for each task independently. These rates are defined as the fraction of responses flagged by the LLM-as-judge: R_NTA = H_NTA / N_NTA and R_DT = H_DT / N_DT, where H_NTA and H_DT are the counts of hallucinated responses in their respective tasks, and N_NTA and N_DT are the total number of samples in each task set.

## Citation

```bibtex
@misc{yin2025reasoningtrap,
  title={The Reasoning Trap: How Enhancing LLM Reasoning Amplifies Tool Hallucination},
  author={Chenlong Yin et al. (2025)},
  year={2025},
  note={arXiv:2510.22977}
}
```

- arXiv: 2510.22977