Meta Tool Eval

---
name: meta-tool-eval
description: Evaluates small language models' capability to adapt to and execute diverse tool-use tasks, including REST API calls, complex SQL queries, web navigation, and command-line interactions. It measures both functional correctness and efficiency under few-shot prompting and lightweight adaptation. Use when the user wants to benchmark on Gorilla APIBench (BFCL V4), Spider 2.0 (Enterprise Subset), WebArena, InterCode (Bash & CTF), or asks about evaluating this task. Reports Execution Success Rate (SR).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.20148
  bibtex_key: [REDACTED]
  confidence: high
---

# meta-tool-eval

> Meta-Tool: Efficient Few-Shot Tool Adaptation for Small Language Models — Kumar et al. (2026) (arXiv:2604.20148, 2026)

## What this evaluates

Evaluates small language models' capability to adapt to and execute diverse tool-use tasks, including REST API calls, complex SQL queries, web navigation, and command-line interactions. It measures both functional correctness and efficiency under few-shot prompting and lightweight adaptation.

## Datasets

- **Gorilla APIBench (BFCL V4)** — total ?; splits: test (-1)
- **Spider 2.0 (Enterprise Subset)** — total ?; splits: test (-1)
- **WebArena** — total ?; splits: test (-1)
- **InterCode (Bash & CTF)** — total ?; splits: test (-1)

## Metrics

- `Execution Success Rate (SR)` **(primary)** — range: percent
  - Percentage of tasks where the agent’s action leads to the desired state change or returns the correct data. Calculated as (correct tasks / total tasks) * 100.
- `Pass@1` — range: percent
  - Percentage of tasks solved on the first attempt. Calculated as (tasks solved on first try / total tasks) * 100.
- `Adaptation Time` — range: other
  - Wall-clock time required to prepare the agent for a new tool, from receiving documentation to readiness.
- `Inference Latency` — range: other
  - Time taken to generate the action.

## Input / output format

**Input**: Task prompts containing tool documentation, few-shot examples, and environment states (API specs, database schemas, web pages, or shell environments) depending on the benchmark.

**Output**: Model-generated actions (e.g., API calls, SQL queries, web navigation steps, or bash commands) executed in their respective sandboxed environments.

## Scoring recipe

```python
correct_count = 0
first_try_count = 0
for task in tasks:
    action = model.generate(prompt)
    result = execute(action, task.environment)
    if result.matches_gold(task):
        correct_count += 1
        first_try_count += 1
sr = (correct_count / len(tasks)) * 100
pass_at_1 = (first_try_count / len(tasks)) * 100
```

## Common pitfalls

- Confusing AST/syntax matching with actual execution success; the paper explicitly uses the 'Executable' subset where success depends on real API responses and state changes.
- Overlooking the distinction between adaptation time (preparation overhead) and inference latency (generation speed), which measure different efficiency bottlenecks.
- Assuming larger specialized models automatically outperform smaller adapted ones without accounting for context window limits and retrieval memory constraints.

## Evidence (verbatim from paper)

> Execution Success Rate (SR): The primary metric. The percentage of tasks where the agent’s action leads to the desired state change or returns the correct data. Pass@1: The percentage of tasks solved on the first attempt.

## Citation

```bibtex
@misc{kumar2026metatool,
  title={Meta-Tool: Efficient Few-Shot Tool Adaptation for Small Language Models},
  author={Kumar et al. (2026)},
  year={2026},
  note={arXiv:2604.20148}
}
```

- arXiv: 2604.20148