Function Calling Eval

---
name: function-calling-eval
description: Evaluates an LLM's ability to correctly identify, retrieve, and invoke external APIs or functions based on a user query. It probes zero-shot and multi-turn function-calling capabilities, including handling live vs. non-live APIs, detecting irrelevant queries, and mitigating hallucinations. Use when the user wants to benchmark on BFCL-v3, API-Bank, or asks about evaluating this task. Reports AST.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2409.00920
  bibtex_key: liu2024toolace
  confidence: high
---

# function-calling-eval

> ToolACE: Winning the Points of LLM Function Calling — Liu et al. (2024) (arXiv:2409.00920, 2024)

## What this evaluates

Evaluates an LLM's ability to correctly identify, retrieve, and invoke external APIs or functions based on a user query. It probes zero-shot and multi-turn function-calling capabilities, including handling live vs. non-live APIs, detecting irrelevant queries, and mitigating hallucinations.

## Datasets

- **BFCL-v3** — total ?; splits: test (-1)
- **API-Bank** — total ?; splits: test (-1)

## Metrics

- `AST` **(primary)** — range: percent
  - Percentage of test cases where the model's predicted function call matches the ground truth in abstract syntax tree structure.
- `Exec` — range: percent
  - Percentage of test cases where the predicted function call is syntactically valid and executable.
- `Overall` — range: percent
  - Composite score combining AST and Exec accuracy across single-turn and multi-turn scenarios.
- `Relevance` — range: percent
  - Accuracy in correctly identifying whether a query requires a function call.
- `Irrelevance` — range: percent
  - Accuracy in correctly identifying whether a query does not require a function call.

## Input / output format

**Input**: Natural language user query or instruction, optionally with conversation history for multi-turn settings.

**Output**: Structured function call specification (e.g., JSON or code) containing the function name and arguments.

## Scoring recipe

```python
def compute_metrics(predictions, gold):
    ast_correct = sum(1 for p, g in zip(predictions, gold) if p['ast'] == g['ast'])
    exec_correct = sum(1 for p, g in zip(predictions, gold) if p['exec'] == g['exec'])
    rel_correct = sum(1 for p, g in zip(predictions, gold) if p['rel'] == g['rel'])
    irrel_correct = sum(1 for p, g in zip(predictions, gold) if p['irrel'] == g['irrel'])
    n = len(predictions)
    return {
        'AST': ast_correct / n * 100,
        'Exec': exec_correct / n * 100,
        'Overall': (ast_correct + exec_correct) / (2 * n) * 100,
        'Relevance': rel_correct / n * 100,
        'Irrelevance': irrel_correct / n * 100
    }
```

## Common pitfalls

- Models often hallucinate non-existent APIs or invent arguments when faced with out-of-distribution queries.
- Multi-turn evaluation requires strict context management; forgetting previous turns or mixing up API states leads to severe accuracy drops.
- Live API benchmarks require handling real-time network calls, which can introduce latency and non-determinism not present in non-live evaluations.

## Evidence (verbatim from paper)

> The findings in BFCL indicate that API-based models demonstrate significant advantages over open-source models, such as the Claude series and the GPT-4 series. Open-source models fine-tuned for function calling, such as Functionary and xLAM, exhibit competitive performance, but still fall short of the leading models. Our ToolACE-8B model outperforms most API-based and open-source models in both the AST and Exec categories of BFCL, and continues to exhibit substantial advantages over all the open-source models in the context of API-Bank, demonstrating the effectiveness of our training data for functional calling.

## Citation

```bibtex
@misc{liu2024toolace,
  title={ToolACE: Winning the Points of LLM Function Calling},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2409.00920}
}
```

- arXiv: 2409.00920