Bfcl Eval

---
name: bfcl-eval
description: Evaluates an LLM's ability to generate correct function calls from natural language prompts, covering single, multiple, parallel, and parallel-multiple API invocations across different programming languages. Use when the user wants to benchmark on Berkeley Function-Calling Benchmark (BFCL), or asks about evaluating this task. Reports Overall Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.18518
  bibtex_key: liu2024apigen
  confidence: high
---

# bfcl-eval

> APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets — Liu et al. (2024) (arXiv:2406.18518, 2024)

## What this evaluates

Evaluates an LLM's ability to generate correct function calls from natural language prompts, covering single, multiple, parallel, and parallel-multiple API invocations across different programming languages.

## Datasets

- **Berkeley Function-Calling Benchmark (BFCL)** — total 2000; splits: test (2000)

## Metrics

- `Overall Accuracy` **(primary)** — range: percent
  - Weighted average of AST Evaluation and Executable Function Evaluation scores across four complexity levels (Simple, Multiple, Parallel, Parallel Multiple), combined with Relevance Detection.
- `AST Evaluation` — range: percent
  - Measures syntactic accuracy of generated function calls against predefined documentation, checking function names, required parameters, and data types.
- `Executable Function Evaluation` — range: percent
  - Runs the generated function calls to verify operational correctness and expected results.

## Input / output format

**Input**: Natural language queries requiring API calls, covering single, multiple, parallel, and parallel-multiple function invocations across Java, JavaScript, and Python APIs.

**Output**: JSON-format function calls specifying the function name and parameters.

## Scoring recipe

```python
def score_bfcl(predictions, gold):
    ast_scores, exec_scores = [], []
    for pred, gold_item in zip(predictions, gold):
        ast_scores.append(check_ast_structure(pred, gold_item.schema))
        exec_scores.append(run_and_compare(pred, gold_item.expected_output))
    # Weighted average across complexity levels (Simple, Multiple, Parallel, Parallel Multiple)
    overall = weighted_average(ast_scores, exec_scores, gold.complexity_levels)
    return overall
```

## Common pitfalls

- AST accuracy does not guarantee runtime correctness; models may output syntactically valid JSON that fails execution due to type mismatches or missing dependencies.
- Parallel and multiple function-calling scenarios are significantly harder and underrepresented in public datasets, leading to performance drops if models are only trained on simple single-call data.
- Relevance detection (identifying whether a function should be called at all) is evaluated separately from function call accuracy, and poor relevance detection can skew overall scores.

## Evidence (verbatim from paper)

> Benchmark. We evaluate the trained models' performance on the Berkeley Function-Calling Benchmark (BFCL) [9], which provides a comprehensive evaluation framework for assessing the function-calling capabilities of LLMs across various programming languages and application domains. ... The Berkeley Function-Calling Leaderboard (BFCL) evaluates LLMs using two main categories: Abstract Syntax Tree (AST) Evaluation and Executable Function Evaluation. ... The rank is based on the overall accuracy, which is a weighted average of different evaluation categories.

## Citation

```bibtex
@misc{liu2024apigen,
  title={APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2406.18518}
}
```

- arXiv: 2406.18518