Gorilla Api Eval

---
name: gorilla-api-eval
description: Evaluates an LLM's ability to generate correct API invocation code from natural language prompts, with or without retrieved documentation. It measures how well the model selects the appropriate API, avoids hallucinating non-existent APIs, and respects functional constraints like accuracy thresholds. Use when the user wants to benchmark on APIBench, or asks about evaluating this task. Reports AST accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.15334
  bibtex_key: patil2023gorilla
  confidence: high
---

# gorilla-api-eval

> Gorilla: Large Language Model Connected with Massive APIs — Patil et al. (2023) (arXiv:2305.15334, 2023)

## What this evaluates

Evaluates an LLM's ability to generate correct API invocation code from natural language prompts, with or without retrieved documentation. It measures how well the model selects the appropriate API, avoids hallucinating non-existent APIs, and respects functional constraints like accuracy thresholds.

## Datasets

- **APIBench** — total ?; splits: train (-1), test (-1)

## Metrics

- `AST accuracy` **(primary)** — range: percent
  - Percentage of generated API calls whose Abstract Syntax Tree (AST) exactly matches the ground truth API call. Reported alongside hallucination error rate and wrong-API selection error rate.

## Input / output format

**Input**: Natural language user prompt. Optionally concatenated with retrieved API documentation in the format: <user_prompt>Use this API documentation for reference: <retrieved_API_doc_json>.

**Output**: Python/API invocation code string (e.g., torchvision.models.resnet50()).

## Scoring recipe

```python
def score_api_call(generated_code, gold_code):
    gen_ast = parse_ast(generated_code)
    gold_ast = parse_ast(gold_code)
    if gen_ast == gold_ast:
        return 'correct'
    elif is_hallucination(generated_code):
        return 'hallucination'
    else:
        return 'wrong_api'
# Overall accuracy = count(correct) / total * 100
```

## Common pitfalls

- Using a non-optimal retriever (e.g., BM25) at test time can significantly misguide the model and drop accuracy by 20-50% compared to zero-shot prompting.
- For HuggingFace, the dataset is not exhaustive, so evaluation for baseline models is reduced to a multiple-choice domain selection task, making direct comparison with TorchHub/TensorFlow Hub AST accuracy metrics difficult.
- Hallucination metrics are highly sensitive to RLHF tuning; GPT-3.5 sometimes exhibits fewer hallucinations than GPT-4 in zero-shot settings, contrary to expectations.

## Evidence (verbatim from paper)

> We report the overall accuracy, the error by hallucination and the error by selecting wrong API call. Note that for TorchHub and TensorHub, we evaluate all the models using AST tree accuracy score.

## Citation

```bibtex
@misc{patil2023gorilla,
  title={Gorilla: Large Language Model Connected with Massive APIs},
  author={Patil et al. (2023)},
  year={2023},
  note={arXiv:2305.15334}
}
```

- arXiv: 2305.15334