Gaia Eval

---
name: gaia-eval
description: GAIA probes the ability of AI assistants to perform real-world, conceptually simple tasks that require multi-step reasoning, tool use, and multi-modal processing. It measures robustness in practical everyday reasoning and factual validation on questions explicitly designed to be outside the model's training data. Use when the user wants to benchmark on GAIA, or asks about evaluating this task. Reports score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2311.12983
  bibtex_key: mialon2023gaia
  confidence: high
---

# gaia-eval

> GAIA: a benchmark for General AI Assistants — Mialon et al. (2023) (arXiv:2311.12983, 2023)

## What this evaluates

GAIA probes the ability of AI assistants to perform real-world, conceptually simple tasks that require multi-step reasoning, tool use, and multi-modal processing. It measures robustness in practical everyday reasoning and factual validation on questions explicitly designed to be outside the model's training data.

## Datasets

- **GAIA** — total 466; splits: test (466)

## Metrics

- `score` **(primary)** — range: [0, 1]
  - The fraction of questions answered correctly out of the total benchmark set. Answers are automatically validated against ground truth to ensure unambiguous evaluation.

## Input / output format

**Input**: A prefix prompt specifying the required answer format, followed by a real-world question that may require web search, file reading, computation, or multi-modal processing.

**Output**: A structured answer adhering to the format specified in the prefix prompt (exact format details are in Figure 2 of the paper).

## Scoring recipe

```python
correct = 0
for question, gold_answer in dataset:
    prediction = model.generate(prefix_prompt + question)
    if prediction == gold_answer:
        correct += 1
accuracy = correct / len(dataset)
```

## Common pitfalls

- GPT-4 with plugins score is an 'oracle' estimate because plugins are manually selected per task and are unstable or disappear, making it non-reproducible.
- Non-tool models can achieve non-zero scores on tool/modality tasks due to memorization of intermediate steps or alternative solving paths not intended by annotators.
- AutoGPT results are heavily influenced by its specific API prompt/generation parameters rather than just the underlying LLM capability.

## Evidence (verbatim from paper)

> Figure 4 Scores and time to answer per method and level. As stated in the main text, GPT4 + plugins score should be seen as an oracle since the plugins were chosen manually depending on the question.

## Citation

```bibtex
@misc{mialon2023gaia,
  title={GAIA: a benchmark for General AI Assistants},
  author={Mialon et al. (2023)},
  year={2023},
  note={arXiv:2311.12983}
}
```

- arXiv: 2311.12983