Omnigaiatoolreasoning Eval

---
name: omnigaiatoolreasoning-eval
description: Evaluates native omni-modal AI agents' ability to perform multi-hop cross-modal reasoning across video, audio, and image inputs. It probes their capacity to integrate external tools (web search, browser, code execution) for evidence gathering and to produce verifiable open-form answers under varying task difficulties. Use when the user wants to benchmark on OmniGAIA, or asks about evaluating this task. Reports Pass@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.22897
  bibtex_key: li2026omnigaiatowards
  confidence: high
---

# omnigaiatoolreasoning-eval

> OmniGAIA: Towards Native Omni-Modal AI Agents — Xiaoxi Li et al. (2026) (arXiv:2602.22897, 2026)

## What this evaluates

Evaluates native omni-modal AI agents' ability to perform multi-hop cross-modal reasoning across video, audio, and image inputs. It probes their capacity to integrate external tools (web search, browser, code execution) for evidence gathering and to produce verifiable open-form answers under varying task difficulties.

## Datasets

- **OmniGAIA** — total ?; splits: test (-1); repo https://github.com/RUC-NLPIR/OmnigAIA

## Metrics

- `Pass@1` **(primary)** — range: percent
  - The percentage of tasks where the model's final answer is judged semantically equivalent to the ground truth. Equivalence is determined by an LLM-as-a-Judge (DeepSeek-V3.2) using a standardized prompt.

## Input / output format

**Input**: Multi-modal inputs (video, audio, images) containing real-world events, accompanied by multi-hop natural language queries. Models are provided access to external tools (web search, browser, code executor).

**Output**: A final open-form textual answer to the query.

## Scoring recipe

```python
def compute_pass_at_1(predictions, gold_answers, judge_model="DeepSeek-V3.2"):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if judge_model.evaluate_equivalence(pred, gold):
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- LLM-as-a-Judge equivalence judgments are prompt-sensitive; results depend heavily on the specific judging prompt design.
- High tool-call frequency does not guarantee success; models may exhibit 'thrashing' behaviors that waste budget without improving accuracy.
- Native perception vs. tool-based perception trade-offs are task-dependent; tools can help on Easy/Medium tasks but degrade Hard task performance.

## Evidence (verbatim from paper)

> Pass@1 is reported, where a trial is considered correct if the model's final answer is judged equivalent to the ground truth. The judging prompt is detailed in Appendix B. All models are provided with the same external tools, including web search, browser, and code executor.

## Citation

```bibtex
@misc{li2026omnigaiatowards,
  title={OmniGAIA: Towards Native Omni-Modal AI Agents},
  author={Xiaoxi Li et al. (2026)},
  year={2026},
  note={arXiv:2602.22897}
}
```

- arXiv: 2602.22897