Omnimodal Qa Eval

---
name: omnimodal-qa-eval
description: This benchmark probes an agent's ability to perform long-horizon question answering over audio-video streams. It specifically tests fine-grained multimodal retrieval, multi-turn tool calling, and budget-aware reasoning when key evidence is scattered across time. Use when the user wants to benchmark on OmniVideoBench, WorldSense, Daily-Omni, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.03707
  bibtex_key: [REDACTED]
  confidence: high
---

# omnimodal-qa-eval

> OmniRAG-Agent: Agentic Omnimodal Reasoning for Low-Resource Long Audio-Video Question Answering — Zhu et al. (2026) (arXiv:2602.03707, 2026)

## What this evaluates

This benchmark probes an agent's ability to perform long-horizon question answering over audio-video streams. It specifically tests fine-grained multimodal retrieval, multi-turn tool calling, and budget-aware reasoning when key evidence is scattered across time.

## Datasets

- **OmniVideoBench** — total ?; splits: test (-1)
- **WorldSense** — total ?; splits: test (-1)
- **Daily-Omni** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions. The paper reports accuracy on each ability subset and averages them to measure overall performance across heterogeneous reasoning skills.

## Input / output format

**Input**: Long audio-video streams paired with natural language questions.

**Output**: Textual answers to the questions.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Averaging accuracy across heterogeneous ability subsets may mask performance on specific reasoning skills.
- The exact answer verification method (string match vs. LLM judge) follows each benchmark's official protocol but is not explicitly detailed in the main text.
- Performance varies significantly with the retrieval budget (number of clips) and RL training state, so results are not static.

## Evidence (verbatim from paper)

> For these datasets, we report accuracy on each ability subset and their average score, which measures overall performance across heterogeneous reasoning skills.

## Citation

```bibtex
@misc{zhu2026omniragagent,
  title={OmniRAG-Agent: Agentic Omnimodal Reasoning for Low-Resource Long Audio-Video Question Answering},
  author={Zhu et al. (2026)},
  year={2026},
  note={arXiv:2602.03707}
}
```

- arXiv: 2602.03707