Insight O3 Eval

---
name: insight-o3-eval
description: Evaluates multimodal reasoning and generalized visual search capabilities, measuring how well models can locate and reason about high-information-density images using a multi-agent framework with a dedicated visual search agent. Use when the user wants to benchmark on V*-Bench, Tree-Bench, VisualProbe-Hard, HR-Bench, MME-RealWorld, O3-Bench, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2512.18745
  bibtex_key: li2025insighto3
  confidence: high
---

# insight-o3-eval

> InSight-o3: Empowering Multimodal Foundation Models with Generalized Visual Search — Li et al. (2025) (arXiv:2512.18745, 2025)

## What this evaluates

Evaluates multimodal reasoning and generalized visual search capabilities, measuring how well models can locate and reason about high-information-density images using a multi-agent framework with a dedicated visual search agent.

## Datasets

- **V*-Bench** — total ?; splits: test (-1)
- **Tree-Bench** — total ?; splits: test (-1)
- **VisualProbe-Hard** — total ?; splits: test (-1)
- **HR-Bench** — total ?; splits: test (-1)
- **MME-RealWorld** — total 1919; splits: lite (1919)
- **O3-Bench** — total ?; splits: test (-1)

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions. Calculated as the number of correct predictions divided by the total number of predictions, multiplied by 100.

## Input / output format

**Input**: An image and a natural language question or query.

**Output**: A textual answer and/or bounding box coordinates for visual search tasks.

## Scoring recipe

```python
def calculate_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Image resolution constraints vary significantly by model API (e.g., OpenAI limits to 1280x1280px, while Gemini allows 3500x3500px), which impacts performance.
- Models may be reluctant to call the vSearcher tool or may use it incorrectly, leading to performance drops compared to standalone evaluation.

## Evidence (verbatim from paper)

> We evaluate a range of open and proprietary models/systems on the following benchmarks: (1) Natural-image benchmarks: V⋆-Bench, Tree-Bench, and VisualProbe-Hard. (2) Mixed benchmarks: HR-Bench and MME-RealWorld. ... (3) Our O3-Bench. In particular, the accuracy of GPT-5-mini on O3-Bench has improved from 39.0% to 61.5%.

## Citation

```bibtex
@misc{li2025insighto3,
  title={InSight-o3: Empowering Multimodal Foundation Models with Generalized Visual Search},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2512.18745}
}
```

- arXiv: 2512.18745