Omibench Eval

---
name: omibench-eval
description: Evaluates large vision-language models on Olympiad-level multi-image reasoning tasks across biology, chemistry, mathematics, and physics. It probes the model's ability to integrate complementary visual and textual evidence across multiple images to generate stepwise rationales and select or produce correct final answers. Use when the user wants to benchmark on OMIBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.20806
  bibtex_key: [REDACTED]
  confidence: medium
---

# omibench-eval

> OMIBench: Benchmarking Olympiad-Level Multi-Image Reasoning in Large Vision-Language Model — Chen et al. (2026) (arXiv:2604.20806, 2026)

## What this evaluates

Evaluates large vision-language models on Olympiad-level multi-image reasoning tasks across biology, chemistry, mathematics, and physics. It probes the model's ability to integrate complementary visual and textual evidence across multiple images to generate stepwise rationales and select or produce correct final answers.

## Datasets

- **OMIBench** — total 1000; splits: test (-1); repo https://github.com/LightChen233/OMIBench

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard exact-match accuracy: the proportion of questions where the model's final answer matches the ground truth option or value.

## Input / output format

**Input**: A set of $n \geq 2$ images $\mathcal{I}$, a question $Q$, a context $C$, and a set of options $\mathcal{O}$ (for multiple-choice tasks) or an open answer space (for open-ended tasks).

**Output**: A stepwise rationale $\mathcal{R}_m = \{s_1, \ldots, s_m\}$ followed by a final answer $\mathcal{Y}$, either selected from $\mathcal{O}$ or generated from $\mathcal{A}$.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_answers):
    if pred == gold:
        correct += 1
return correct / len(predictions)
```

## Common pitfalls

- Models often analyze images in isolation rather than cross-referencing complementary evidence across them.
- Open-ended tasks rely on exact-match accuracy, which may penalize semantically correct but differently phrased answers.

## Evidence (verbatim from paper)

> Unlike single-image multimodal CoT, multi-image CoT considers a set of images $\mathcal{I}\={I_{1},I_{2},\ldots,I_{n}|n\geq 2}$, a question $Q$, and a context $C$. The task is to answer $Q$ by integrating evidence across multiple images... Finally, the model selects the final answer $\mathcal{Y}$ from close option set $\mathcal{O}$... revealing that even top LVLMs like Gemini-3-Pro achieve only ~50% accuracy

## Citation

```bibtex
@misc{chen2026omibench,
  title={OMIBench: Benchmarking Olympiad-Level Multi-Image Reasoning in Large Vision-Language Model},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2604.20806}
}
```

- arXiv: 2604.20806