Internvl35 Multimodal Eval

---
name: internvl35-multimodal-eval
description: Evaluates multimodal large language models across general understanding, complex reasoning, mathematics, OCR, document comprehension, and agentic/GUI interaction tasks. Use when the user wants to benchmark on MMMU, MathVista, MMStar, MMVet, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.18265
  bibtex_key: wang2025internvl35
  confidence: high
---

# internvl35-multimodal-eval

> InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency — Wang et al. (2025) (arXiv:2508.18265, 2025)

## What this evaluates

Evaluates multimodal large language models across general understanding, complex reasoning, mathematics, OCR, document comprehension, and agentic/GUI interaction tasks.

## Datasets

- **MMMU** — total ?; splits: val (-1)
- **MathVista** — total ?; splits: mini (-1)
- **MMStar** — total ?; splits: test (-1)
- **MMVet** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly predicted answers out of the total number of instances. Calculated as (correct predictions / total instances) * 100. For benchmarks like MMVet, scores are derived via LLM-as-judge evaluation.

## Input / output format

**Input**: Image or video frames paired with a text prompt/question. For agentic tasks, GUI screenshots or environment states with action instructions.

**Output**: Text response containing the final answer, reasoning steps, or action sequence.

## Scoring recipe

```python
def calculate_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if normalize_answer(p) == g)
    return (correct / len(gold)) * 100

# For LLM-as-judge benchmarks (e.g., MMVet):
# score = llm_judge.evaluate(predictions, gold, prompt_template=judge_prompt)
# return score
```

## Common pitfalls

- Benchmark versions vary significantly (e.g., MMBench V1.1 vs V2.0); results are not directly comparable without specifying the exact version.
- MMVet relies on LLM-as-judge scoring, which is highly sensitive to the judge model, version, and prompt template used.
- Agentic benchmarks (e.g., OSWorld-G, ScreenSpot) require interactive environment execution, not just static QA generation.

## Evidence (verbatim from paper)

> To comprehensively evaluate the multimodal reasoning and mathematical capabilities of InternVL3.5, we conduct extensive experiments across a series of benchmarks, including MMMU for multidisciplinary reasoning, MathVista, MathVision, and MathVerse for mathematical reasoning... The overall score is the average score of all benchmarks.

## Citation

```bibtex
@misc{wang2025internvl35,
  title={InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2508.18265}
}
```

- arXiv: 2508.18265