Vista Multimodal Eval

---
name: vista-multimodal-eval
description: Evaluates cross-modal vision-text alignment in Multimodal Large Language Models (MLLMs) across high-level semantic VQA, general multimodal understanding, and fine-grained visual perception/retrieval tasks. Use when the user wants to benchmark on VQAv2, OK-VQA, GQA, TextVQA, RealWorldQA, DocVQA, MMBench, SEED, AI2D, MMMU, MMStar, MME, RefCOCO, RefCOCO+, RefCOCOg, or asks about evaluating this task. Reports performance.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.10917
  bibtex_key: li2025vista
  confidence: high
---

# vista-multimodal-eval

> VISTA: Enhancing Vision-Text Alignment in MLLMs via Cross-Modal Mutual Information Maximization — Li et al. (2025) (arXiv:2505.10917, 2025)

## What this evaluates

Evaluates cross-modal vision-text alignment in Multimodal Large Language Models (MLLMs) across high-level semantic VQA, general multimodal understanding, and fine-grained visual perception/retrieval tasks.

## Datasets

- **VQAv2** — total ?; splits: test (-1)
- **OK-VQA** — total ?; splits: test (-1)
- **GQA** — total ?; splits: test (-1)
- **TextVQA** — total ?; splits: test (-1)
- **RealWorldQA** — total ?; splits: test (-1)
- **DocVQA** — total ?; splits: test (-1)
- **MMBench** — total ?; splits: test (-1)
- **SEED** — total ?; splits: test (-1)
- **AI2D** — total ?; splits: test (-1)
- **MMMU** — total ?; splits: test (-1)
- **MMStar** — total ?; splits: test (-1)
- **MME** — total ?; splits: test (-1)
- **RefCOCO** — total ?; splits: test (-1)
- **RefCOCO+** — total ?; splits: test (-1)
- **RefCOCOg** — total ?; splits: test (-1)

## Metrics

- `performance` **(primary)** — range: percent
  - Percentage of correct predictions or benchmark-specific score. Aggregated as average improvement over baseline. Evaluated via lmms-eval framework.

## Input / output format

**Input**: Paired image and text prompt/question.

**Output**: Generated text response or selected multiple-choice option.

## Scoring recipe

```python
def compute_performance(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if normalize_answer(p) == normalize_answer(g))
    return correct / len(gold) * 100
# For MME/RefCOCO, use lmms-eval built-in scorers which follow standard benchmark protocols.
```

## Common pitfalls

- RefCOCO performance is sensitive to query length; shorter queries may show dips while longer ones improve.
- Benchmarks like AI2D and MMBench-cn rely heavily on text understanding, so vision-text alignment methods may cause slight performance drops compared to text-only baselines.
- MME reports separate Perception and Cognition scores; aggregating them incorrectly can misrepresent model capabilities.

## Evidence (verbatim from paper)

> VISTA consistently improves performance on high-level semantic VQA tasks, validating its effectiveness in enhancing vision-text alignment for complex reasoning. As shown in Table[1], integrating our proposed VISTA alignment objective into both TinyLLaVA-3B and LLaVA-v1.5-7B models yields significant performance gains over their original counterparts on high-level VQA datasets.

## Citation

```bibtex
@misc{li2025vista,
  title={VISTA: Enhancing Vision-Text Alignment in MLLMs via Cross-Modal Mutual Information Maximization},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2505.10917}
}
```

- arXiv: 2505.10917