Matvqa Eval

---
name: matvqa-eval
description: Evaluates multimodal large language models' ability to perform fine-grained visual-scientific reasoning in materials science. It probes structure-property-performance relationships through quantitative, comparative, causal, and hypothetical variation tasks, requiring models to integrate visual data from experimental figures with domain-specific knowledge rather than relying on textual shortcuts. Use when the user wants to benchmark on MatVQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.18319
  bibtex_key: wu2025matvqa
  confidence: high
---

# matvqa-eval

> Seeing Beyond Words: MatVQA for Challenging Visual-Scientific Reasoning in Materials Science — Wu et al. (2025) (arXiv:2505.18319, 2025)

## What this evaluates

Evaluates multimodal large language models' ability to perform fine-grained visual-scientific reasoning in materials science. It probes structure-property-performance relationships through quantitative, comparative, causal, and hypothetical variation tasks, requiring models to integrate visual data from experimental figures with domain-specific knowledge rather than relying on textual shortcuts.

## Datasets

- **MatVQA** — total 1325; splits: Causal (950), Quantitative (7), Comparative (-1), Hypothetical (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions. Calculated as the number of predictions matching the ground-truth answer divided by the total number of instances, reported per task split and overall.

## Input / output format

**Input**: An experimental figure (e.g., material structure diagram or property plot) paired with a natural language question and multiple-choice options.

**Output**: A selected answer choice (letter or text) corresponding to the correct option, optionally accompanied by a chain-of-thought explanation.

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
    return (correct / len(gold)) * 100
```

## Common pitfalls

- Models may exploit textual or caption shortcuts rather than performing genuine visual analysis, artificially inflating scores on raw questions.
- The Quantitative split contains only 7 items, so overall accuracy is heavily dominated by Causal and Comparative tasks and does not reliably reflect numeric reasoning capability.
- Domain-specific fine-tuning on unrelated visual modalities (e.g., optical chemical structures) can negatively bias performance on this benchmark.

## Evidence (verbatim from paper)

> The uniformly low accuracy proved that MatVQA is challenging for both large language models and small language models. These limitations likely stem from a combination of factors, including the nuanced visual perception required for material-scientific figures and the sophisticated reasoning demanded by tasks such as comparative and hypothetical analysis, which were identified as particularly challenging.

## Citation

```bibtex
@misc{wu2025matvqa,
  title={Seeing Beyond Words: MatVQA for Challenging Visual-Scientific Reasoning in Materials Science},
  author={Wu et al. (2025)},
  year={2025},
  note={arXiv:2505.18319}
}
```

- arXiv: 2505.18319