Mme Sci Eval

---
name: mme-sci-eval
description: Evaluates multimodal large language models on scientific reasoning across four disciplines (math, physics, chemistry, biology) and five languages. It probes cross-lingual consistency, modality robustness (text-only vs. image-only vs. image-text), and fine-grained domain knowledge under varying visual complexity. Use when the user wants to benchmark on MME-SCI, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2508.13938
  bibtex_key: ruan2025mme_sci
  confidence: high
---

# mme-sci-eval

> MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models — Ruan et al. (2025) (arXiv:2508.13938, 2025)

## What this evaluates

Evaluates multimodal large language models on scientific reasoning across four disciplines (math, physics, chemistry, biology) and five languages. It probes cross-lingual consistency, modality robustness (text-only vs. image-only vs. image-text), and fine-grained domain knowledge under varying visual complexity.

## Datasets

- **MME-SCI** — total 1019; splits: test (-1); repo https://github.com/JCruan519/MME-SCI

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions. Correctness is determined by an LLM-as-a-Judge comparing the model's generated response against the ground-truth answer using language-specific evaluation templates.

## Input / output format

**Input**: A scientific question in one of five languages, optionally accompanied by an image (e.g., screenshot of a diagram or problem). The evaluation covers three modalities: text-only, image-only, and image-text.

**Output**: A natural language response answering the scientific question.

## Scoring recipe

```python
judge = LLM(temperature=0, max_tokens=8192)
correct = 0
for pred, gold, lang in zip(predictions, gold_answers, languages):
    prompt = build_judge_template(pred, gold, lang)
    verdict = judge.generate(prompt)
    if is_correct(verdict): correct += 1
accuracy = (correct / len(predictions)) * 100
```

## Common pitfalls

- Image-only evaluation uses screenshot-based inputs that heavily penalize models with weak OCR or vision-language alignment, causing disproportionate accuracy drops for open-source models.
- LLM-as-a-Judge scoring requires language-specific templates; using a single English template across multilingual data introduces bias and inflates or deflates accuracy unfairly.
- Performance varies drastically by domain (e.g., physics vs. chemistry) and reasoning capability, so reporting only aggregate accuracy masks critical modality or subject-specific failures.

## Evidence (verbatim from paper)

> We adopt the ‘LLM-as-a-Judge’ paradigm (Zheng et al. [2023]; Gu et al. [2024]) and introduce different evaluation templates for various languages to assess the models. Unless specified otherwise, we configured the maximum number of new tokens to 8,192, and the temperature was set to 0. Compared to advanced closed-source models, the most powerful open-source models (Large group) demonstrates an average accuracy reduction of 13.94% across six scenarios.

## Citation

```bibtex
@misc{ruan2025mme_sci,
  title={MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models},
  author={Ruan et al. (2025)},
  year={2025},
  note={arXiv:2508.13938}
}
```

- arXiv: 2508.13938