Matscibench Eval

---
name: matscibench-eval
description: This benchmark evaluates the reasoning capabilities of large language models in materials science, covering six primary fields and 31 sub-fields. It probes domain knowledge, mathematical/formula reasoning, and multimodal visual comprehension through expert-curated problems with three-tier difficulty classifications. Use when the user wants to benchmark on MatSciBench, or asks about evaluating this task. Reports Accuracy Score (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.12171
  bibtex_key: [REDACTED]
  confidence: high
---

# matscibench-eval

> MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Science — Zhang et al. (2025) (arXiv:2510.12171, 2025)

## What this evaluates

This benchmark evaluates the reasoning capabilities of large language models in materials science, covering six primary fields and 31 sub-fields. It probes domain knowledge, mathematical/formula reasoning, and multimodal visual comprehension through expert-curated problems with three-tier difficulty classifications.

## Datasets

- **MatSciBench** — total 1340; splits: test (-1); repo https://github.com/Jun-Kai-Zhang/MatSciBench.git

## Metrics

- `Accuracy Score (%)` **(primary)** — range: percent
  - Percentage of correctly answered questions. Evaluated via a hybrid rule-based and LLM-based system. Rule-based checks apply a relaxed 5% numerical tolerance for calculations. For complex formulas/equations, Gemini-2.0-Flash provides the final correctness judgment.

## Input / output format

**Input**: Text-based questions or multimodal questions containing images, presented with prompts for basic chain-of-thought (CoT), self-correction (3-round iterative refinement), or tool-augmentation (Python code generation and execution).

**Output**: Final answer string (numerical value, formula, or text). For tool-augmentation, the model outputs Python code which is executed to derive the answer.

## Scoring recipe

```python
def score_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        if is_formula_type(gold):
            if llm_judge(pred, gold): correct += 1
        else:
            if rule_check(pred, gold, tolerance=0.05): correct += 1
    return (correct / len(golds)) * 100
```

## Common pitfalls

- Self-correction prompting often degrades performance across most models rather than improving it.
- Tool-augmentation yields inconsistent results; some models show significant accuracy drops when using code interpreters.
- Multimodal evaluation suffers from severe performance drops compared to text-only, largely due to visual context misinterpretation by models.

## Evidence (verbatim from paper)

> The performance in terms of accuracy score of all models on text-only questions is presented in Table 1, and the performance of multimodal models on images-included questions is presented in Table 2. The correctness of the output answers is evaluated using a hybrid approach that combines rule-based evaluation and LLM-based evaluation. We adapt the rule-based evaluation system from Qwen-2.5 Math (Yang et al., [2024a]). Following the previous works(Methani et al., [2020]; Gupta et al., [2024]), we apply a relaxed numerical tolerance of 5% to account for approximation errors in calculations and image recognition. To address the limitations of rule-based systems in handling complex formulas and equations, we supplement this approach with Gemini-2.0-Flash for formula-type questions. The LLM’s judgment serves as the final determinant of correctness for these complex mathematical expressions.

## Citation

```bibtex
@misc{zhang2025matscibench,
  title={MatSciBench: Benchmarking the Reasoning Ability of Large Language Models in Materials Science},
  author={Zhang et al. (2025)},
  year={2025},
  note={arXiv:2510.12171}
}
```

- arXiv: 2510.12171