Mattermech Eval

---
name: mattermech-eval
description: Evaluates large language models' ability to reason about physicochemical principles in nanomaterial synthesis. It probes whether models can generate scientifically valid hypotheses and understand conceptual mechanisms from literature abstracts, rather than relying on abstract logic alone. Use when the user wants to benchmark on MatterMech, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.25281
  bibtex_key: pu2025mattermech
  confidence: high
---

# mattermech-eval

> Mechanisms of Matter: Language Inferential Benchmark on Physicochemical Hypothesis in Materials Synthesis — Pu et al. (2025) (arXiv:2509.25281, 2025)

## What this evaluates

Evaluates large language models' ability to reason about physicochemical principles in nanomaterial synthesis. It probes whether models can generate scientifically valid hypotheses and understand conceptual mechanisms from literature abstracts, rather than relying on abstract logic alone.

## Datasets

- **MatterMech** — total ?; splits: test (-1); repo https://github.com/amair-lab/MatterMech

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Proportion of correctly answered multiple-choice (MCQ) and text-insertion format (TIF) questions. Calculated as the number of correct predictions divided by the total number of evaluated questions.

## Input / output format

**Input**: Context: paper abstract (and metadata). Prompt: A multiple-choice question or a text-insertion prompt requiring a single-word/phrase completion based on physicochemical principles.

**Output**: For MCQ: the selected option letter/text. For TIF: a single word or phrase.

## Scoring recipe

```python
correct = 0
for pred, gold in zip(predictions, gold_answers):
    if pred.strip().lower() == gold.strip().lower():
        correct += 1
return correct / len(predictions)
```

## Common pitfalls

- Models may hallucinate physicochemical terminology not present in the source abstract, leading to biased or incorrect reasoning.
- Distractors in MCQs might be scientifically plausible or trivially incorrect, requiring careful validation to avoid easy guessing.
- TIF format evaluation is sensitive to exact string matching; minor formatting or synonym variations can incorrectly penalize valid answers.

## Evidence (verbatim from paper)

> Reviewer evaluates their accuracy and checks for potential hallucinations, and the Critic concludes the discussion upon reaching consensus.

## Citation

```bibtex
@misc{pu2025mattermech,
  title={Mechanisms of Matter: Language Inferential Benchmark on Physicochemical Hypothesis in Materials Synthesis},
  author={Pu et al. (2025)},
  year={2025},
  note={arXiv:2509.25281}
}
```

- arXiv: 2509.25281