Mascqa Eval

---
name: mascqa-eval
description: Evaluates large language models' domain-specific reasoning and numerical problem-solving capabilities in materials science and metallurgical engineering. It probes their ability to accurately answer multiple-choice, matching, and numerical questions, highlighting gaps in scientific reasoning and computational precision. Use when the user wants to benchmark on MaScQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.04277
  bibtex_key: [REDACTED]
  confidence: high
---

# mascqa-eval

> Exploring the Expertise of Large Language Models in Materials Science and Metallurgical Engineering — Bajan et al. (2025) (arXiv:2501.04277, 2025)

## What this evaluates

Evaluates large language models' domain-specific reasoning and numerical problem-solving capabilities in materials science and metallurgical engineering. It probes their ability to accurately answer multiple-choice, matching, and numerical questions, highlighting gaps in scientific reasoning and computational precision.

## Datasets

- **MaScQA** — total 644; splits: test (644); repo https://github.com/Lambard-ML-Team/LLM

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of questions answered correctly, averaged over three submissions per question. Correctness is determined by an AI judge that verifies answer equivalence and reasoning soundness.

## Input / output format

**Input**: A prompt containing the question, the correct answer, the model's predicted answer, and the question type (MCQ, MATCH, MCQN, or NUM).

**Output**: A structured judgment block: <judgment>\nExplanation: [brief explanation]\nResult: [CORRECT or INCORRECT]\n</judgment>

## Scoring recipe

```python
correct_count = 0
total = len(dataset) * 3
for q in dataset:
    for _ in range(3):
        pred = model.generate(q)
        judge_output = run_judge(q, q.correct_answer, pred, q.type)
        if 'CORRECT' in judge_output:
            correct_count += 1
accuracy = (correct_count / total) * 100
```

## Common pitfalls

- Numerical questions (NUM/MCQN) are particularly challenging due to tokenization inefficiencies and multi-step reasoning requirements, often causing models to perform near random baseline.
- Models may select correct options in MCQ/MATCH categories through pattern exploitation or guessing without sound underlying reasoning.
- Hardware constraints (e.g., running on MAC vs GPU) significantly impact open-source model performance and stability.

## Evidence (verbatim from paper)

> a) Based on the question and the correct answer, You must tell if the other answer is correct or not by answering only with Correct or Incorrect ... The evaluation of the LLMs, on Table 5 and Figure 6, demonstrates that Claude-3.5-Sonnet and GPT-4o are among the top performers, achieving overall accuracies of approximately 84% (see Figure 1 in the supplementary materials for details concerning the LLMs' average accuracy on each category MATCH, MCQ, MCQN, and NUM).

## Citation

```bibtex
@misc{bajan2025exploring,
  title={Exploring the Expertise of Large Language Models in Materials Science and Metallurgical Engineering},
  author={Bajan et al. (2025)},
  year={2025},
  note={arXiv:2501.04277}
}
```

- arXiv: 2501.04277