Msqa Eval

---
name: msqa-eval
description: Evaluates graduate-level materials science reasoning and factual knowledge through long-form explanatory answers and binary true/false questions. It probes model capabilities in domain-specific knowledge retrieval, multi-step scientific reasoning, and accuracy under both direct prompting and retrieval-augmented generation (RAG) settings. Use when the user wants to benchmark on MSQA, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.23982
  bibtex_key: cheung2025msqa
  confidence: high
---

# msqa-eval

> MSQA: Benchmarking LLMs on Graduate-Level Materials Science Reasoning and Knowledge — Cheung et al. (2025) (arXiv:2505.23982, 2025)

## What this evaluates

Evaluates graduate-level materials science reasoning and factual knowledge through long-form explanatory answers and binary true/false questions. It probes model capabilities in domain-specific knowledge retrieval, multi-step scientific reasoning, and accuracy under both direct prompting and retrieval-augmented generation (RAG) settings.

## Datasets

- **MSQA** — total 1757; splits: test (1757); repo https://github.com/jerry3027/MSQA

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of correctly answered questions. For binary tasks, exact keyword matching for 'YES' or 'NO'. For long-answer tasks, GPT-4o judge classifies responses as 'correct', 'mostly correct', or 'incorrect'; the first two are counted as correct.

## Input / output format

**Input**: Question text. Optionally augmented with top-5 BM25-retrieved paragraphs from Methods/Results sections. For binary tasks, optionally includes chain-of-thought prompting instructions.

**Output**: Long-answer: free-text scientific explanation. Binary-answer: exact string 'YES' or 'NO'.

## Scoring recipe

```python
def compute_accuracy(preds, golds, task_type):
    correct = 0
    for p, g in zip(preds, golds):
        if task_type == 'binary':
            if p.strip().upper() in ('YES', 'NO') and p.strip().upper() == g.strip().upper():
                correct += 1
        elif task_type == 'long':
            judge_label = llm_judge_eval(p, g)
            if judge_label in ('correct', 'mostly correct'):
                correct += 1
    return (correct / len(preds)) * 100
```

## Common pitfalls

- Domain-specific fine-tuned models often underperform general-purpose LLMs due to overfitting and distributional shifts between their training data and the benchmark.
- Chain-of-thought prompting can degrade binary-answer accuracy by introducing factually incorrect intermediate reasoning steps.
- LLM-as-judge (e.g., GPT-4o-mini) exhibits verbosity bias and may accept vague explanations, requiring validation against stronger judges or human experts.

## Evidence (verbatim from paper)

> For binary-answer evaluations, accuracy is determined by exact keyword matching for responses containing either “YES” or “NO”. Performance results are uniformly reported as accuracy percentages across all experiments. Evaluating the long-answer responses is conducted through GPT-4o acting as an LLM judge, assessing responses as either “correct” or “mostly correct”, both categories counted as correct in our metrics.

## Citation

```bibtex
@misc{cheung2025msqa,
  title={MSQA: Benchmarking LLMs on Graduate-Level Materials Science Reasoning and Knowledge},
  author={Cheung et al. (2025)},
  year={2025},
  note={arXiv:2505.23982}
}
```

- arXiv: 2505.23982