Megascience Eval

---
name: megascience-eval
description: Evaluates large language models' scientific reasoning capabilities across general science, specialized domains (chemistry, CS, medicine, physics), and mathematical problem-solving. It tests the model's ability to follow chain-of-thought prompting, extract precise answers (including units), and correctly identify multiple-choice options. Use when the user wants to benchmark on MegaScience Evaluation Suite (MMLU, GPQA-Diamond, MMLU-Pro, SuperGPQA, SciBench, OlympicArena, ChemBench, CS-Bench, MedQA, MedMCQA, PubMedQA, PIQA, GSM8K, MATH, MATH500), or asks about evaluating this task. Reports EM (Exact Match).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2507.16812
  bibtex_key: fan2025megascience
  confidence: high
---

# megascience-eval

> MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoning — Fan et al. (2025) (arXiv:2507.16812, 2025)

## What this evaluates

Evaluates large language models' scientific reasoning capabilities across general science, specialized domains (chemistry, CS, medicine, physics), and mathematical problem-solving. It tests the model's ability to follow chain-of-thought prompting, extract precise answers (including units), and correctly identify multiple-choice options.

## Datasets

- **MegaScience Evaluation Suite (MMLU, GPQA-Diamond, MMLU-Pro, SuperGPQA, SciBench, OlympicArena, ChemBench, CS-Bench, MedQA, MedMCQA, PubMedQA, PIQA, GSM8K, MATH, MATH500)** — total ?; splits: test (-1)

## Metrics

- `EM (Exact Match)` **(primary)** — range: [0, 1]
  - Exact match accuracy between the extracted model prediction and the gold answer. For computational problems requiring units, both the numerical value and the unit must match exactly.

## Input / output format

**Input**: Benchmark questions (multiple-choice or computational problems) provided with chain-of-thought prompting templates.

**Output**: Free-text model responses containing reasoning steps and a final answer, which may be formatted using \boxed{}, option labels, or plain text.

## Scoring recipe

```python
def score(predictions, golds):
    matches = 0
    for pred, gold in zip(predictions, golds):
        # Stage 1: Identify answer indicator phrases
        answer = extract_by_indicator(pred)
        # Stage 2: Fallback to formatting patterns (\boxed{}, \mathrm{}, etc.)
        if not answer:
            answer = extract_by_format(pred)
        # Stage 3: For MCQs, match option content if label extraction fails
        if not answer:
            answer = match_option_label(pred)
        # Stage 4: Exact match (including unit if required)
        if normalize(answer) == normalize(gold):
            matches += 1
    return matches / len(predictions)
```

## Common pitfalls

- Relying solely on \boxed{} extraction misses valid answers that lack this formatting, incorrectly penalizing models.
- Failing to match option labels for multiple-choice questions when direct extraction of the label fails.
- Ignoring unit requirements for computational problems, leading to false negatives when only the numerical value matches.

## Evidence (verbatim from paper)

> EM (unit) represents exact match accuracy for both the numerical answer and its corresponding unit. To enhance extraction precision, we develop a comprehensive set of rule-based methods tailored to extract answers across diverse question types. Our answer extraction method operates through a two-stage process: (1) identifying answer indicator phrases that signal the presence of a final answer, and (2) extracting the answer content from various formatting patterns.

## Citation

```bibtex
@misc{fan2025megascience,
  title={MegaScience: Pushing the Frontiers of Post-Training Datasets for Science Reasoning},
  author={Fan et al. (2025)},
  year={2025},
  note={arXiv:2507.16812}
}
```

- arXiv: 2507.16812