Wildsci Eval

---
name: wildsci-eval
description: Evaluates language models' scientific reasoning capabilities by testing their ability to answer domain-specific multiple-choice questions derived from peer-reviewed literature and established scientific benchmarks. Use when the user wants to benchmark on WildSci-Val, GPQA-Aug, SuperGPQA, MMLU-Pro, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2601.05567
  bibtex_key: liu2026wildsci
  confidence: high
---

# wildsci-eval

> WildSci: Advancing Scientific Reasoning from In-the-Wild Literature — Liu et al. (2026) (arXiv:2601.05567, 2026)

## What this evaluates

Evaluates language models' scientific reasoning capabilities by testing their ability to answer domain-specific multiple-choice questions derived from peer-reviewed literature and established scientific benchmarks.

## Datasets

- **WildSci-Val** — total 900; splits: val (900)
- **GPQA-Aug** — total 792; splits: test (792)
- **SuperGPQA** — total 26529; splits: test (26529)
- **MMLU-Pro** — total 12032; splits: test (12032)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Calculated as the proportion of questions where the model's final selected option exactly matches the ground-truth correct answer.

## Input / output format

**Input**: Multiple-choice scientific questions with 4 or 10 options and a single correct answer.

**Output**: The model's final selected option (e.g., 'A', 'B', 'C', or 'D').

## Scoring recipe

```python
def compute_accuracy(predictions, gold):
    correct = sum(1 for pred, gold_ans in zip(predictions, gold) if pred.strip().upper() == gold_ans.strip().upper())
    return correct / len(predictions)
```

## Common pitfalls

- Answer choice order bias is explicitly mitigated for GPQA by using GPQA-Aug (4 permutations), but this augmentation is not applied to other datasets.
- Models often output chain-of-thought reasoning before the final answer; only the final selected option is scored, requiring robust parsing to ignore intermediate text.

## Evidence (verbatim from paper)

> We report accuracy for each dataset by checking whether the final selected option in the model’s response is correct.

## Citation

```bibtex
@misc{liu2026wildsci,
  title={WildSci: Advancing Scientific Reasoning from In-the-Wild Literature},
  author={Liu et al. (2026)},
  year={2026},
  note={arXiv:2601.05567}
}
```

- arXiv: 2601.05567