Scp 116k Eval

---
name: scp-116k-eval
description: Evaluates the scientific reasoning and problem-solving capabilities of LLMs on graduate-level higher education science problems. It measures how well models can parse and solve complex scientific questions involving formulas and equations. Use when the user wants to benchmark on SCP-116K, or asks about evaluating this task. Reports Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.15587
  bibtex_key: lu2025scp116k
  confidence: high
---

# scp-116k-eval

> SCP-116K: A High-Quality Problem-Solution Dataset and a Generalized Pipeline for Automated Extraction in the Higher Education Science Domain — Lu et al. (2025) (arXiv:2501.15587, 2025)

## What this evaluates

Evaluates the scientific reasoning and problem-solving capabilities of LLMs on graduate-level higher education science problems. It measures how well models can parse and solve complex scientific questions involving formulas and equations.

## Datasets

- **SCP-116K** — total 116756; splits: (unstated); repo https://github.com/AQA6666/SCP-116K-open

## Metrics

- `Accuracy` **(primary)** — range: percent
  - Percentage of correctly solved problems out of the total number of problems in the dataset.

## Input / output format

**Input**: Scientific problem statements, often containing complex formulas and equations, extracted from higher education science materials.

**Output**: Not explicitly specified in the provided text.

## Scoring recipe

```python
def calculate_accuracy(predictions, gold):
    correct = sum(1 for pred, gold in zip(predictions, gold) if pred == gold)
    return (correct / len(gold)) * 100
```

## Common pitfalls

- The evaluation section does not specify train/val/test splits for SCP-116K, making reproducibility of baseline splits difficult.
- Distillation experiments evaluate final performance on GPQA-diamond rather than SCP-116K, which may not directly reflect improvement on the target dataset.

## Evidence (verbatim from paper)

> These models achieve accuracy rates of 58.40% and 55.79%, respectively, on SCP-116K.

## Citation

```bibtex
@misc{lu2025scp116k,
  title={SCP-116K: A High-Quality Problem-Solution Dataset and a Generalized Pipeline for Automated Extraction in the Higher Education Science Domain},
  author={Lu et al. (2025)},
  year={2025},
  note={arXiv:2501.15587}
}
```

- arXiv: 2501.15587