Spice Reasoning Eval

---
name: spice-reasoning-eval
description: This evaluation probes a model's ability to solve challenging mathematical and general reasoning tasks, both from standard benchmarks and document-grounded self-play generated questions. It measures how well the model can extract information, perform multi-step logical deduction, and produce verifiable answers across diverse academic and competition-level datasets. Use when the user wants to benchmark on MATH-500, OlympiadBench, Minerva Math, GSM8K, AMC, AIME'24, AIME'25, SuperGPQA, GPQA-Diamond, MMLU-Pro, BBEH, or asks about evaluating this task. Reports pass rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.24684
  bibtex_key: liu2025spice
  confidence: high
---

# spice-reasoning-eval

> SPICE: Self-Play In Corpus Environments Improves Reasoning — Liu et al. (2025) (arXiv:2510.24684, 2025)

## What this evaluates

This evaluation probes a model's ability to solve challenging mathematical and general reasoning tasks, both from standard benchmarks and document-grounded self-play generated questions. It measures how well the model can extract information, perform multi-step logical deduction, and produce verifiable answers across diverse academic and competition-level datasets.

## Datasets

- **MATH-500** — total ?; splits: test (-1)
- **OlympiadBench** — total ?; splits: test (-1)
- **Minerva Math** — total ?; splits: test (-1)
- **GSM8K** — total ?; splits: test (-1)
- **AMC** — total ?; splits: test (-1)
- **AIME'24** — total ?; splits: test (-1)
- **AIME'25** — total ?; splits: test (-1)
- **SuperGPQA** — total ?; splits: test (-1)
- **GPQA-Diamond** — total ?; splits: test (-1)
- **MMLU-Pro** — total ?; splits: test (-1)
- **BBEH** — total ?; splits: test (-1)

## Metrics

- `pass rate` **(primary)** — range: percent
  - Percentage of correctly answered questions out of the total evaluated. Correctness is determined by checking answer equivalence using GPT-4o rather than exact string matching.

## Input / output format

**Input**: Standard benchmark questions or document-grounded questions, formatted with training-consistent prompts following Ma et al. (2025).

**Output**: Chain-of-thought reasoning steps followed by a final answer.

## Scoring recipe

```python
def compute_pass_rate(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if gpt4o_check_equivalence(pred, gold):
            correct += 1
    return (correct / len(predictions)) * 100
# Note: AIME'24 and AIME'25 require averaging over 32 sampling runs instead of greedy decoding.
```

## Common pitfalls

- AIME'24 and AIME'25 require averaging results over 32 sampling runs, not greedy decoding.
- Answer correctness is evaluated via GPT-4o equivalence checking, not exact string or regex matching.
- Evaluation prompts must strictly follow the training-consistent templates from Ma et al. (2025) to ensure fair comparison with the training setup.

## Evidence (verbatim from paper)

> Figure 3: Reasoner pass rates when evaluating SPICE checkpoints at steps 200-640 against a fixed step-200 checkpoint on 128 documents. (a) Fixed Reasoner: Pass rate decreases from 55% to 35% as later Challenger checkpoints generate harder questions. (b) Fixed Challenger: Pass rate increases from 55% to 85% as later Reasoner checkpoints improve at solving questions.

## Citation

```bibtex
@misc{liu2025spice,
  title={SPICE: Self-Play In Corpus Environments Improves Reasoning},
  author={Liu et al. (2025)},
  year={2025},
  note={arXiv:2510.24684}
}
```

- arXiv: 2510.24684