Sbsc Math Olympiad Eval

---
name: sbsc-math-olympiad-eval
description: Evaluates LLMs' ability to solve complex, Olympiad-level mathematics problems across Algebra, Combinatorics, Number Theory, and Geometry. It specifically probes multi-turn code generation and execution feedback for iterative problem decomposition and constraint handling. Use when the user wants to benchmark on AIME, AMC-12, MathOdyssey, OlympiadBench, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.16666
  bibtex_key: singh2025sbsc
  confidence: high
---

# sbsc-math-olympiad-eval

> SBSC: Step-By-Step Coding for Improving Mathematical Olympiad Performance — Kunal Singh et al. (2025) (arXiv:2502.16666, 2025)

## What this evaluates

Evaluates LLMs' ability to solve complex, Olympiad-level mathematics problems across Algebra, Combinatorics, Number Theory, and Geometry. It specifically probes multi-turn code generation and execution feedback for iterative problem decomposition and constraint handling.

## Datasets

- **AIME** — total 330; splits: test (330)
- **AMC-12** — total 475; splits: test (475)
- **MathOdyssey** — total 158; splits: test (158)
- **OlympiadBench** — total 504; splits: test (504)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Standard exact-match accuracy: 1 if the model's final extracted integer answer matches the gold integer answer, 0 otherwise. Averaged across all test instances.

## Input / output format

**Input**: LaTeX-formatted math competition problems, modified to require integer answers. Evaluated in a 4-shot setting with carefully selected exemplars that all baseline methods solved correctly.

**Output**: Multi-turn code generation and execution traces, culminating in a final integer answer. For self-consistency decoding, 7 samples are generated and the majority vote is taken.

## Scoring recipe

```python
def compute_accuracy(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        pred_int = extract_integer_from_text(pred)
        if pred_int == gold:
            correct += 1
    return correct / len(golds)
```

## Common pitfalls

- AMC-12 questions are originally multiple-choice; the paper removes answer choices and forces integer answers, altering the standard evaluation format.
- MathOdyssey and OlympiadBench questions are post-processed via GPT-4o to append integer-answer prompts, which may not reflect the original benchmark's intended format.
- The 4-shot exemplars are specifically curated to be correctly solved by all methods, potentially inflating performance compared to random or zero-shot settings.

## Evidence (verbatim from paper)

> Note: we experimentally observe that for n > 4, there is insignificant increase in accuracy for TIR-ToRA so we set n=4 for TIR-ToRA during SC decoding.

## Citation

```bibtex
@misc{singh2025sbsc,
  title={SBSC: Step-By-Step Coding for Improving Mathematical Olympiad Performance},
  author={Kunal Singh et al. (2025)},
  year={2025},
  note={arXiv:2502.16666}
}
```

- arXiv: 2502.16666