Mathcoder Eval

---
name: mathcoder-eval
description: Evaluates large language models' ability to solve mathematical word problems across varying difficulty levels and subjects, including elementary, high school, and collegiate mathematics. It specifically probes the model's capacity for code-interleaved reasoning and execution-aware autoregression. Use when the user wants to benchmark on GSM8K, MATH, SVAMP, Mathematics, SimulEq, or asks about evaluating this task. Reports accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.03731
  bibtex_key: wang2023mathcoder
  confidence: high
---

# mathcoder-eval

> MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning — Wang et al. (2023) (arXiv:2310.03731, 2023)

## What this evaluates

Evaluates large language models' ability to solve mathematical word problems across varying difficulty levels and subjects, including elementary, high school, and collegiate mathematics. It specifically probes the model's capacity for code-interleaved reasoning and execution-aware autoregression.

## Datasets

- **GSM8K** — total ?; splits: test (-1)
- **MATH** — total ?; splits: test (-1)
- **SVAMP** — total ?; splits: test (-1)
- **Mathematics** — total ?; splits: test (-1)
- **SimulEq** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: percent
  - Exact-match accuracy: the percentage of problems where the model's predicted final answer exactly matches the ground truth answer.

## Input / output format

**Input**: Natural language mathematical word problems.

**Output**: A final numerical or symbolic answer, optionally preceded by reasoning steps and code execution blocks.

## Scoring recipe

```python
def compute_accuracy(predictions, gold_answers):
    correct = 0
    for pred, gold in zip(predictions, gold_answers):
        if normalize_answer(pred) == normalize_answer(gold):
            correct += 1
    return (correct / len(predictions)) * 100
```

## Common pitfalls

- Baselines are evaluated with CoT or few-shot prompting to maximize performance, while MathCoder is strictly zero-shot; comparing them directly without noting the prompt difference is misleading.
- The paper distinguishes between models that predict execution results internally versus those that use an external Python interpreter during inference; results vary significantly between these two settings.

## Evidence (verbatim from paper)

> Tab. 5 (#3) improves the accuracy across four out of five datasets, resulting in a rise in the average accuracy from 69.1% to 70.2%, which aligns with the hypothesis that by computing the loss only on natural language and code, the model can focus more on the math problem-solving skills itself, thus making the supervised fine-tuning more effective.

## Citation

```bibtex
@misc{wang2023mathcoder,
  title={MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoning},
  author={Wang et al. (2023)},
  year={2023},
  note={arXiv:2310.03731}
}
```

- arXiv: 2310.03731