R1 Code Interpreter Eval

---
name: r1-code-interpreter-eval
description: Probes LLMs' ability to autonomously generate and execute code for complex reasoning and planning tasks across logic, spatial, order, optimization, search, and math domains. It evaluates how well models can iteratively explore, optimize, and self-check solutions in multi-turn code execution environments. Use when the user wants to benchmark on SymBench, Big-Bench-Hard, Reasoning-Gym, or asks about evaluating this task. Reports exact match or constraint check.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.21668
  bibtex_key: chen2025r1codeinterpreter
  confidence: high
---

# r1-code-interpreter-eval

> R1-Code-Interpreter: LLMs Reason with Code via Supervised and Multi-stage Reinforcement Learning — Chen et al. (2025) (arXiv:2505.21668, 2025)

## What this evaluates

Probes LLMs' ability to autonomously generate and execute code for complex reasoning and planning tasks across logic, spatial, order, optimization, search, and math domains. It evaluates how well models can iteratively explore, optimize, and self-check solutions in multi-turn code execution environments.

## Datasets

- **SymBench** — total ?; splits: test (-1)
- **Big-Bench-Hard** — total ?; splits: test (-1)
- **Reasoning-Gym** — total ?; splits: test (-1); repo https://github.com/open-thought/reasoning-gym

## Metrics

- `exact match or constraint check` **(primary)** — range: [0, 1]
  - Rule-based evaluation where the model's final output or executed code result is compared against ground truth using exact match or predefined constraint checks.

## Input / output format

**Input**: Standardized task prompts covering diverse reasoning and planning challenges (logic, spatial, order, optimization, search, math).

**Output**: Autonomously generated code and/or final answer, subject to rule-based exact match or constraint verification.

## Scoring recipe

```python
def score(predictions, gold):
    if predictions == gold:
        return 1.0
    if check_constraints(predictions):
        return 1.0
    return 0.0
```

## Common pitfalls

- The optimal reasoning mode (code vs. text) is highly task-dependent and hard to predict a priori.
- LLM-generated code frequently degenerates into hard-coded, text-like scripts that lack symbolic execution utility.
- Many tasks require iterative multi-turn exploration and optimization, which standard single-turn generation fails to capture.

## Evidence (verbatim from paper)

> All tasks are standardized into a unified format and evaluated using rule-based criteria (e.g., exact match or constraint checks) for efficient rollout and testing.

## Citation

```bibtex
@misc{chen2025r1codeinterpreter,
  title={R1-Code-Interpreter: LLMs Reason with Code via Supervised and Multi-stage Reinforcement Learning},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2505.21668}
}
```

- arXiv: 2505.21668