Cruxeval Eval

---
name: cruxeval-eval
description: Evaluates a model's ability to reason about and execute short Python functions by predicting outputs given inputs (CRUXEval-I) and predicting inputs given outputs (CRUXEval-O). It probes fundamental code execution and understanding capabilities beyond simple code generation. Use when the user wants to benchmark on CRUXEval, or asks about evaluating this task. Reports pass@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2401.03065
  bibtex_key: gu2024cruxeval
  confidence: high
---

# cruxeval-eval

> CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution — Gu et al. (2024) (arXiv:2401.03065, 2024)

## What this evaluates

Evaluates a model's ability to reason about and execute short Python functions by predicting outputs given inputs (CRUXEval-I) and predicting inputs given outputs (CRUXEval-O). It probes fundamental code execution and understanding capabilities beyond simple code generation.

## Datasets

- **CRUXEval** — total 800; splits: test (-1)

## Metrics

- `pass@1` **(primary)** — range: [0, 1]
  - The fraction of problems where at least one of the N generated samples matches the gold output. Evaluated with generation temperature T=0.2.
- `pass@5` — range: [0, 1]
  - The fraction of problems where at least one of the N generated samples matches the gold output. Evaluated with generation temperature T=0.8.

## Input / output format

**Input**: A short Python function (3–13 lines) with a signature and docstring, plus either input arguments (for input-output prediction) or expected output (for output-input prediction).

**Output**: The predicted output value(s) or input arguments that satisfy the function's specification.

## Scoring recipe

```python
def compute_pass_at_k(gold_outputs, generated_samples, k):
    correct = sum(1 for s in generated_samples if s in gold_outputs)
    return correct / len(gold_outputs)

# Per problem:
# N = 100 for non-GPT models, N = 10 for GPT models
# pass@1 uses T=0.2, pass@5 uses T=0.8
# Final score = mean(compute_pass_at_k(gold, samples, k) over all problems)
```

## Common pitfalls

- Temperature settings differ between pass@1 (T=0.2) and pass@5 (T=0.8); using a single temperature for both will invalidate the reported scores.
- Statistical significance is assessed via paired bootstrap over 10,000 samples rather than standard t-tests; replicating confidence intervals requires resampling the exact dataset splits.
- The benchmark uses a generate-and-filter pipeline; models must be evaluated on the original 800 functions, not on filtered or modified subsets.

## Evidence (verbatim from paper)

> We report both pass@1 scores $(T = 0.2)$ and pass@5 scores $(T = 0.8)$. The results are shown in Fig. 2, and raw scores are provided in the Appendix in Table 2.

## Citation

```bibtex
@misc{gu2024cruxeval,
  title={CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution},
  author={Gu et al. (2024)},
  year={2024},
  note={arXiv:2401.03065}
}
```

- arXiv: 2401.03065