Leetcode Dataset Eval

---
name: leetcode-dataset-eval
description: Evaluates code generation and algorithmic reasoning capabilities on competitive programming problems. It specifically probes temporal robustness by testing on problems released after a strict cutoff date to detect data contamination, and measures performance across difficulty levels and algorithmic topics. Use when the user wants to benchmark on LeetCodeDataset, or asks about evaluating this task. Reports pass@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.14655
  bibtex_key: xia2025leetcode
  confidence: high
---

# leetcode-dataset-eval

> LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs — Xia et al. (2025) (arXiv:2504.14655, 2025)

## What this evaluates

Evaluates code generation and algorithmic reasoning capabilities on competitive programming problems. It specifically probes temporal robustness by testing on problems released after a strict cutoff date to detect data contamination, and measures performance across difficulty levels and algorithmic topics.

## Datasets

- **LeetCodeDataset** — total 256; splits: test (256); repo https://github.com/newfacade/LeetCodeDataset

## Metrics

- `pass@1` **(primary)** — range: percent
  - The proportion of problems where the model's single generated solution passes all hidden test cases. Calculated as (number of correctly solved problems) / (total number of problems).

## Input / output format

**Input**: Programming problem description including statement, constraints, and example I/O.

**Output**: A single Python code solution for the given problem.

## Scoring recipe

```python
def compute_pass_at_1(predictions, gold_tests):
    passed = 0
    for pred, tests in zip(predictions, gold_tests):
        if run_tests(pred, tests):  # Executes code against all test cases
            passed += 1
    return (passed / len(predictions)) * 100
```

## Common pitfalls

- Using generation parameters other than temperature=0.2 and top_p=0.95 yields non-comparable pass rates.
- Assuming temporal split guarantees zero contamination without verifying exact problem release dates against model training cutoffs.
- Aggregating pass rates across difficulty levels or topics without weighting by problem count misrepresents overall performance.

## Evidence (verbatim from paper)

> We evaluate six models on the LeetCodeDataset test set, comprising 256 programming problems that were newly released after July 1, 2024. The evaluation highlights DeepSeek-R1 (pass@1 rate = 65.23%) and QwQ-Plus (pass@1 rate = 56.25%) as top performers, demonstrating the substantial advantage of long-CoT reasoning models in solving complex competition-level coding problems. Following LiveCodeBench’s temporal evaluation methodology, we analyze monthly accuracy change relative to problem release months as shown in [Figure 3], and summarize model pass rates across difficulty levels in [Table 2].

## Citation

```bibtex
@misc{xia2025leetcode,
  title={LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs},
  author={Xia et al. (2025)},
  year={2025},
  note={arXiv:2504.14655}
}
```

- arXiv: 2504.14655