Competitive Coding Eval

---
name: competitive-coding-eval
description: Evaluates the ability of large language models to generate correct, executable Python solutions for competitive programming problems. It probes algorithmic reasoning, code synthesis, and adherence to problem constraints under strict time and complexity limits. Use when the user wants to benchmark on LiveCodeBench, CodeContests, or asks about evaluating this task. Reports pass@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.01943
  bibtex_key: ahmad2025opencodereasoning
  confidence: high
---

# competitive-coding-eval

> OpenCodeReasoning: Advancing Data Distillation for Competitive Coding — Wasi Uddin Ahmad et al. (2025) (arXiv:2504.01943, 2025)

## What this evaluates

Evaluates the ability of large language models to generate correct, executable Python solutions for competitive programming problems. It probes algorithmic reasoning, code synthesis, and adherence to problem constraints under strict time and complexity limits.

## Datasets

- **LiveCodeBench** — total 279; splits: test (279)
- **CodeContests** — total ?; splits: Public (-1), Private (-1), Generated (-1), All (-1)

## Metrics

- `pass@1` **(primary)** — range: percent
  - Average pass@1 calculated by averaging 64 inference runs for LiveCodeBench and 16 runs for CodeContests. pass@1 measures the probability that at least one of the generated solutions passes all hidden test cases.

## Input / output format

**Input**: Natural language problem description for competitive programming tasks, including constraints and examples.

**Output**: Python code solution (optionally with reasoning traces), generated via temperature-based nucleus sampling with a maximum length of 30,720 tokens.

## Scoring recipe

```python
def compute_pass_at_1(generated_solutions, test_cases):
    for code in generated_solutions:
        if execute_and_check(code, test_cases):
            return 1.0
    return 0.0

# For each problem:
#   solutions = sample_n_times(model, prompt, n=64 for LCB, n=16 for CC)
#   score = compute_pass_at_1(solutions, problem.test_cases)
# Final metric = mean(score across all problems) * 100
```

## Common pitfalls

- Averaging over multiple inference runs (64 for LiveCodeBench, 16 for CodeContests) is mandatory to reduce variance, whereas baselines were evaluated with a single run.
- Execution filtering during data curation was found to harm benchmark accuracy, so models should be evaluated on raw generated code without post-hoc execution filtering.
- The metric is reported as a percentage in the tables, but pass@1 is fundamentally a probability in [0,1].

## Evidence (verbatim from paper)

> To mitigate performance variance inherent in single-run evaluations, we report the average pass@1 metric, calculated by averaging 64 inference runs for LiveCodeBench and 16 runs for CodeContests.

## Citation

```bibtex
@misc{ahmad2025opencodereasoning,
  title={OpenCodeReasoning: Advancing Data Distillation for Competitive Coding},
  author={Wasi Uddin Ahmad et al. (2025)},
  year={2025},
  note={arXiv:2504.01943}
}
```

- arXiv: 2504.01943