Opencodeinstruct Eval

---
name: opencodeinstruct-eval
description: Evaluates the code generation, algorithmic problem-solving, and complex function-calling capabilities of instruction-tuned LLMs across multiple standardized coding benchmarks. Use when the user wants to benchmark on HumanEval, MBPP, LiveCodeBench, BigIntCodeBench-Instruct, or asks about evaluating this task. Reports pass@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.04030
  bibtex_key: ahmad2025opencodeinstruct
  confidence: high
---

# opencodeinstruct-eval

> OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs — Ahmad et al. (2025) (arXiv:2504.04030, 2025)

## What this evaluates

Evaluates the code generation, algorithmic problem-solving, and complex function-calling capabilities of instruction-tuned LLMs across multiple standardized coding benchmarks.

## Datasets

- **HumanEval** — total ?; splits: test (-1)
- **MBPP** — total ?; splits: test (-1)
- **LiveCodeBench** — total 713; splits: test (713)
- **BigIntCodeBench-Instruct** — total 1140; splits: test (1140)

## Metrics

- `pass@1` **(primary)** — range: percent
  - Percentage of generated solutions that pass all provided unit tests for a given problem. Computed as the fraction of problems where at least one generated sample passes all test cases.

## Input / output format

**Input**: Natural language problem description, often including a function signature or stub, and sometimes library constraints or competitive programming context.

**Output**: Python code implementing the requested function or script.

## Scoring recipe

```python
def compute_pass_at_1(predictions, gold_tests):
    passed_count = 0
    for code, tests in zip(predictions, gold_tests):
        if execute_and_check(code, tests):
            passed_count += 1
    return (passed_count / len(predictions)) * 100
```

## Common pitfalls

- Continuously updated benchmarks like LiveCodeBench require strict contamination checks to ensure test cases were not in the training data.
- pass@1 scores can be inflated by weak or incomplete unit tests; some benchmarks use minimal test coverage.

## Evidence (verbatim from paper)

> HumanEval and MBPP We reported the evaluations on HumanEval (Chen et al., 2021), MBPP (Austin et al., 2021), HumaneEval+ (Liu et al., 2023), and MBPP+ (Liu et al., 2023) which are the most common benchmarks for function-level code generation. LiveCodeBench (Jain et al., 2025) is an extensive, contamination-free benchmark created to assess the coding capabilities of LLMs. BigIntCodeBench-Instruct, a natural language instruction adaptation of BigCodeBench (Zhuo et al., 2025), challenges LLMs with complex function calling tasks.

## Citation

```bibtex
@misc{ahmad2025opencodeinstruct,
  title={OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs},
  author={Ahmad et al. (2025)},
  year={2025},
  note={arXiv:2504.04030}
}
```

- arXiv: 2504.04030