Bigcodebench Eval

---
name: bigcodebench-eval
description: Evaluates large language models' ability to generate correct, executable code for complex programming tasks requiring diverse function calls and compositional reasoning. It also probes instruction-following capabilities by comparing performance on verbose prompts versus condensed natural-language instructions. Use when the user wants to benchmark on BigCodeBench, or asks about evaluating this task. Reports Pass@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2406.15877
  bibtex_key: zhuo2024bigcodebench
  confidence: high
---

# bigcodebench-eval

> BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions — Terry Yue Zhuo et al. (arXiv:2406.15877, 2024)

## What this evaluates

Evaluates large language models' ability to generate correct, executable code for complex programming tasks requiring diverse function calls and compositional reasoning. It also probes instruction-following capabilities by comparing performance on verbose prompts versus condensed natural-language instructions.

## Datasets

- **BigCodeBench** — total 1140; splits: test (1140); repo https://github.com/bigcode-project/bigcodebench

## Metrics

- `Pass@1` **(primary)** — range: [0, 1]
  - Unbiased Pass@K (Chen et al., 2021) estimates the probability that at least one of K generated samples passes all test cases. Calibrated Pass@1 injects missing imports/constants before execution to account for model laziness.

## Input / output format

**Input**: Complex programming task descriptions requiring specific function calls from various libraries, provided as zero-shot prompts. For the Instruct variant, prompts are condensed natural-language instructions.

**Output**: Python code snippets intended to solve the given task.

## Scoring recipe

```python
def calculate_pass_at_1(generated_code, test_cases):
    calibrated_code = add_missing_setup(generated_code)
    return all(run_test(calibrated_code, tc) for tc in test_cases)
# Aggregate pass rates across all tasks to compute mean Pass@1
```

## Common pitfalls

- Model laziness causes omission of essential imports/constants, requiring calibration to avoid false negatives.
- Computational limits often force N=5 samples, violating the N>=K requirement for unbiased Pass@K estimation.
- Condensed instructions in the Instruct variant introduce ambiguity, lowering scores independent of coding ability.

## Evidence (verbatim from paper)

> Our evaluation uses the unbiased version of Pass@K (Chen et al., 2021) to accurately assess the functional correctness of generated code snippets by LLMs. ... we calibrate the generation quality by adding the missing setup and calculate Pass@1, which is denoted as calibrated Pass@1.

## Citation

```bibtex
@misc{zhuo2024bigcodebench,
  title={BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions},
  author={Terry Yue Zhuo et al.},
  year={2024},
  note={arXiv:2406.15877}
}
```

- arXiv: 2406.15877