Humaneval Eval

---
name: humaneval-eval
description: Evaluates a model's ability to generate correct, executable Python code from natural language function descriptions and signatures. It measures functional correctness by checking if generated code passes hidden unit tests. Use when the user wants to benchmark on HumanEval, or asks about evaluating this task. Reports pass@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.06161
  bibtex_key: li2023starcoder
  confidence: high
---

# humaneval-eval

> StarCoder: may the source be with you! — Raymond Li et al. (2023) (arXiv:2305.06161, 2023)

## What this evaluates

Evaluates a model's ability to generate correct, executable Python code from natural language function descriptions and signatures. It measures functional correctness by checking if generated code passes hidden unit tests.

## Datasets

- **HumanEval** — total 164; splits: test (164); HF `openai/humaneval`

## Metrics

- `pass@1` **(primary)** — range: [0, 1]
  - The fraction of benchmark problems for which at least one of k generated samples passes all provided test cases. The paper focuses on pass@1 (k=1), using sampling temperature 0.2 and generating n=200 samples per problem to estimate the metric reliably.

## Input / output format

**Input**: Python function signature, docstring containing a natural language description, and a set of hidden test cases/assertions.

**Output**: Generated Python code (typically the function body or full function) intended to satisfy the docstring and pass all test cases.

## Scoring recipe

```python
def compute_pass_at_1(generated_samples, test_cases):
    passed = 0
    for problem in generated_samples:
        solved = False
        for sample in problem:
            if run_and_check_tests(sample, test_cases):
                solved = True
                break
        if solved:
            passed += 1
    return passed / len(generated_samples)
```

## Common pitfalls

- Using the wrong sampling temperature (0.2 for pass@1 vs 0.8 for pass@k>1) leads to non-comparable scores.
- Failing to execute generated code in a sandboxed environment or using insufficient test cases can inflate scores.
- Not generating enough samples (n=200 recommended) to accurately estimate pass@1, especially for harder problems.

## Evidence (verbatim from paper)

> We report performance using the pass@k metric (Chen et al., 2021): the total fraction of benchmark problems solved, where a problem is considered solved if any one of k code samples passes every test case. Like Chen et al. (2021), we use sampling temperature 0.2 for pass@1, and temperature 0.8 for k > 1. We generate n = 200 samples for all experiments with open-access models.

## Citation

```bibtex
@misc{li2023starcoder,
  title={StarCoder: may the source be with you!},
  author={Raymond Li et al. (2023)},
  year={2023},
  note={arXiv:2305.06161}
}
```

- arXiv: 2305.06161