Human Eval Functional Accuracy

---
name: human-eval-functional-accuracy
description: Evaluates a code generation model's ability to produce correct, executable Python functions from docstrings and function signatures. It measures whether the generated code passes all provided unit tests for each programming problem. Use when the user wants to benchmark on HumanEval, or asks about evaluating this task. Reports functional accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2202.12299
  bibtex_key: jones2022capturing
  confidence: high
---

# human-eval-functional-accuracy

> Capturing Failures of Large Language Models via Human Cognitive Biases — Jones et al. (2022) (arXiv:2202.12299, 2022)

## What this evaluates

Evaluates a code generation model's ability to produce correct, executable Python functions from docstrings and function signatures. It measures whether the generated code passes all provided unit tests for each programming problem.

## Datasets

- **HumanEval** — total 164; splits: test (164); repo https://github.com/openai/human-eval

## Metrics

- `functional accuracy` **(primary)** — range: [0, 1]
  - The fraction of programs that pass all of the provided test cases. Calculated as (number of problems where generated code passes all tests) / (total number of problems).

## Input / output format

**Input**: Function signature and docstring describing the desired functionality, optionally with prepended irrelevant preceding functions or anchor functions containing similar but incorrect code.

**Output**: Python code completion (function body) generated via greedy decoding.

## Scoring recipe

```python
def compute_functional_accuracy(generated_code, test_cases):
    passed = 0
    for code, tests in zip(generated_code, test_cases):
        try:
            exec(code)
            if all(test() for test in tests):
                passed += 1
        except Exception:
            pass
    return passed / len(generated_code)
```

## Common pitfalls

- Models may output the prepended framing or anchor line verbatim instead of solving the problem, artificially lowering accuracy without reflecting true reasoning failure.
- Prepending anchor functions can sometimes lead to correct solutions that incorporate the anchor's logic, meaning accuracy drops do not always indicate a complete failure to solve the task.
- Greedy decoding is used, which may not reflect the model's peak capability compared to sampling-based decoding.

## Evidence (verbatim from paper)

> Following Chen et al. (2021), we measure performance on HumanEval with functional accuracy: the fraction of programs that pass all of the test cases provided at the url: https://github.com/openai/human-eval.

## Citation

```bibtex
@misc{jones2022capturing,
  title={Capturing Failures of Large Language Models via Human Cognitive Biases},
  author={Jones et al. (2022)},
  year={2022},
  note={arXiv:2202.12299}
}
```

- arXiv: 2202.12299