Humanevalfix Eval

---
name: humanevalfix-eval
description: Evaluates a model's ability to debug and fix buggy code by generating corrected implementations that pass provided unit tests. It probes code repair capabilities across multiple programming languages. Use when the user wants to benchmark on HumanEvalFix, or asks about evaluating this task. Reports pass rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2308.07124
  bibtex_key: muennighoff2023octopack
  confidence: high
---

# humanevalfix-eval

> OctoPack: Instruction Tuning Code Large Language Models — Muennighoff et al. (2023) (arXiv:2308.07124, 2023)

## What this evaluates

Evaluates a model's ability to debug and fix buggy code by generating corrected implementations that pass provided unit tests. It probes code repair capabilities across multiple programming languages.

## Datasets

- **HumanEvalFix** — total ?; splits: test (-1)

## Metrics

- `pass rate` **(primary)** — range: percent
  - Percentage of generated code fixes that pass all provided unit tests for a given problem. Calculated as (number of passing fixes / total problems) * 100.

## Input / output format

**Input**: A buggy function signature and body, accompanied by its unit tests. Optionally, docstrings may be provided as ground truth hints.

**Output**: The corrected function implementation (code only).

## Scoring recipe

```python
def score(predictions, gold_tests):
    passed = 0
    for pred_code, tests in zip(predictions, gold_tests):
        if run_tests(pred_code, tests): # All unit tests pass
            passed += 1
    return (passed / len(predictions)) * 100
```

## Common pitfalls

- Models may ignore the buggy function and generate entirely new code from scratch if docstrings are provided, bypassing the actual repair task.
- Prompting format significantly impacts performance; commit-style prompts may outperform standard instruction prompts depending on pretraining data.

## Evidence (verbatim from paper)

> The default version of HUMANEVALFIX does not include docstrings, but only provides the unit tests to the model alongside the buggy function. Solving from docstrings is generally easier for models than from tests, as models can also solve it via pure code synthesis without looking at the buggy function at all.

## Citation

```bibtex
@misc{muennighoff2023octopack,
  title={OctoPack: Instruction Tuning Code Large Language Models},
  author={Muennighoff et al. (2023)},
  year={2023},
  note={arXiv:2308.07124}
}
```

- arXiv: 2308.07124