Starcoder2 Code Eval

---
name: starcoder2-code-eval
description: Evaluates code generation, completion, and bug-fixing capabilities across multiple programming languages and libraries. It probes a model's ability to write correct functions from prompts, translate code across languages, and fix existing buggy code using standard and enhanced benchmarks. Use when the user wants to benchmark on HumanEval, MBPP, EvalPlus, MultiPL-E, DS-1000, HumanEvalFix, or asks about evaluating this task. Reports pass@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.19173
  bibtex_key: lozhkov2024starcoder2
  confidence: high
---

# starcoder2-code-eval

> StarCoder 2 and The Stack v2: The Next Generation — Lozhkov et al. (2024) (arXiv:2402.19173, 2024)

## What this evaluates

Evaluates code generation, completion, and bug-fixing capabilities across multiple programming languages and libraries. It probes a model's ability to write correct functions from prompts, translate code across languages, and fix existing buggy code using standard and enhanced benchmarks.

## Datasets

- **HumanEval** — total 164; splits: test (164)
- **MBPP** — total 500; splits: test (500)
- **EvalPlus** — total ?; splits: test (-1); repo https://github.com/evalplus/evalplus
- **MultiPL-E** — total ?; splits: test (-1)
- **DS-1000** — total 1000; splits: test (1000)
- **HumanEvalFix** — total ?; splits: test (-1)

## Metrics

- `pass@1` **(primary)** — range: [0, 1]
  - The probability that at least one of the k=1 generated samples passes all hidden unit tests for a given problem. Reported as mean pass@1 across all problems.

## Input / output format

**Input**: Function signature and docstring (HumanEval/MBPP), natural language description followed by tests (MBPP), or buggy code snippet (HumanEvalFix).

**Output**: Completed function code or fixed code snippet.

## Scoring recipe

```python
def pass_at_1(generated_code, tests):
    for test in tests:
        if not test.run(generated_code):
            return 0.0
    return 1.0

mean_pass_at_1 = sum(pass_at_1(gen, tests) for gen, tests in dataset) / len(dataset)
```

## Common pitfalls

- Using original HumanEval/MBPP instead of EvalPlus variants yields inflated scores due to inadequate hidden tests that miss subtle bugs.
- Base models score poorly on HumanEvalFix unless prompted with specific formats (e.g., 'Issue' vs 'Instruct'); instruction-tuned models are compared separately.
- MultiPL-E and DS-1000 results are averaged over 50 and 40 samples respectively at temperature 0.2 and top-p 0.95, not greedy decoding, which differs from HumanEval/MBPP evaluation settings.

## Evidence (verbatim from paper)

> Following recent work on Code LLMs (Rozière et al., 2023; Guo et al., 2024), we use greedy decoding and report the mean pass@1 (mean success rate) for all problems in the benchmark. ... For rigorous evaluation, we adopt the EvalPlus framework in this study.

## Citation

```bibtex
@misc{lozhkov2024starcoder2,
  title={StarCoder 2 and The Stack v2: The Next Generation},
  author={Lozhkov et al. (2024)},
  year={2024},
  note={arXiv:2402.19173}
}
```

- arXiv: 2402.19173