Humaneval X Eval

---
name: humaneval-x-eval
description: Evaluates multilingual code generation and code translation capabilities across five programming languages (C++, Java, JavaScript, Go, Python). It measures functional correctness by executing generated code against a suite of test cases for each problem. Use when the user wants to benchmark on HumanEval-X, or asks about evaluating this task. Reports pass@k.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2303.17568
  bibtex_key: zheng2023codegeex
  confidence: high
---

# humaneval-x-eval

> CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X — Zheng et al. (2023) (arXiv:2303.17568, 2023)

## What this evaluates

Evaluates multilingual code generation and code translation capabilities across five programming languages (C++, Java, JavaScript, Go, Python). It measures functional correctness by executing generated code against a suite of test cases for each problem.

## Datasets

- **HumanEval-X** — total 820; splits: test (820)

## Metrics

- `pass@k` **(primary)** — range: [0, 1]
  - Estimates the probability that at least one of k samples passes all test cases out of n total generations: pass@k = E[1 - C(n-c, k) / C(n, k)], where c is the number of samples passing all tests, n=200, and k∈{1, 10, 100}. Averaged over all problems.

## Input / output format

**Input**: For code generation: function declaration and docstring. For code translation: function declaration in the target language and canonical solution in the source language.

**Output**: Generated function implementation in the target language.

## Scoring recipe

```python
def compute_pass_at_k(generations, test_cases, k=10, n=200):
    passed = [1 if run_tests(gen, test_cases) else 0 for gen in generations]
    c = sum(passed)
    if n < k or c == n: return 1.0
    return 1.0 - comb(n - c, k) / comb(n, k)

# Average single-problem pass@k over all problems in the dataset
```

## Common pitfalls

- Using string similarity metrics like BLEU or CodeBLEU instead of functional correctness via test execution.
- Ignoring language-specific behaviors in test cases (e.g., rounding rules, string prefixes) which can cause false negatives if not handled per language.
- Failing to account for budget allocation when evaluating multilingual models across multiple languages simultaneously.

## Evidence (verbatim from paper)

> Metric. For both tasks, we use test cases to evaluate the exact functional correctness of the generated code, measuring the performance with pass@$k$ (Kulal et al., 2019), making it real-world useful and also completely different from the string similarity metrics like BLEU(Papineni et al., 2002), and CodeBLEU(Ren et al., 2020; Lu et al., 2021; Zhu et al., 2022). Specifically, we use the unbiased method to estimate pass@$k$ (Chen et al., 2021): pass@k:=E[1-(n-c k)/(n k)],n=200,k∈{1,10,100} where n is the total number of generation (n=200 in this work), k is the sampling budget (typically k∈{1, 10, 100}) and c is the number of samples that pass all test cases.

## Citation

```bibtex
@misc{zheng2023codegeex,
  title={CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X},
  author={Zheng et al. (2023)},
  year={2023},
  note={arXiv:2303.17568}
}
```

- arXiv: 2303.17568