Codearena Eval

---
name: codearena-eval
description: Evaluates code LLMs' alignment with human preferences on real-world, non-algorithmic coding tasks. It measures how well model-generated code matches human-like quality and preference compared to a baseline, rather than just syntactic or execution correctness. Use when the user wants to benchmark on CodeArena, EvalPlus, MultiPL-E, or asks about evaluating this task. Reports Pass@1, Win rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.05210
  bibtex_key: yang2024codearena
  confidence: high
---

# codearena-eval

> Evaluating and Aligning CodeLLMs on Human Preference — Jian Yang et al. (2024) (arXiv:2412.05210, 2024)

## What this evaluates

Evaluates code LLMs' alignment with human preferences on real-world, non-algorithmic coding tasks. It measures how well model-generated code matches human-like quality and preference compared to a baseline, rather than just syntactic or execution correctness.

## Datasets

- **CodeArena** — total ?; splits: test (-1)
- **EvalPlus** — total ?; splits: test (-1)
- **MultiPL-E** — total ?; splits: test (-1)

## Metrics

- `Pass@1` **(primary)** — range: percent
  - Greedy Pass@1 measures the fraction of generated solutions that pass all provided test cases. The expected function is extracted from the model output and executed against the test suite to verify correctness.
- `Win rate` **(primary)** — range: percent
  - Pairwise preference score calculated by an LLM judge (GPT-4o) comparing a model response against a baseline. Each question is scored twice with swapped input orderings (A,B and B,A) to mitigate position bias, and the win rate is averaged across these runs.

## Input / output format

**Input**: For code execution benchmarks (EvalPlus, MultiPL-E): a coding problem description and associated test cases. For preference benchmark (CodeArena): a coding question and two model-generated code responses (baseline and target model).

**Output**: For code execution benchmarks: a generated code snippet containing the expected function. For preference benchmark: a judgment indicating which response is better or a tie.

## Scoring recipe

```python
def score_pass_at_1(predictions, test_cases):
    correct = 0
    for pred in predictions:
        func = extract_function(pred)
        if run_tests(func, test_cases):
            correct += 1
    return correct / len(predictions)

def score_win_rate(predictions, baseline, question):
    j1 = llm_judge(question, baseline, predictions)
    j2 = llm_judge(question, predictions, baseline)
    wins = sum(1 for j in [j1, j2] if j == 'model_wins')
    return wins / 2.0
```

## Common pitfalls

- CodeArena tasks are non-algorithmic and cannot be evaluated via automated code execution; they strictly require pairwise LLM judging.
- The LLM judge's win rate must be averaged over two different input orderings (A,B and B,A) to account for position bias, not just a single comparison.
- Pass@1 is computed using greedy decoding, not sampling, which may underestimate performance if diverse solutions are needed.

## Evidence (verbatim from paper)

> Different from the EvalPlus and MultiPL-E, CodeArena consists of many non-algorihtmic, which is not suitable for code-execution-based evaluation. Each question is scored twice to calculate the win rate and tie rate by GPT-4o using a different input order “A, B” and “B, A”, where “A” is the baseline from gpt-4-turbo-2024-04-09 and “B” is the model-generated response. Given the model-generated response, we extract the expected function and feed the test cases into the extracted function to verify the correctness of the generation. We adopt greedy Pass@1 to report the results on EvalPlus and MultiPL-E.

## Citation

```bibtex
@misc{yang2024codearena,
  title={Evaluating and Aligning CodeLLMs on Human Preference},
  author={Jian Yang et al. (2024)},
  year={2024},
  note={arXiv:2412.05210}
}
```

- arXiv: 2412.05210