Execrepo Bench Eval

---
name: execrepo-bench-eval
description: Evaluates repository-level code completion capabilities of LLMs across multiple granularities (span, line, expression, statement, function) using executable validation and string similarity metrics. Use when the user wants to benchmark on ExecRepoBench, or asks about evaluating this task. Reports Pass@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2412.11990
  bibtex_key: yang2024execrepo
  confidence: high
---

# execrepo-bench-eval

> ExecRepoBench: Multi-level Executable Code Completion Evaluation — Yang et al. (2024) (arXiv:2412.11990, 2024)

## What this evaluates

Evaluates repository-level code completion capabilities of LLMs across multiple granularities (span, line, expression, statement, function) using executable validation and string similarity metrics.

## Datasets

- **ExecRepoBench** — total 1200; splits: test (1200)

## Metrics

- `Pass@1` **(primary)** — range: percent
  - Percentage of generated code completions that pass all associated unit tests when evaluated with greedy decoding.
- `Edit Similarity (ES)` — range: percent
  - String-based similarity score calculated by comparing the generated code against the ground-truth code using edit distance.

## Input / output format

**Input**: Repository-level code snippets with masked sections at various granularities (span, single-line, multi-line, expression, statement, function) requiring the model to complete the code.

**Output**: Completed code snippet to replace the masked section.

## Scoring recipe

```python
# For each sample in the benchmark:
generated_code = model.predict(context)
ground_truth_code = sample.gold
unit_tests = sample.tests

# Edit Similarity
es_score = 1 - edit_distance(generated_code, ground_truth_code) / max(len(generated_code), len(ground_truth_code))

# Pass@1 (Greedy)
test_results = run_unit_tests(generated_code, unit_tests)
pass_at_1 = 1.0 if all(test_results) else 0.0

# Aggregate
avg_es = mean(es_scores)
avg_pass_at_1 = mean(pass_at_1_scores)
```

## Common pitfalls

- High ES scores do not guarantee functional correctness, as semantic equivalence may exist without exact string matching.
- Greedy decoding for Pass@1 may underestimate model capability compared to sampling-based Pass@k evaluations.
- Repository-level completion requires handling cross-file dependencies, which standard in-file benchmarks ignore.

## Evidence (verbatim from paper)

> We compare the generated code and the ground-truth code using edit similarity (ES) to report string-based scores. Similar to the in-file benchmark HumanEval/MBPP, we employ the Pass@k metric based on the executable results to get the reliability evaluation results. In this work, we report the greedy Pass@1 score of all LLMs with greedy inference for a fair comparison.

## Citation

```bibtex
@misc{yang2024execrepo,
  title={ExecRepoBench: Multi-level Executable Code Completion Evaluation},
  author={Yang et al. (2024)},
  year={2024},
  note={arXiv:2412.11990}
}
```

- arXiv: 2412.11990