Repocoder Eval

---
name: repocoder-eval
description: This benchmark evaluates repository-level code completion by measuring how accurately a model predicts missing code segments given surrounding context and retrieved repository snippets. It probes both syntactic similarity and functional correctness across line, API, and function-level granularity. Use when the user wants to benchmark on RepoEval, or asks about evaluating this task. Reports Exact Match (EM).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2303.12570
  bibtex_key: zhang2023repocoder
  confidence: high
---

# repocoder-eval

> RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation — Fengji Zhang et al. (2023) (arXiv:2303.12570, 2023)

## What this evaluates

This benchmark evaluates repository-level code completion by measuring how accurately a model predicts missing code segments given surrounding context and retrieved repository snippets. It probes both syntactic similarity and functional correctness across line, API, and function-level granularity.

## Datasets

- **RepoEval** — total ?; splits: test (-1)

## Metrics

- `Exact Match (EM)` **(primary)** — range: [0, 1]
  - Binary metric; returns 1 if the predicted code exactly matches the ground truth, and 0 otherwise.
- `Edit Similarity (ES)` — range: [0, 1]
  - Fine-grained similarity calculated as 1 - (Levenshtein distance between prediction and ground truth) / max(len(prediction), len(ground truth)).
- `Pass Rate (PR)` — range: [0, 1]
  - Functional correctness metric; returns 1 if the generated code passes all corresponding unit tests, and 0 otherwise.

## Input / output format

**Input**: Unfinished code context (prompt) and optionally retrieved code snippets from the repository.

**Output**: Predicted code completion text string (Ŷ).

## Scoring recipe

```python
def compute_metrics(predictions, golds, unit_tests=None):
    em_scores = [1.0 if p == g else 0.0 for p, g in zip(predictions, golds)]
    es_scores = []
    for p, g in zip(predictions, golds):
        lev = levenshtein_distance(p, g)
        es_scores.append(1.0 - lev / max(len(p), len(g)))
    if unit_tests:
        pr_scores = [1.0 if run_tests(p, unit_tests) else 0.0 for p in predictions]
    else:
        pr_scores = None
    return {'EM': sum(em_scores)/len(em_scores), 'ES': sum(es_scores)/len(es_scores), 'PR': pr_scores}
```

## Common pitfalls

- Relying solely on EM/ES for function completion ignores functional correctness; unit test execution is required for accurate assessment.
- Hyperparameters like sliding window size (Sw) and retrieval count (K) are highly task-dependent and require adjustment for different programming languages or contexts.
- The Oracle baseline uses ground-truth code for retrieval, which provides an unrealistic upper bound that cannot be replicated in real-world deployment.

## Evidence (verbatim from paper)

> Similarity-based Evaluation: Following established practices in code completion research (Lu et al., 2021, 2022), we evaluate our line and API completion datasets using two metrics: Exact Match (EM) and Edit Similarity (ES). The EM score is a binary metric that takes the value of 1 if the predicted code exactly matches the ground truth code, and 0 otherwise. The ES metric provides a more fine-grained evaluation and is calculated as $ES = 1 - \frac{\operatorname{Lev}(\hat{Y}, Y)}{\max(|\hat{Y}|, |Y|)}$, where Lev represents the Levenshtein distance (Levenshtein et al., 1966).

## Citation

```bibtex
@misc{zhang2023repocoder,
  title={RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation},
  author={Fengji Zhang et al. (2023)},
  year={2023},
  note={arXiv:2303.12570}
}
```

- arXiv: 2303.12570