Repobench P Eval

---
name: repobench-p-eval
description: Evaluates an end-to-end pipeline that first retrieves cross-file snippets and then predicts the next line of code using both the in-file context and retrieved snippets. Use when the user wants to benchmark on RepoBench-P, or asks about evaluating this task. Reports Exact Match (EM).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.03091
  bibtex_key: liu2023repobench
  confidence: high
---

# repobench-p-eval

> RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems — Liu et al. (2023) (arXiv:2306.03091, 2023)

## What this evaluates

Evaluates an end-to-end pipeline that first retrieves cross-file snippets and then predicts the next line of code using both the in-file context and retrieved snippets.

## Datasets

- **RepoBench-P** — total ?; splits: test (-1)

## Metrics

- `Exact Match (EM)` **(primary)** — range: [0, 1]
  - 1 if the predicted line exactly matches the gold line, 0 otherwise.
- `Edit Similarity (ES)` — range: [0, 1]
  - Token-level edit similarity between predicted and gold lines.

## Input / output format

**Input**: In-file context (up to 1600 tokens, max 60 lines) and retrieved cross-file snippets (up to 6400 total tokens).

**Output**: Predicted next line of code.

## Scoring recipe

```python
def score(pred, gold):
    em = 1.0 if pred.strip() == gold.strip() else 0.0
    es = edit_similarity(pred, gold)
    return em, es
```

## Common pitfalls

- Retrieval strategies (Gold-Only, Gold-Filled, UniXcoder, Random, Baseline) drastically affect results.
- Gold snippets are included for reference only and do not represent oracle capabilities.

## Evidence (verbatim from paper)

> Table 4: Comparison of various retrieval strategies on the RepoBench-P ... evaluated in terms of Exact Match (EM) and Edit Similarity (ES) metrics

## Citation

```bibtex
@misc{liu2023repobench,
  title={RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems},
  author={Liu et al. (2023)},
  year={2023},
  note={arXiv:2306.03091}
}
```

- arXiv: 2306.03091