Repobench C Eval

---
name: repobench-c-eval
description: Evaluates autoregressive language models on predicting the next line of code using provided in-file and cross-file contexts. Use when the user wants to benchmark on RepoBench-C, or asks about evaluating this task. Reports Exact Match (EM).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.03091
  bibtex_key: liu2023repobench
  confidence: high
---

# repobench-c-eval

> RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems — Liu et al. (2023) (arXiv:2306.03091, 2023)

## What this evaluates

Evaluates autoregressive language models on predicting the next line of code using provided in-file and cross-file contexts.

## Datasets

- **RepoBench-C** — total ?; splits: test (-1)

## Metrics

- `Exact Match (EM)` **(primary)** — range: [0, 1]
  - 1 if the predicted line exactly matches the gold line, 0 otherwise.
- `Edit Similarity (ES)` — range: [0, 1]
  - Token-level edit similarity between predicted and gold lines.

## Input / output format

**Input**: In-file context (import statements and preceding lines) and cross-file context (snippets from other files parsed by import statements).

**Output**: Predicted next line of code.

## Scoring recipe

```python
def score(pred, gold):
    em = 1.0 if pred.strip() == gold.strip() else 0.0
    es = edit_similarity(pred, gold)  # token-level
    return em, es
```

## Common pitfalls

- Model-language pairings must be strictly matched (e.g., CodeGen-Mono for Python, CodeGen-Multi for Java).
- Context length limits (2k vs 8k) significantly impact performance and can cause out-of-distribution issues.

## Evidence (verbatim from paper)

> Table 3: RepoBench-C performance ... evaluated using Exact Match (EM), Edit Similarity (ES).

## Citation

```bibtex
@misc{liu2023repobench,
  title={RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems},
  author={Liu et al. (2023)},
  year={2023},
  note={arXiv:2306.03091}
}
```

- arXiv: 2306.03091