Repobench R Eval

---
name: repobench-r-eval
description: Evaluates a model's ability to retrieve relevant cross-file code snippets given an in-file context for predicting the next line of code. Use when the user wants to benchmark on RepoBench-R, or asks about evaluating this task. Reports acc@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2306.03091
  bibtex_key: liu2023repobench
  confidence: high
---

# repobench-r-eval

> RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems — Liu et al. (2023) (arXiv:2306.03091, 2023)

## What this evaluates

Evaluates a model's ability to retrieve relevant cross-file code snippets given an in-file context for predicting the next line of code.

## Datasets

- **RepoBench-R** — total ?; splits: test (-1)

## Metrics

- `acc@1` **(primary)** — range: [0, 1]
  - Accuracy at top-1 retrieval. 1 if the gold snippet is ranked first, 0 otherwise.
- `acc@3` — range: [0, 1]
  - Accuracy at top-3 retrieval.
- `acc@5` — range: [0, 1]
  - Accuracy at top-5 retrieval.

## Input / output format

**Input**: Cropped in-file code (typically last 3 lines) and a pool of candidate code snippets from other files in the repository.

**Output**: Top-k ranked candidate code snippets.

## Scoring recipe

```python
def score(predictions, gold):
    acc1 = 1.0 if gold in predictions[:1] else 0.0
    acc3 = 1.0 if gold in predictions[:3] else 0.0
    acc5 = 1.0 if gold in predictions[:5] else 0.0
    return acc1, acc3, acc5
```

## Common pitfalls

- Random retrieval baseline requires averaging over 100 runs for stability.
- Cropping m=3 lines is standard but ablation studies show sensitivity to the number of kept lines.

## Evidence (verbatim from paper)

> Table 2 presents a detailed comparison of different retrieval strategies in RepoBench-R. | acc@1 | acc@3 | acc@5 |

## Citation

```bibtex
@misc{liu2023repobench,
  title={RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems},
  author={Liu et al. (2023)},
  year={2023},
  note={arXiv:2306.03091}
}
```

- arXiv: 2306.03091