Rexbench Eval

---
name: rexbench-eval
description: This benchmark evaluates the ability of LLM-based coding agents to autonomously implement research extensions by modifying existing AI/ML codebases based on domain-expert instructions. It probes complex, multi-step software engineering capabilities, including codebase navigation, hypothesis-driven implementation, and producing executable patches. Use when the user wants to benchmark on REXBench, or asks about evaluating this task. Reports final success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.22598
  bibtex_key: [REDACTED]
  confidence: high
---

# rexbench-eval

> RExBench: Can coding agents autonomously implement AI research extensions? — Edwards et al. (2025) (arXiv:2506.22598, 2025)

## What this evaluates

This benchmark evaluates the ability of LLM-based coding agents to autonomously implement research extensions by modifying existing AI/ML codebases based on domain-expert instructions. It probes complex, multi-step software engineering capabilities, including codebase navigation, hypothesis-driven implementation, and producing executable patches.

## Datasets

- **REXBench** — total 12; splits: test (12); repo https://github.com/tinlaboratory/RexBench

## Metrics

- `final success rate` **(primary)** — range: [0, 1]
  - Percentage of successful task executions out of total attempts. Each task is run three times to account for agent randomness, and a run is successful if the generated git patch applies and executes correctly in an isolated environment.

## Input / output format

**Input**: Existing codebase, research paper, and domain-expert-written instructions for a specific modification task.

**Output**: Git patch files containing the proposed code modifications.

## Scoring recipe

```python
total_runs = 0
successful_runs = 0
for task in tasks:
    patch = agent.generate(task.inputs)
    for _ in range(3):
        total_runs += 1
        if apply_patch_and_execute(patch, isolated_env):
            successful_runs += 1
return successful_runs / total_runs
```

## Common pitfalls

- Agents often achieve high file recall by correctly locating edit targets but still fail to produce executable code.
- Hints are cumulative; second-level hints do not consistently yield additional benefits over first-level hints.
- Evaluation requires running each task three times to account for agent randomness, which significantly impacts reported success rates.
- Python execution is disabled during the agent's generation phase, but patches are evaluated post-hoc via automated execution.

## Evidence (verbatim from paper)

> Our main results are shown in Figure 2. We see that most agents struggle with the task, with the best performing agents (OpenHands + Claude 3.7 Sonnet and Claude Code) achieving 25% average final success rates.

## Citation

```bibtex
@misc{edwards2025rexbench,
  title={RExBench: Can coding agents autonomously implement AI research extensions?},
  author={Edwards et al. (2025)},
  year={2025},
  note={arXiv:2506.22598}
}
```

- arXiv: 2506.22598