Swe Rebench V2 Eval

---
name: swe-rebench-v2-eval
description: Evaluates the ability of LLM-based agents to autonomously resolve software engineering issues by modifying code in real-world repositories. It probes environment setup, code generation, and test execution capabilities across multiple programming languages. Use when the user wants to benchmark on SWE-rebench V2, or asks about evaluating this task. Reports pass@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.23866
  bibtex_key: badertdinov2026swe_rebench_v2
  confidence: high
---

# swe-rebench-v2-eval

> SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale — Badertdinov et al. (2026) (arXiv:2602.23866, 2026)

## What this evaluates

Evaluates the ability of LLM-based agents to autonomously resolve software engineering issues by modifying code in real-world repositories. It probes environment setup, code generation, and test execution capabilities across multiple programming languages.

## Datasets

- **SWE-rebench V2** — total 32000; splits: test (300)

## Metrics

- `pass@1` **(primary)** — range: percent
  - Fraction of tasks where at least one out of k attempts successfully passes the test suite.

## Input / output format

**Input**: Repository codebase, issue description, and test suite provided in an interactive shell environment with base dependencies installed.

**Output**: Modified source code files and/or shell commands to resolve the issue. Success is determined by running the provided test suite.

## Scoring recipe

```python
def compute_pass_at_k(successes_per_task, k):
    # successes_per_task: list of lists of booleans (k attempts per task)
    passed = [any(attempts) for attempts in successes_per_task]
    return sum(passed) / len(passed)
```

## Common pitfalls

- Test suite coupling: models may correctly fix the target issue but fail due to regressions in unrelated code paths caught by the test suite.
- Implicit naming requirements: tests often expect specific implementation details not specified in the problem statement.
- External dependencies: problem statements may reference inaccessible URLs or auth-walled resources, causing agent failure.

## Evidence (verbatim from paper)

> Table 6: Pass rates (%) by model and programming language. | Model | Py | JS | Go | Rust | Scala | pass@1 | pass@3 |

## Citation

```bibtex
@misc{badertdinov2026swe_rebench_v2,
  title={SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale},
  author={Badertdinov et al. (2026)},
  year={2026},
  note={arXiv:2602.23866}
}
```

- arXiv: 2602.23866