Swebench Eval

---
name: swebench-eval
description: Evaluates language models' ability to resolve real-world software engineering issues by generating code patches. It probes long-context reasoning, cross-file dependency understanding, and execution-based validation within large, complex codebases. Use when the user wants to benchmark on SWE-bench, or asks about evaluating this task. Reports resolve_rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2310.06770
  bibtex_key: cobbe2023swebench
  confidence: high
---

# swebench-eval

> SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — Cobbe et al. (2023) (arXiv:2310.06770, 2023)

## What this evaluates

Evaluates language models' ability to resolve real-world software engineering issues by generating code patches. It probes long-context reasoning, cross-file dependency understanding, and execution-based validation within large, complex codebases.

## Datasets

- **SWE-bench** — total ?; splits: (unstated)

## Metrics

- `resolve_rate` **(primary)** — range: percent
  - Percentage of benchmark instances where the model's generated patch successfully resolves the issue, typically validated by passing all associated repository tests.

## Input / output format

**Input**: Task instructions, issue description text, retrieved source files (via BM25 or oracle retrieval), documentation, an example patch file, and a prompt instructing the model to generate a patch.

**Output**: A code patch file intended to resolve the specified GitHub issue.

## Scoring recipe

```python
resolved = 0
for instance in dataset:
    patch = model.generate(instance.input)
    if apply_and_run_tests(patch, instance.repo) == 'PASS_ALL':
        resolved += 1
resolve_rate = (resolved / len(dataset)) * 100
```

## Common pitfalls

- Context window limits force aggressive file retrieval, which may exclude necessary context or include irrelevant files.
- Oracle retrieval assumes knowledge of edited files, making it unrealistic for actual engineering workflows.
- Tokenization differences across models (e.g., Llama vs GPT-4) cause non-standard context length measurements, affecting fair comparison.

## Evidence (verbatim from paper)

> SWE-bench instances provide an issue description and a codebase as input to the model... we construct the input to the model consisting of task instructions, the issue text, retrieved files and documentation, and finally an example patch file and prompt for generating the patch file. Table 2: Model resolve rates with BM25 retrieval, with different maximum context lengths.

## Citation

```bibtex
@misc{cobbe2023swebench,
  title={SWE-bench: Can Language Models Resolve Real-World GitHub Issues?},
  author={Cobbe et al. (2023)},
  year={2023},
  note={arXiv:2310.06770}
}
```

- arXiv: 2310.06770