Swe Bench Eval

---
name: swe-bench-eval
description: Evaluates an agent's ability to automatically resolve software engineering issues by generating and applying code patches to open-source repositories. It measures functional correctness by running the target repository's test suite after the patch is applied. Use when the user wants to benchmark on SWE-bench, HumanEvalFix, or asks about evaluating this task. Reports pass@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2405.15793
  bibtex_key: yang2024sweagent
  confidence: high
---

# swe-bench-eval

> SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering — Yang et al. (2024) (arXiv:2405.15793, 2024)

## What this evaluates

Evaluates an agent's ability to automatically resolve software engineering issues by generating and applying code patches to open-source repositories. It measures functional correctness by running the target repository's test suite after the patch is applied.

## Datasets

- **SWE-bench** — total 2294; splits: test (2294), lite (300)
- **HumanEvalFix** — total ?; splits: test (-1)

## Metrics

- `pass@1` **(primary)** — range: percent
  - The proportion of task instances for which all repository tests pass successfully after the model's generated patch is applied to the codebase.
- `Avg. Cost` — range: other
  - The average API inference cost incurred by the agent, calculated only over instances that were successfully resolved.

## Input / output format

**Input**: System prompt, issue description, optionally a demonstration, and the full codebase context.

**Output**: A generated patch file or series of code edits applied to the repository.

## Scoring recipe

```python
resolved_count = 0
for pred in predictions:
    apply_patch_to_repo(pred)
    if run_repo_tests().all_pass:
        resolved_count += 1
return (resolved_count / len(predictions)) * 100
```

## Common pitfalls

- The evaluation requires running the full test suite of the target repository, which can be computationally expensive and sensitive to environment setup.
- A per-instance budget cap ($4) is enforced; runs exceeding this budget are automatically submitted, which may artificially lower the pass rate if the agent hasn't finished editing.

## Evidence (verbatim from paper)

> We report % Resolved or pass@1 as the main metric, which is the proportion of instances for which all tests pass successfully after the model generated patch is applied to the repository [20].

## Citation

```bibtex
@misc{yang2024sweagent,
  title={SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering},
  author={Yang et al. (2024)},
  year={2024},
  note={arXiv:2405.15793}
}
```

- arXiv: 2405.15793