Swe Bench Eval

---
name: swe-bench-eval
description: Evaluate a code-editing agent / LLM on SWE-Bench (real GitHub issues → patches that pass the PR's tests). Use when the user wants to benchmark a coding agent on realistic software-engineering tasks, mentions "SWE-Bench / SWE-Bench Lite / SWE-Bench Verified", or asks "can my model fix real GitHub issues?". Reports resolve_rate (% issues whose generated patch passes the original PR's hidden tests).
metadata:
  source_paper: arXiv:2310.06770
  source_repo: https://github.com/princeton-nlp/SWE-bench
  task_type: agentic-code-editing
  metrics: [resolve_rate]
  splits: [SWE-bench (2294), SWE-bench-Lite (300), SWE-bench-Verified (500)]
---

# swe-bench-eval — Evaluate code agents on real GitHub issues

Source: Jimenez et al., "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" (arXiv:2310.06770).

## What this benchmark tests

Given a real GitHub issue + the **state of the codebase before the fix PR**, the model must produce a patch (`unified diff`) that, when applied, makes the original PR's hidden test suite pass. This exercises long-context reasoning, cross-file edits, and execution-grounded correctness — far beyond function-level codegen.

## When to invoke this skill

- User says "evaluate / benchmark / test on SWE-Bench"
- User asks how good a coding agent is at "real bug fixes" / "GitHub issues" / "multi-file edits"
- User mentions any variant: `SWE-Bench`, `SWE-Bench Lite`, `SWE-Bench Verified`, `SWE-Bench Multimodal`, `SWE-Bench Live`

## Variants — pick before running

| Variant | Size | When to use |
|---|---:|---|
| `princeton-nlp/SWE-bench` | 2,294 | full bench, most expensive; only run when you have hours of compute |
| `princeton-nlp/SWE-bench_Lite` | 300 | filtered for easier tasks (single-file edits, clear issues) — typical first pass |
| `princeton-nlp/SWE-bench_Verified` | 500 | OpenAI human-verified subset, **the most credible number** to publish |

Default to `_Verified` unless the user wants the full sweep.

## Setup

```bash
pip install swebench  # official harness with docker eval images
```

Each instance has: `repo`, `instance_id`, `base_commit`, `problem_statement` (issue text), `hints_text`, `test_patch` (the gold tests, hidden from the model), `patch` (the gold diff, hidden from the model), `FAIL_TO_PASS` / `PASS_TO_PASS` test lists.

## Evaluation protocol (verbatim from §4 + §5)

1. **Input to the agent**: `problem_statement` + a way to read the codebase at `base_commit`. Either:
   - `oracle` retrieval — give the model exactly the files edited in gold patch (upper-bound study only)
   - `bm25` retrieval — top-k files by BM25 over issue text (paper's main setting)
   - `agentic` — let the agent navigate the repo itself (most modern submissions)
2. **Output**: a `model_patch` field (unified diff against `base_commit`).
3. **Scoring**: `swebench` harness builds a docker image per instance, applies `model_patch`, then runs `FAIL_TO_PASS` + `PASS_TO_PASS` tests. Instance is **resolved** iff all `FAIL_TO_PASS` pass AND all `PASS_TO_PASS` still pass.
4. **Headline metric**: `resolve_rate = resolved / total`. Always report alongside `applied_rate` (% patches that even apply cleanly) for diagnosis.

## Reference scoring (sketch)

```bash
# 1. Save predictions as JSONL: one line per instance with model_name_or_path + instance_id + model_patch
# 2. Run the harness:
python -m swebench.harness.run_evaluation \
    --dataset_name princeton-nlp/SWE-bench_Verified \
    --predictions_path predictions.jsonl \
    --max_workers 8 \
    --run_id my-eval-2026
# Outputs: ./logs/run_evaluation/my-eval-2026/<model>/results.json with resolve_rate.
```

## Output format the agent must produce

```jsonl
{"instance_id": "django__django-12345", "model_name_or_path": "claude-opus-4-7", "model_patch": "diff --git a/...\n--- a/...\n+++ b/...\n@@ ..."}
```

Failure modes that produce 0% resolve rate even with smart edits:
- Patch references file paths that don't exist at `base_commit` (model hallucinated structure)
- Patch breaks `PASS_TO_PASS` tests (regression)
- Patch doesn't apply because of context drift (whitespace / line numbers off)

## Don'ts

- Don't compute resolve_rate yourself by diffing predictions against the gold patch — patches that look different can still resolve the issue. **Always run the harness.**
- Don't conflate `_Lite` / `_Verified` / full numbers in the same table without labelling.
- Don't forget `PASS_TO_PASS`; without that you're rewarding regressions.

## Citation

```bibtex
@inproceedings{jimenez2024swebench,
  title={SWE-bench: Can Language Models Resolve Real-World GitHub Issues?},
  author={Jimenez, Carlos E. and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik},
  booktitle={ICLR}, year={2024}
}
```