Swe Bench Repair Eval

---
name: swe-bench-repair-eval
description: This evaluation probes an LLM-based agent's ability to automatically locate faults and generate correct code patches for real-world software issues. It tests both traditional text-only bug fixing and multimodal reasoning where visual UI behavior must be understood alongside code. Use when the user wants to benchmark on SWE-bench Lite, SWE-bench Multimodal, or asks about evaluating this task. Reports %Resolved.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2603.01048
  bibtex_key: pan2026reporepair
  confidence: high
---

# swe-bench-repair-eval

> RepoRepair: Leveraging Code Documentation for Repository-Level Automated Program Repair — Zhongqiang Pan et al. (2026) (arXiv:2603.01048, 2026)

## What this evaluates

This evaluation probes an LLM-based agent's ability to automatically locate faults and generate correct code patches for real-world software issues. It tests both traditional text-only bug fixing and multimodal reasoning where visual UI behavior must be understood alongside code.

## Datasets

- **SWE-bench Lite** — total 323; splits: dev (23), test (300)
- **SWE-bench Multimodal** — total 619; splits: dev (102), test (517)

## Metrics

- `%Resolved` **(primary)** — range: percent
  - Percentage of issues successfully fixed in the benchmark. All patches are validated via automated test suites; any patch that passes is considered correct.
- `Avg. $Cost` — range: other
  - Average inference cost per tool run.
- `%Correct Localization` — range: percent
  - Percentage of cases where the tool’s localized file set fully contains all developer-patched files. It is a recall-oriented metric.

## Input / output format

**Input**: Issue description (text-only or multimodal with images/GIFs/videos), repository codebase, and retrieved context (top-50 relevant files, then top-5 suspicious files with functions/classes).

**Output**: A code patch (diff) intended to fix the reported issue, validated against the repository's test suite.

## Scoring recipe

```python
def compute_resolved_rate(predictions, test_suites):
    resolved = 0
    for issue_id, patch in predictions.items():
        if test_suites[issue_id].run(patch).passes:
            resolved += 1
    return (resolved / len(predictions)) * 100
```

## Common pitfalls

- The 'plausible patch' distinction is explicitly discarded for repository-level evaluation; only patches that pass the full automated test suite count as correct.
- The localization metric is recall-oriented: the predicted file set must fully contain all developer-patched files, but may include additional false positives.
- Multimodal inputs like GIFs or videos require preprocessing via SSIM-based keyframe extraction before being fed to the LLM.

## Evidence (verbatim from paper)

> For evaluation on SWE-bench Lite and SWE-bench Multimodal, we adopt the established metrics from prior work: (1) %Resolved - the percentage of issues successfully fixed in the benchmark. Notably, for repository-level evaluation, all patches are validated via automated test suites; any patch that passes is considered correct, and the distinction of ”plausible patch” is no longer used. (2) Avg. $Cost - the average inference cost per tool run. Additionally, we introduce %Correct Localization - the percentage of cases where the tool’s localized files exactly match those modified in developer patches.

## Citation

```bibtex
@misc{pan2026reporepair,
  title={RepoRepair: Leveraging Code Documentation for Repository-Level Automated Program Repair},
  author={Zhongqiang Pan et al. (2026)},
  year={2026},
  note={arXiv:2603.01048}
}
```

- arXiv: 2603.01048