Multi Swe Bench Eval

---
name: multi-swe-bench-eval
description: This benchmark evaluates an LLM's ability to resolve software engineering issues across multiple programming languages. It probes capabilities in long-context reasoning, multi-file code patching, and fault localization by requiring models to generate executable fixes for real-world repository issues. Use when the user wants to benchmark on Multi-SWE-bench, or asks about evaluating this task. Reports Resolved Rate (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2504.02605
  bibtex_key: zan2025multi
  confidence: high
---

# multi-swe-bench-eval

> Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving — Zan et al. (2025) (arXiv:2504.02605, 2025)

## What this evaluates

This benchmark evaluates an LLM's ability to resolve software engineering issues across multiple programming languages. It probes capabilities in long-context reasoning, multi-file code patching, and fault localization by requiring models to generate executable fixes for real-world repository issues.

## Datasets

- **Multi-SWE-bench** — total 1632; splits: test (1632); repo https://github.com/multi-swe-bench/multi-swe-bench

## Metrics

- `Resolved Rate (%)` **(primary)** — range: percent
  - Percentage of issues for which the model's generated patch successfully resolves the reported bug or feature request, typically validated via reproduction tests.
- `Success Location (%)` — range: percent
  - Accuracy of fault localization at the file level, measuring whether the model correctly identifies the file containing the bug.
- `Average Cost ($)` — range: other
  - Average monetary cost incurred per issue resolved, accounting for API calls and execution resources.

## Input / output format

**Input**: Per instance, the model receives the issue description, full file contents of the target repository, and a pruned repository directory structure. Language-specific environment setup and test commands are also provided.

**Output**: The model must produce a code patch (diff) or a sequence of file edits intended to resolve the reported issue, which can be applied via git or direct file modification.

## Scoring recipe

```python
def evaluate(predictions, gold_instances):
    resolved = 0
    loc_correct = 0
    total_cost = 0.0
    for pred, gold in zip(predictions, gold_instances):
        if apply_patch_and_run_tests(pred.patch, gold.repo, gold.test_script):
            resolved += 1
        if pred.located_file == gold.target_file:
            loc_correct += 1
        total_cost += pred.api_cost
    return {
        'Resolved Rate (%)': (resolved / len(gold_instances)) * 100,
        'Success Location (%)': (loc_correct / len(gold_instances)) * 100,
        'Average Cost ($)': total_cost / len(gold_instances)
    }
```

## Common pitfalls

- Extracting file skeletons is challenging in some programming languages, requiring full file content inputs that may exceed LLM context limits.
- Compiled artifacts (e.g., .o, .bin) can interfere with git apply if not explicitly excluded via .gitignore.
- Regression and reproduction testing is cumbersome to implement across languages, so patch validation pipelines must be carefully adapted per language.

## Evidence (verbatim from paper)

> Following SWE-Bench*[Jimenez et al., [2023]]* and SWE-Lancer*[Miserendino et al., [2025]]*, we adopt Resolved Rate (%) as our primary evaluation metric, measuring the percentage of issues resolved. In addition, we report several other metrics to provide a more detailed analysis: Success Location (%) — the accuracy of fault localization at file level; and Average Cost ($) — the average cost per issue.

## Citation

```bibtex
@misc{zan2025multi,
  title={Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving},
  author={Zan et al. (2025)},
  year={2025},
  note={arXiv:2504.02605}
}
```

- arXiv: 2504.02605