Swe Bench Java Eval

---
name: swe-bench-java-eval
description: This benchmark evaluates an AI agent's ability to autonomously resolve real-world GitHub issues in Java projects. It probes capabilities in code patch generation, repository navigation, test case reasoning, and handling runtime environment dependencies. Use when the user wants to benchmark on SWE-bench-java-verified, or asks about evaluating this task. Reports Resolved Rate (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2408.14354
  bibtex_key: zan2024swebenchjava
  confidence: high
---

# swe-bench-java-eval

> SWE-bench-java: A GitHub Issue Resolving Benchmark for Java — Zan et al. (2024) (arXiv:2408.14354, 2024)

## What this evaluates

This benchmark evaluates an AI agent's ability to autonomously resolve real-world GitHub issues in Java projects. It probes capabilities in code patch generation, repository navigation, test case reasoning, and handling runtime environment dependencies.

## Datasets

- **SWE-bench-java-verified** — total 137; splits: test (137)

## Metrics

- `Resolved Rate (%)` **(primary)** — range: percent
  - The proportion of issues successfully resolved out of the total dataset. An issue is marked resolved only if all provided test cases pass after applying the model's generated patch.

## Input / output format

**Input**: GitHub issue description, full repository codebase, and a set of failing test cases associated with the issue.

**Output**: A code patch (modified source files) intended to fix the issue and make all associated test cases pass.

## Scoring recipe

```python
resolved_count = 0
for instance in dataset:
    patch = model.generate_patch(instance.issue, instance.repo, instance.tests)
    if run_tests(instance.repo, patch, instance.tests).all_pass:
        resolved_count += 1
return (resolved_count / len(dataset)) * 100
```

## Common pitfalls

- Runtime environment must be correctly configured for each Java issue; missing setup prevents accurate issue reproduction and artificially lowers scores.
- Resolution requires ALL given test cases to pass, not just a subset or the main failing test.
- The benchmark evaluates agentic workflows (SWE-agent) rather than direct code completion, so evaluation depends on the agent's ability to navigate, edit, and execute tests autonomously.

## Evidence (verbatim from paper)

> Following SWE-bench [1], we adopt the Resolved Rate (%) as our evaluation metric. This metric indicates the proportion of issues in SWE-bench-java-verified that are successfully resolved. An issue is considered resolved only if all given test cases pass. This metric provides a precise measure of effectiveness in resolving real-world Java GitHub issues.

## Citation

```bibtex
@misc{zan2024swebenchjava,
  title={SWE-bench-java: A GitHub Issue Resolving Benchmark for Java},
  author={Zan et al. (2024)},
  year={2024},
  note={arXiv:2408.14354}
}
```

- arXiv: 2408.14354