Llm Apr Eval

---
name: llm-apr-eval
description: Evaluates the ability of large language models to automatically generate correct code patches for buggy functions across Java, JavaScript, Python, and PHP. It probes language-specific repair capabilities, the impact of providing test case information, and the sensitivity to fault localization granularity. Use when the user wants to benchmark on Defects4J, BugsInPy, BugsJS, BugsPHP, or asks about evaluating this task. Reports plausible@1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2506.03283
  bibtex_key: campos2025empirical
  confidence: high
---

# llm-apr-eval

> Empirical Evaluation of Generalizable Automated Program Repair with Large Language Models — Campos et al. (2025) (arXiv:2506.03283, 2025)

## What this evaluates

Evaluates the ability of large language models to automatically generate correct code patches for buggy functions across Java, JavaScript, Python, and PHP. It probes language-specific repair capabilities, the impact of providing test case information, and the sensitivity to fault localization granularity.

## Datasets

- **Defects4J** — total ?; splits: test (-1)
- **BugsInPy** — total ?; splits: test (-1)
- **BugsJS** — total ?; splits: test (-1)
- **BugsPHP** — total ?; splits: test (-1)

## Metrics

- `plausible@1` **(primary)** — range: percent
  - Success rate when generating a single plausible patch. A patch is considered plausible if it passes all provided test cases without introducing new failures. Computed as the fraction of problems where the top-1 generated patch is plausible.
- `plausible@5` — range: percent
  - Success rate among the top 5 generated candidates. Computed as the fraction of problems where at least one of the top-5 generated patches is plausible.
- `p_combined@k` — range: percent
  - Probability that at least one model in a committee produces a plausible top-k result. Formula: E[1 - prod_{i=1 to m} (C(n-c_i, k) / C(n, k))], where n is total problems, c_i is correct generations by model i, and m is committee size.

## Input / output format

**Input**: Buggy function code (base prompt), optionally augmented with line-level fault localization hints (comments marking suspicious lines) and/or names, source code, and error messages of failing test cases.

**Output**: A single code patch (function replacement or diff) intended to fix the buggy function.

## Scoring recipe

```python
def compute_plausible_k(predictions, test_cases, k=1):
    plausible_count = 0
    for problem_preds in predictions:
        top_k_patches = problem_preds[:k]
        if any(patch_passes_tests(patch, test_cases) for patch in top_k_patches):
            plausible_count += 1
    return (plausible_count / len(predictions)) * 100
```

## Common pitfalls

- Python indentation errors in generated patches frequently cause execution failures, artificially lowering plausible scores unless a post-processing linting/formatting step is applied.
- Assuming perfect fault localization (method- or line-level) overestimates real-world performance, as noisy tool-generated localization causes significant accuracy drops.
- Model performance is highly language-specific; averaging across languages without stratification masks specialization and committee complementarity effects.

## Evidence (verbatim from paper)

> Table II summarizes the performance of all models across the four benchmarks, reporting plausible@1 (the success rate when generating a single fix) and plausible@5 (success among five generated candidates).

## Citation

```bibtex
@misc{campos2025empirical,
  title={Empirical Evaluation of Generalizable Automated Program Repair with Large Language Models},
  author={Campos et al. (2025)},
  year={2025},
  note={arXiv:2506.03283}
}
```

- arXiv: 2506.03283