Codeflaws Repair Eval

---
name: codeflaws-repair-eval
description: This evaluation probes an LLM's ability to automatically detect and fix bugs in C programs by generating correct patches. It measures how effectively the model leverages test feedback, fault localization scores, and iterative reasoning to pass all provided test cases for each buggy submission. Use when the user wants to benchmark on Codeflaws, or asks about evaluating this task. Reports Repair Accuracy.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.01947
  bibtex_key: [REDACTED]
  confidence: high
---

# codeflaws-repair-eval

> Automated Repair of C Programs Using Large Language Models — Farzandway et al. (2025) (arXiv:2509.01947, 2025)

## What this evaluates

This evaluation probes an LLM's ability to automatically detect and fix bugs in C programs by generating correct patches. It measures how effectively the model leverages test feedback, fault localization scores, and iterative reasoning to pass all provided test cases for each buggy submission.

## Datasets

- **Codeflaws** — total 3902; splits: test (3902)

## Metrics

- `Repair Accuracy` **(primary)** — range: percent
  - A bug is considered correctly repaired if the generated patch passes all test cases. Calculated as the fraction of bugs where the patch passes every provided test.
- `Time-to-Repair` — range: other
  - Total elapsed time including fault localization, patch generation, and validation.
- `Partial Repair Rate` — range: percent
  - The proportion of errors where the repair attempt produces a patch that passes some, but not all, of the previously failing test cases.

## Input / output format

**Input**: Buggy C source code, associated test cases with pass/fail results, expected outputs, and optionally SBFL suspiciousness scores, compiler/runtime logs, and historical debugging context.

**Output**: Corrected C source code patch.

## Scoring recipe

```python
def compute_metrics(predictions, gold_tests):
    total = len(predictions)
    repaired = 0
    partially_repaired = 0
    for patch, tests in zip(predictions, gold_tests):
        results = run_tests(patch, tests)
        if all(results):
            repaired += 1
        elif any(results):
            partially_repaired += 1
    accuracy = repaired / total
    partial_rate = partially_repaired / total
    return accuracy, partial_rate
```

## Common pitfalls

- Evaluating on only a subset of the provided test cases instead of requiring the patch to pass all of them for a positive repair accuracy score.
- Ignoring compilation failures or infinite loops, which should be treated as test failures or handled with strict timeouts (e.g., 2 minutes per test).
- Counting a bug as repaired if it passes only the initially failing tests, rather than verifying it passes the entire test suite including previously passing cases.

## Evidence (verbatim from paper)

> We evaluated the system’s performance using the following metrics: Repair Accuracy: A bug is considered correctly repaired if the generated patch passes *all* test cases. Time-to-Repair: Measured as the total elapsed time including fault localization, patch generation, and validation. Partial Repair Rate: The proportion of errors where the repair attempt produces a patch that passes some, but not all, of the previously failing test cases.

## Citation

```bibtex
@misc{farzandway2025automated,
  title={Automated Repair of C Programs Using Large Language Models},
  author={Farzandway et al. (2025)},
  year={2025},
  note={arXiv:2509.01947}
}
```

- arXiv: 2509.01947