Reproducibility Repair Eval

---
name: reproducibility-repair-eval
description: Evaluates the ability of LLMs and AI agents to automatically repair broken R-based social science code and restore computational reproducibility. It probes how well different workflows handle varying error complexities and contextual information. Use when the user wants to benchmark on Custom R-based Social Science Code Dataset, or asks about evaluating this task. Reports reproduction_success_rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.08561
  bibtex_key: [REDACTED]
  confidence: high
---

# reproducibility-repair-eval

> Automating Computational Reproducibility in Social Science: Comparing Prompt-Based and Agent-Based Approaches — Shah et al. (2026) (arXiv:2602.08561, 2026)

## What this evaluates

Evaluates the ability of LLMs and AI agents to automatically repair broken R-based social science code and restore computational reproducibility. It probes how well different workflows handle varying error complexities and contextual information.

## Datasets

- **Custom R-based Social Science Code Dataset** — total ?; splits: test (-1); repo https://github.com/Mehtab07/Automating-Computational-Reproducibility

## Metrics

- `reproduction_success_rate` **(primary)** — range: percent
  - Percentage of test cases where the repaired script executes successfully and produces output identical to the original study's ground truth results. Calculated as (successful reproductions / total cases) × 100.

## Input / output format

**Input**: For prompt-based: R script with injected errors, error logs, and context prompts (minimal, medium, or full including original paper/scripts). For agent-based: Same code + containerized execution environment allowing iterative inspection, modification, and re-execution.

**Output**: Repaired R script intended to restore original analytical outputs.

## Scoring recipe

```python
def calculate_success_rate(predictions, gold_outputs):
    successful = 0
    for pred, gold in zip(predictions, gold_outputs):
        result = execute_script_in_isolated_env(pred)
        if result is not None and result == gold:
            successful += 1
    return (successful / len(predictions)) * 100
```

## Common pitfalls

- Success is strictly defined by exact output reproduction matching the original study, not just successful code execution.
- Performance is highly sensitive to prompt context richness; adding irrelevant or excessive context can sometimes degrade LLM performance.
- Error complexity is categorized into three levels (A: straightforward, B: missing libraries/small code, C: complex logic), which drastically affect success rates.

## Evidence (verbatim from paper)

> We evaluated performance using the same metrics across workflows, focusing on reproduction success rates broken down by error category, prompt type, model or agent used, and reproducibility status as determined by output validation.

## Citation

```bibtex
@misc{shah2026automating,
  title={Automating Computational Reproducibility in Social Science: Comparing Prompt-Based and Agent-Based Approaches},
  author={Shah et al. (2026)},
  year={2026},
  note={arXiv:2602.08561}
}
```

- arXiv: 2602.08561