Paperbench Eval

---
name: paperbench-eval
description: Evaluates an autonomous agent's ability to replicate top-tier conference ML papers from scratch. It probes long-horizon engineering capabilities by measuring performance across 20 diverse tasks under a strict 24-hour time and compute budget. Use when the user wants to benchmark on PaperBench, or asks about evaluating this task. Reports Average Score.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.13018
  bibtex_key: chen2026toward
  confidence: high
---

# paperbench-eval

> Toward Autonomous Long-Horizon Engineering for ML Research — Chen et al. (2026) (arXiv:2604.13018, 2026)

## What this evaluates

Evaluates an autonomous agent's ability to replicate top-tier conference ML papers from scratch. It probes long-horizon engineering capabilities by measuring performance across 20 diverse tasks under a strict 24-hour time and compute budget.

## Datasets

- **PaperBench** — total 20; splits: test (20)

## Metrics

- `Average Score` **(primary)** — range: percent
  - Arithmetic mean of task-level scores across all 20 tasks. Scores are percentages derived from the official grading protocol.

## Input / output format

**Input**: A top-tier conference ML paper to be replicated from scratch, with a 24-hour time budget and one H20 GPU allocated per task.

**Output**: Replicated code, experimental setups, and results for the target paper.

## Scoring recipe

```python
def score_paperbench(predictions, gold):
    task_scores = []
    for task in predictions:
        # Graded by GPT-5.4 using official PaperBench protocol
        score = grade_with_gpt54(task.output, task.gold)
        task_scores.append(score)
    return sum(task_scores) / len(task_scores)
```

## Common pitfalls

- Grading relies on GPT-5.4, which costs ~$832 for the full 20-task evaluation, severely limiting repeated runs.
- Cost per task varies significantly across baselines (e.g., $27.44 vs $15.67), so efficiency must be tracked alongside accuracy.

## Evidence (verbatim from paper)

> Table 1: Main results on PaperBench full evaluation. Values in red indicate AiScientist’s gains over the best baseline. Bold and underlined denote the best and second-best results within each LLM.

| Average Score | 19.26 | 20.60 | 30.52 | +9.92 | 22.58 | 22.37 | 33.73 | +11.15 |

## Citation

```bibtex
@misc{chen2026toward,
  title={Toward Autonomous Long-Horizon Engineering for ML Research},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2604.13018}
}
```

- arXiv: 2604.13018