Exp Bench Eval

---
name: exp-bench-eval
description: Evaluates AI agents' end-to-end capability to conduct real AI research experiments, including designing methodologies, implementing code, executing experiments, and drawing conclusions. Use when the user wants to benchmark on EXP-Bench, or asks about evaluating this task. Reports All·E✓.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2505.24785
  bibtex_key: kon2025expbench
  confidence: high
---

# exp-bench-eval

> EXP-Bench: Can AI Conduct AI Research Experiments? — Kon et al. (2025) (arXiv:2505.24785, 2025)

## What this evaluates

Evaluates AI agents' end-to-end capability to conduct real AI research experiments, including designing methodologies, implementing code, executing experiments, and drawing conclusions.

## Datasets

- **EXP-Bench** — total 461; splits: test (461); repo https://github.com/Just-Curieous/Curie

## Metrics

- `All·E✓` **(primary)** — range: percent
  - Conjunctive metric requiring full correctness in design (D), implementation (I), conclusion (C), and executability (E). Score is the product of the four component scores.
- `D` — range: percent
  - Design correctness, defined as the proportion of design criteria met by the agent's experimental plan.
- `I` — range: percent
  - Implementation correctness, defined as the proportion of implementation components satisfied by the agent's code.
- `C` — range: percent
  - Conclusion correctness, assessing whether the agent's analytical conclusions align with ground truth.
- `E` — range: percent
  - Executability, a binary metric indicating whether the agent-generated code runs successfully in a clean containerized environment and produces expected outputs.
- `I·E` — range: percent
  - Conjunctive metric combining implementation correctness and executability.
- `All✓` — range: percent
  - Tasks fully correct in terms of D, I, and C.
- `M` — range: percent
  - Integrity check monitoring for disallowed behaviors (accessing PDF, git operations, or using fake data).

## Input / output format

**Input**: Masked GitHub repository of a target AI paper (with task-specific scripts removed), experimental instructions, and relevant context (e.g., API credentials), executed in an Ubuntu 24.04 Docker container with 4× Nvidia A40 GPUs.

**Output**: Agent-generated code modifications, experimental design documents, implementation scripts, and analytical conclusions.

## Scoring recipe

```python
def evaluate(task, agent_output, gt):
    m = 0 if (accessed_pdf or git_ops or fake_data) else 1
    if m == 0: return 0.0
    d = proportion_met(agent_output.design, gt.design)
    i = proportion_met(agent_output.code, gt.code)
    c = 1.0 if conclusion_matches(agent_output.conclusion, gt.conclusion) else 0.0
    e = 1.0 if run_in_clean_container(agent_output.code) and produces_expected_output() else 0.0
    return {
        'M': m, 'D': d, 'I': i, 'C': c, 'E': e,
        'I·E': i * e,
        'All✓': d * i * c,
        'All·E✓': d * i * c * e
    }
```

## Common pitfalls

- Agents can produce plausible but unfounded conclusions (C) or mock implementations that successfully execute (E), causing overestimation bias in partial metrics.
- Execution checking is computationally expensive, so only a subset of traces are run through the Code Execution Validator (#E), which may skew results if early failures are automatically discarded.
- Conjunctive metrics (e.g., All·E✓) drop scores drastically compared to partial metrics, masking agent capabilities if only strict end-to-end scores are reported.

## Evidence (verbatim from paper)

> These assessments yield scores for: D (design correctness, i.e., proportion of design criteria met), I (implementation correctness, i.e., proportion of implementation components satisfied), and C (conclusion correctness). This executability metric is denoted as E. All✓ denotes tasks that are fully correct in terms of D, I, and C, while All·E✓ adds the executability requirement.

## Citation

```bibtex
@misc{kon2025expbench,
  title={EXP-Bench: Can AI Conduct AI Research Experiments?},
  author={Kon et al. (2025)},
  year={2025},
  note={arXiv:2505.24785}
}
```

- arXiv: 2505.24785