Researchgym Eval

---
name: researchgym-eval
description: Evaluates the capability of LLM agents to conduct closed-loop scientific research by proposing hypotheses, executing experiments, and outperforming human baselines on repurposed real-world AI papers. It probes long-horizon planning, resource management, and autonomous experimentation under realistic tool constraints. Use when the user wants to benchmark on ResearchGym, or asks about evaluating this task. Reports improvement over baselines.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.15112
  bibtex_key: garikaparthi2026researchgym
  confidence: medium
---

# researchgym-eval

> ResearchGym: Evaluating Language Model Agents on Real-World AI Research — Garikaparthi et al. (2026) (arXiv:2602.15112, 2026)

## What this evaluates

Evaluates the capability of LLM agents to conduct closed-loop scientific research by proposing hypotheses, executing experiments, and outperforming human baselines on repurposed real-world AI papers. It probes long-horizon planning, resource management, and autonomous experimentation under realistic tool constraints.

## Datasets

- **ResearchGym** — total ?; splits: test (-1); repo https://github.com/Anikethh/ResearchGym

## Metrics

- `improvement over baselines` **(primary)** — range: percent
  - Percentage improvement of the agent's final experimental result over the strong human baseline provided for each task. Calculated as ((agent_result - human_baseline) / human_baseline) * 100.
- `sub-task completion rate` — range: percent
  - Percentage of required research sub-tasks (e.g., hypothesis generation, experiment execution, analysis) successfully completed by the agent within the allocated budget and time.

## Input / output format

**Input**: Real-world ML/AI research papers with methods withheld, accompanied by API access to HuggingFace, Semantic Scholar, Kaggle, and Exa search. Web search is filtered with an October 2024 cutoff and 160 paper-related URLs are blocked.

**Output**: Agent-generated hypotheses, executable experiment code/scripts, and final research submissions/outcomes.

## Scoring recipe

```python
def score(predictions, gold):
    # predictions: dict with 'final_result' and 'sub_task_status'
    # gold: dict with 'human_baseline' and 'sub_tasks' list
    improvement = ((predictions['final_result'] - gold['human_baseline']) / gold['human_baseline']) * 100
    completed = sum(1 for sub in gold['sub_tasks'] if predictions['sub_task_status'].get(sub, False))
    completion_rate = (completed / len(gold['sub_tasks'])) * 100
    return improvement, completion_rate
```

## Common pitfalls

- Budget and time limits are strict and asymmetric: open agents get 10$ / 12hrs, while proprietary scaffolds get 20$ / 24hrs, heavily impacting long-horizon task completion.
- Web search is artificially constrained (Oct '24 cutoff, 160 blocked URLs), which may unfairly penalize agents relying on recent literature or specific repositories.
- Results are reported as best@k across 3 independent runs with mean ± std, not single-run accuracy, requiring careful aggregation and multiple trials.

## Evidence (verbatim from paper)

> GPT-5 achieves only 6.7% improvement over baselines across 15 runs and completes just 26.5% of sub-tasks, revealing persistent failure modes like impatience, poor resource management, overconfidence, and context limits—despite occasional state-of-the-art performance.

## Citation

```bibtex
@misc{garikaparthi2026researchgym,
  title={ResearchGym: Evaluating Language Model Agents on Real-World AI Research},
  author={Garikaparthi et al. (2026)},
  year={2026},
  note={arXiv:2602.15112}
}
```

- arXiv: 2602.15112