Mlgym Eval

---
name: mlgym-eval
description: Evaluates LLM agents on open-ended AI research tasks across 13 diverse benchmarks. It measures the agent's ability to navigate codebases, run experiments, and improve model performance, assessing capabilities from reproducing existing research to achieving state-of-the-art results. Use when the user wants to benchmark on MLGym Benchmarks, or asks about evaluating this task. Reports AutoML-inspired optimization metric.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2502.14499
  bibtex_key: [REDACTED]
  confidence: medium
---

# mlgym-eval

> MLGym: A New Framework and Benchmark for Advancing AI Research Agents — Nathani et al. (2025) (arXiv:2502.14499, 2025)

## What this evaluates

Evaluates LLM agents on open-ended AI research tasks across 13 diverse benchmarks. It measures the agent's ability to navigate codebases, run experiments, and improve model performance, assessing capabilities from reproducing existing research to achieving state-of-the-art results.

## Datasets

- **MLGym Benchmarks** — total ?; splits: test (-1); repo https://github.com/facebookresearch/MLGym

## Metrics

- `AutoML-inspired optimization metric` **(primary)** — range: other
  - Quantifies performance improvement across diverse tasks to enable fair cross-task comparison, drawing from AutoML and optimization literature conventions.

## Input / output format

**Input**: Task description, dataset description, ACI documentation, and lightweight generic instructions to act as an ML researcher.

**Output**: A sequence of up to 50 single-line commands (e.g., file navigation, editing, search, validation, submission) executed in a non-interactive environment.

## Scoring recipe

```python
def evaluate_agent(task, agent):
    steps = 0
    max_steps = 50
    context = []
    while steps < max_steps:
        action = agent.step(task, context)
        obs = env.execute(action, timeout=task_timeout)
        context.append((action, obs))
        if action == 'submit':
            break
        steps += 1
    final_codebase = env.get_state()
    score = validate_and_submit(final_codebase, task)
    return score
```

## Common pitfalls

- Agent runs are hard-capped at 50 steps; the final state is autosubmitted if the limit is reached.
- Interactive commands (e.g., python REPL, vim) are explicitly disabled; agents must rely on non-interactive file and search commands.
- Task-specific timeouts are enforced on training commands to prevent agents from bypassing evaluation by simply increasing model parameters.

## Evidence (verbatim from paper)

> A new evaluation metric inspired by AutoML and optimization literature enables fair cross-task comparison, while the framework supports flexible artifact evaluation (e.g., models, code, training logs) and algorithmic research (e.g., RL, curriculum learning).

## Citation

```bibtex
@misc{nathani2025mlgym,
  title={MLGym: A New Framework and Benchmark for Advancing AI Research Agents},
  author={Nathani et al. (2025)},
  year={2025},
  note={arXiv:2502.14499}
}
```

- arXiv: 2502.14499