Mle Bench Lite Eval

---
name: mle-bench-lite-eval
description: Evaluates an agent's ability to iteratively refine and improve runnable solutions for competition-style ML tasks over a long horizon. It probes sustained experiment improvement and competitive performance rather than just initial submission validity. Use when the user wants to benchmark on MLE-Bench Lite, or asks about evaluating this task. Reports Any Medal%.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.13018
  bibtex_key: chen2026toward
  confidence: high
---

# mle-bench-lite-eval

> Toward Autonomous Long-Horizon Engineering for ML Research — Chen et al. (2026) (arXiv:2604.13018, 2026)

## What this evaluates

Evaluates an agent's ability to iteratively refine and improve runnable solutions for competition-style ML tasks over a long horizon. It probes sustained experiment improvement and competitive performance rather than just initial submission validity.

## Datasets

- **MLE-Bench Lite** — total ?; splits: test (-1)

## Metrics

- `Any Medal%` **(primary)** — range: percent
  - Percentage of tasks where the agent's submission achieves at least a Bronze medal. Also reports ValidSubmission%, AboveMedian%, Bronze%, Silver%, and Gold%.

## Input / output format

**Input**: Competition-style ML tasks requiring iterative experiment improvement, with a 24-hour time budget and one H20 GPU per task.

**Output**: Runnable solutions/submissions for ML tasks.

## Scoring recipe

```python
def score_mle_bench_lite(predictions, gold):
    medal_counts = {'ValidSubmission': 0, 'AboveMedian': 0, 'Bronze': 0, 'Silver': 0, 'Gold': 0, 'AnyMedal': 0}
    for task in predictions:
        grade = grade_competition_submission(task.output, task.gold)
        if grade.is_valid: medal_counts['ValidSubmission'] += 1
        if grade.above_median: medal_counts['AboveMedian'] += 1
        if grade.medal in ['Bronze', 'Silver', 'Gold']: medal_counts[grade.medal] += 1
        if grade.medal != 'None': medal_counts['AnyMedal'] += 1
    total = len(predictions)
    return {k: v/total for k, v in medal_counts.items()}
```

## Common pitfalls

- Official leaderboard results are not directly matched comparisons due to different setups and models, so they should only be used for contextual reference.
- Grading focuses on sustained improvement across multiple rounds, meaning early valid submissions do not guarantee high medal counts.

## Evidence (verbatim from paper)

> MLE-Bench Lite [Chan et al., 2025] evaluates sustained experiment improvement on top-tier competition-style ML tasks, with Any Medal% as the primary metric. ... On the controlled evaluation, AiScientist delivers the strongest overall performance under both backbones. It reaches the same 81.82 Any Medal% with Gemini-3-Flash and GLM-5, improving over the strongest matched baseline by 4.55 and 18.18 points, respectively.

## Citation

```bibtex
@misc{chen2026toward,
  title={Toward Autonomous Long-Horizon Engineering for ML Research},
  author={Chen et al. (2026)},
  year={2026},
  note={arXiv:2604.13018}
}
```

- arXiv: 2604.13018