Fire Bench Eval

---
name: fire-bench-eval
description: Evaluates autonomous coding agents on their ability to rediscover established scientific findings by autonomously planning, implementing, and executing experiments from scratch based only on high-level research questions. It probes end-to-end research workflow capabilities, including experimental design, code generation, and evidence-based conclusion formation. Use when the user wants to benchmark on FIRE-Bench, or asks about evaluating this task. Reports F1.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.02905
  bibtex_key: [REDACTED]
  confidence: low
---

# fire-bench-eval

> FIRE-Bench: Evaluating Agents on the Rediscovery of Scientific Insights — Wang et al. (arXiv:2602.02905, 2026)

## What this evaluates

Evaluates autonomous coding agents on their ability to rediscover established scientific findings by autonomously planning, implementing, and executing experiments from scratch based only on high-level research questions. It probes end-to-end research workflow capabilities, including experimental design, code generation, and evidence-based conclusion formation.

## Datasets

- **FIRE-Bench** — total ?; splits: test (-1)

## Metrics

- `F1` **(primary)** — range: [0, 1]
  - F1 score measuring rediscovery performance; exact calculation details are not specified in the provided section.

## Input / output format

**Input**: Instruction file containing the research question, experimental constraints, and provided datasets, delivered to an agent in a sandbox environment via CLI.

**Output**: Autonomous execution trajectory including research planning, code implementation, experiment execution, and final evidence-based conclusions.

## Scoring recipe

```python
scores = []
for run in range(3):
    trajectory = agent.execute(task_instruction)
    score = compute_rediscovery_f1(trajectory, gold_finding)
    scores.append(score)
mean_score = sum(scores) / len(scores)
std_score = std(scores)
return mean_score, std_score
```

## Common pitfalls

- Agents may bypass the rediscovery task by searching for the original paper online instead of generating an independent plan.
- Environment setup (e.g., package installation) is intentionally not preconfigured, testing the agent's full capability rather than just coding.
- Proprietary agent default checkpoints may change over time, affecting reproducibility across different evaluation windows.

## Evidence (verbatim from paper)

> Results show current agents achieve <50 F1 on rediscovery tasks, with dominant failure modes in research planning and conclusion formation, highlighting persistent gaps in scientific reasoning, experimental design, and evidence integration despite advances in LLM-powered agents.

## Citation

```bibtex
@misc{wang2026firebench,
  title={FIRE-Bench: Evaluating Agents on the Rediscovery of Scientific Insights},
  author={Wang et al.},
  year={2026},
  note={arXiv:2602.02905}
}
```

- arXiv: 2602.02905