Builderbench Eval

---
name: builderbench-eval
description: Evaluates an agent's ability to learn embodied reasoning, long-horizon planning, and physical/geometric intuition through self-supervised exploration, and generalizes these skills to construct unseen block structures. Use when the user wants to benchmark on BuilderBench, or asks about evaluating this task. Reports success_rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.06288
  bibtex_key: ghugare2025builderbench
  confidence: medium
---

# builderbench-eval

> BuilderBench -- A benchmark for generalist agents — Ghugare et al. (2025) (arXiv:2510.06288, 2025)

## What this evaluates

Evaluates an agent's ability to learn embodied reasoning, long-horizon planning, and physical/geometric intuition through self-supervised exploration, and generalizes these skills to construct unseen block structures.

## Datasets

- **BuilderBench** — total 42; splits: train (-1), test (42); repo https://github.com/RajGhugare19/builderbench

## Metrics

- `success_rate` **(primary)** — range: [0, 1]
  - Binary success metric indicating whether the agent's final environment state matches the target block structure goal within a defined tolerance. Calculated as the fraction of test tasks completed successfully.

## Input / output format

**Input**: State observations of the MuJoCo simulation (robotic hand pose, block positions) and a target block structure goal.

**Output**: Control actions for the robotic hand (locomotion, grasping, throwing, pick-and-place) within the MuJoCo environment.

## Scoring recipe

```python
def compute_metric(predictions, gold):
    successes = 0
    for goal in gold:
        final_state = predictions[goal]
        if is_structurally_equivalent(final_state, goal, tol=0.1):
            successes += 1
    return successes / len(gold)
```

## Common pitfalls

- Agents may memorize specific block arrangements instead of learning general construction principles.
- The 'debug' mode allows direct access to test-time goals during training, which inflates performance compared to the strict self-supervised exploration protocol.
- Success tolerance thresholds for matching the target structure are not explicitly defined, requiring careful implementation of geometric/physical matching logic.

## Evidence (verbatim from paper)

> During training, agents must discover such skills through practice. During testing, agents must use those skills to build unseen structures... We open-source BuilderBench, a task-suite of over 40 tasks to evaluate the performance of agents. Each task corresponds to a different block structure.

## Citation

```bibtex
@misc{ghugare2025builderbench,
  title={BuilderBench -- A benchmark for generalist agents},
  author={Ghugare et al. (2025)},
  year={2025},
  note={arXiv:2510.06288}
}
```

- arXiv: 2510.06288