Phyre Eval

---
name: phyre-eval
description: Evaluates an agent's ability to reason about 2D Newtonian physics to solve goal-driven puzzles by placing dynamic objects. It probes sample-efficient learning and generalization across unseen task templates and action spaces. Use when the user wants to benchmark on PHYRE, or asks about evaluating this task. Reports AUCCESSION.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 1908.05656
  bibtex_key: bakhtin2019phyre
  confidence: high
---

# phyre-eval

> PHYRE: A New Benchmark for Physical Reasoning — Bakhtin et al. (2019) (arXiv:1908.05656, 2019)

## What this evaluates

Evaluates an agent's ability to reason about 2D Newtonian physics to solve goal-driven puzzles by placing dynamic objects. It probes sample-efficient learning and generalization across unseen task templates and action spaces.

## Datasets

- **PHYRE** — total ?; splits: train (-1), val (-1), test (-1); repo https://phyre.ai

## Metrics

- `AUCCESSION` **(primary)** — range: percent
  - Area under the success-percentage curve as a function of the number of solution attempts per task (up to 100). Higher values indicate better sample efficiency and overall solving capability.
- `success percentage` — range: percent
  - The percentage of tasks solved out of the total test tasks at a fixed number of solution attempts per task (e.g., k=10).

## Input / output format

**Input**: A 256x256 image encoding the initial state and goal (7 colors representing body properties and goal), plus task template ID in the within-template setting.

**Output**: An action from a 3D or 6D action space (depending on the PHYRE tier), representing the placement and properties of a dynamic object to be placed in the simulation.

## Scoring recipe

```python
def compute_metrics(predictions, gold, max_attempts=100):
    success_curve = []
    for k in range(1, max_attempts + 1):
        solved = 0
        for task in gold:
            for action in predictions[task][:k]:
                if simulate(task, action).solved:
                    solved += 1
                    break
        success_curve.append(solved / len(gold))
    auc = np.trapz(success_curve) * 100
    return auc
```

## Common pitfalls

- Online agents are permitted up to 100 attempts per task regardless of the evaluation point k, meaning they learn from 100 attempts even when measuring success at k<100.
- Hyperparameter tuning must strictly use the validation set; using test performance for tuning violates the benchmark protocol.
- Statistical significance is evaluated using the non-parametric Wilcoxon signed-rank test, not standard t-tests.

## Evidence (verbatim from paper)

> We measure success percentage and AUCCESSION on PHYRE using the learning setting of 3.2. ... Figure 3 presents success-percentage curves for all five agents on both PHYRE tiers (-B and -2B) in both generalization settings (within-template and cross-template): the curves show the percentage of tasks solved as a function of the number of solution attempts per task, and are computed by averaging over all 10 folds in PHYRE.

## Citation

```bibtex
@misc{bakhtin2019phyre,
  title={PHYRE: A New Benchmark for Physical Reasoning},
  author={Bakhtin et al. (2019)},
  year={2019},
  note={arXiv:1908.05656}
}
```

- arXiv: 1908.05656