Peek Robot Zero Shot Eval

---
name: peek-robot-zero-shot-eval
description: Evaluates zero-shot generalization and visual/semantic robustness of robot manipulation policies when transferred to new real-world setups, visual clutter, and unseen object configurations. Use when the user wants to benchmark on Franka Sim-to-Real Custom Setup, BRIDGE-v2, or asks about evaluating this task. Reports success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2509.18282
  bibtex_key: zhang2025peek
  confidence: medium
---

# peek-robot-zero-shot-eval

> PEEK: Guiding and Minimal Image Representations for Zero-Shot Generalization of Robot Manipulation Policies — Jesse Zhang et al. (2025) (arXiv:2509.18282, 2025)

## What this evaluates

Evaluates zero-shot generalization and visual/semantic robustness of robot manipulation policies when transferred to new real-world setups, visual clutter, and unseen object configurations.

## Datasets

- **Franka Sim-to-Real Custom Setup** — total 2500; splits: test (-1)
- **BRIDGE-v2** — total ?; splits: test (-1)

## Metrics

- `success rate` **(primary)** — range: percent
  - Proportion of successful task completions out of total evaluation trials. Calculated as successes / total_trials per task variation.

## Input / output format

**Input**: RGB images, depth/point clouds (for 3D policies), and language task instructions. PEEK overlays predicted end-effector paths and task-relevant masking points onto the observations.

**Output**: Robot end-effector actions/poses for manipulation tasks.

## Scoring recipe

```python
successes = 0
total = 0
for task_var in [Basic, Clutter, Semantic]:
    for trial in range(5):
        if task_completion_is_successful():
            successes += 1
        total += 1
return successes / total
```

## Common pitfalls

- Confusing simulation data collection (2.5k trajectories) with the actual zero-shot evaluation, which occurs entirely in the real world or on a different WidowX setup.
- Assuming the metric is explicitly defined in the text; it relies on standard robotic task success rates without a formal mathematical formula provided.
- Overlooking that PEEK is policy-agnostic and evaluated across multiple base architectures (3DDA, ACT, π₀), not just a single model.

## Evidence (verbatim from paper)

> Each policy is evaluated for 5 trials per task, totaling 220 evaluations across 4 methods and 11 variations. We perform 5 evals per task with randomized object locations.

## Citation

```bibtex
@misc{zhang2025peek,
  title={PEEK: Guiding and Minimal Image Representations for Zero-Shot Generalization of Robot Manipulation Policies},
  author={Jesse Zhang et al. (2025)},
  year={2025},
  note={arXiv:2509.18282}
}
```

- arXiv: 2509.18282