Embodiedgpt Control Eval

---
name: embodiedgpt-control-eval
description: This evaluation probes the model's ability to generate executable sub-goal plans from visual inputs and translate them into low-level control actions in simulated robotic environments. It specifically tests closed-loop planning and few-shot policy adaptation across standard embodied AI benchmarks. Use when the user wants to benchmark on Franka Kitchen, Meta-World, or asks about evaluating this task. Reports success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2305.15021
  bibtex_key: mu2023embodiedgpt
  confidence: high
---

# embodiedgpt-control-eval

> EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought — Yao Mu et al. (2023) (arXiv:2305.15021, 2023)

## What this evaluates

This evaluation probes the model's ability to generate executable sub-goal plans from visual inputs and translate them into low-level control actions in simulated robotic environments. It specifically tests closed-loop planning and few-shot policy adaptation across standard embodied AI benchmarks.

## Datasets

- **Franka Kitchen** — total ?; splits: test (-1)
- **Meta-World** — total ?; splits: test (-1)

## Metrics

- `success rate` **(primary)** — range: percent
  - Average success rate computed over 100 random evaluations per task, aggregated across 5 tasks per benchmark, 5 random seeds, and 2 camera views.

## Input / output format

**Input**: Visual observations from two camera views, combined with a few-shot demonstration set (10 or 25 examples) for policy learning.

**Output**: Low-level control actions executed in the simulation environment.

## Scoring recipe

```python
total_successes = 0
total_trials = 0
for seed in range(5):
    for camera in range(2):
        for task in tasks:
            for _ in range(100):
                obs = get_observation(task, camera, seed)
                action = policy_network(obs, demonstrations)
                if environment_step(action):
                    total_successes += 1
                total_trials += 1
return (total_successes / total_trials) * 100
```

## Common pitfalls

- The policy network is adapted using few-shot demonstrations (10 or 25), not evaluated zero-shot.
- The reported success rate is an average over 5 seeds, 2 camera views, and 5 tasks; single-run scores are not directly comparable.

## Evidence (verbatim from paper)

> In all experiments, the policy network is learned using few-shot learning on a small amount of demonstration data. There are two settings, one of which utilizes 10 demonstrations, and the other utilizes 25 demonstrations. We report the success rate in 100 random evaluations with only visual observations in 5 tasks per benchmark over 5 seeds and 2 different camera views for each setting, respectively.

## Citation

```bibtex
@misc{mu2023embodiedgpt,
  title={EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought},
  author={Yao Mu et al. (2023)},
  year={2023},
  note={arXiv:2305.15021}
}
```

- arXiv: 2305.15021