Embodiedclaw Eval

---
name: embodiedclaw-eval
description: Evaluates the efficiency and executability of conversational workflow automation for embodied AI development tasks, including environment synthesis, trajectory collection, and VLA model evaluation. Use when the user wants to benchmark on RoboTwin, or asks about evaluating this task. Reports average task completion time.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2604.13800
  bibtex_key: [REDACTED]
  confidence: high
---

# embodiedclaw-eval

> EmbodiedClaw: Conversational Workflow Execution for Embodied AI Development — Zhou et al. (2026) (arXiv:2604.13800, 2026)

## What this evaluates

Evaluates the efficiency and executability of conversational workflow automation for embodied AI development tasks, including environment synthesis, trajectory collection, and VLA model evaluation.

## Datasets

- **RoboTwin** — total ?; splits: test (-1)

## Metrics

- `average task completion time` **(primary)** — range: minutes
  - Mean wall-clock time (in minutes) required to complete a development task across participants or runs. Lower values indicate higher efficiency.
- `task completion rate` — range: [0, 1]
  - Proportion of workflow executions that finish correctly and execute successfully without failure. Calculated as successful completions divided by total attempts.

## Input / output format

**Input**: Natural-language instructions or modification requests for embodied AI development tasks (e.g., environment creation, editing, trajectory collection, model evaluation).

**Output**: Executable simulation environments, trajectory data in training-ready formats, and model evaluation results.

## Scoring recipe

```python
def compute_efficiency(times):
    return sum(times) / len(times)

def compute_accuracy(success_flags):
    return sum(success_flags) / len(success_flags)
```

## Common pitfalls

- Task completion rate measures end-to-end workflow executability, not standard prediction accuracy.
- Efficiency gains from EmbodiedClaw may come with slightly longer runtimes on some tasks due to tool encapsulation, so runtime alone does not indicate inferior performance.

## Evidence (verbatim from paper)

> We use average task completion time as the efficiency metric, where lower is better. We further evaluate EmbodiedClaw in terms of task completion rate, which measures whether an embodied development workflow is completed correctly and executed successfully.

## Citation

```bibtex
@misc{zhou2026embodiedclaw,
  title={EmbodiedClaw: Conversational Workflow Execution for Embodied AI Development},
  author={Zhou et al. (2026)},
  year={2026},
  note={arXiv:2604.13800}
}
```

- arXiv: 2604.13800