Canvas Eval

---
name: canvas-eval
description: Evaluates robotic navigation policies under diverse simulated environments, testing robustness to both precise and misleading natural language instructions. Use when the user wants to benchmark on CANVAS, or asks about evaluating this task. Reports success rate (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.05684
  bibtex_key: choi2025d2e
  confidence: high
---

# canvas-eval

> D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI — Suhwan Choi et al. (2025) (arXiv:2510.05684, 2025)

## What this evaluates

Evaluates robotic navigation policies under diverse simulated environments, testing robustness to both precise and misleading natural language instructions.

## Datasets

- **CANVAS** — total ?; splits: test (-1)

## Metrics

- `success rate (%)` **(primary)** — range: percent
  - Percentage of successful navigation rollouts out of total attempts. Calculated as (number of successful episodes / total episodes) * 100.

## Input / output format

**Input**: Visual observations and natural language navigation instructions.

**Output**: Robot navigation commands (e.g., velocity or waypoint).

## Scoring recipe

```python
success_count = 0
for rollout in rollouts:
    if rollout.success:
        success_count += 1
success_rate = (success_count / len(rollouts)) * 100
```

## Common pitfalls

- Success rates are highly sensitive to instruction type (misleading vs. precise), so reporting a single aggregate score hides instruction-following robustness.
- The benchmark uses simulated environments, which may not capture real-world sensor noise or navigation obstacles.

## Evidence (verbatim from paper)

> Table 9: Results on CANVAS tasks (success rates, %)

## Citation

```bibtex
@misc{choi2025d2e,
  title={D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI},
  author={Suhwan Choi et al. (2025)},
  year={2025},
  note={arXiv:2510.05684}
}
```

- arXiv: 2510.05684