The Colosseum Eval

---
name: the-colosseum-eval
description: Evaluates the generalization and robustness of robotic behavior cloning models under various environmental perturbations. It probes how well models trained on clean demonstrations can complete manipulation tasks when faced with changes in lighting, color, distractors, camera pose, and object properties. Use when the user wants to benchmark on The Colosseum, or asks about evaluating this task. Reports task-averaged success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2402.08191
  bibtex_key: pumacay2024colosseum
  confidence: high
---

# the-colosseum-eval

> THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation — Pumacay et al. (2024) (arXiv:2402.08191, 2024)

## What this evaluates

Evaluates the generalization and robustness of robotic behavior cloning models under various environmental perturbations. It probes how well models trained on clean demonstrations can complete manipulation tasks when faced with changes in lighting, color, distractors, camera pose, and object properties.

## Datasets

- **The Colosseum** — total ?; splits: test (-1)

## Metrics

- `task-averaged success rate` **(primary)** — range: percent
  - Percentage of episodes where the model successfully completes the manipulation task. Calculated as the average success rate across all 20 tasks for a given perturbation condition.

## Input / output format

**Input**: Four camera RGB or RGBD views, a language instruction, and robot proprioception (arm pose).

**Output**: Continuous 7-DoF keypoint pose, discretized rotation (Euler angles), and binary gripper open/close state.

## Scoring recipe

```python
success_count = 0
for episode in episodes:
    if model_completes_task(episode):
        success_count += 1
success_rate = (success_count / len(episodes)) * 100
task_averaged_success_rate = np.mean([success_rate_per_task for task in tasks])
```

## Common pitfalls

- Confusing RLBench default task variations (e.g., drawer location) with The Colosseum's environmental perturbations.
- Averaging success rates across tasks without reporting per-task performance, which masks task-specific failure modes.
- Using the final training checkpoint rather than validation-based selection, as the protocol fixes evaluation to the last checkpoint.

## Evidence (verbatim from paper)

> A test episode is successful if the model completes the task fully. We report the average success rate for each test set, further averaged across tasks, referred to as task-averaged success rate hereon.

## Citation

```bibtex
@misc{pumacay2024colosseum,
  title={THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation},
  author={Pumacay et al. (2024)},
  year={2024},
  note={arXiv:2402.08191}
}
```

- arXiv: 2402.08191