Calvin Long Horizon Eval

---
name: calvin-long-horizon-eval
description: Evaluates a robot policy's ability to chain multiple sub-goals sequentially using only visual observations and goal images. It probes long-horizon planning, goal-conditioned control, and the capacity to generalize to unseen goal configurations without explicit reward signals. Use when the user wants to benchmark on CALVIN, or asks about evaluating this task. Reports Success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2209.08959
  bibtex_key: rosetebeas2022latentplans
  confidence: high
---

# calvin-long-horizon-eval

> Latent Plans for Task-Agnostic Offline Reinforcement Learning — Erick Rosete-Beas et al. (2022) (arXiv:2209.08959, 2022)

## What this evaluates

Evaluates a robot policy's ability to chain multiple sub-goals sequentially using only visual observations and goal images. It probes long-horizon planning, goal-conditioned control, and the capacity to generalize to unseen goal configurations without explicit reward signals.

## Datasets

- **CALVIN** — total ?; splits: train (-1)

## Metrics

- `Success rate` **(primary)** — range: percent
  - Percentage of successful rollouts out of the total number of evaluated chains or tasks. A rollout is successful if the agent reaches the specified goal image/state within the timestep limit before failure.
- `Average Length` — range: other
  - Average number of sub-goals successfully completed in a row before the agent fails or hits the maximum timestep limit.

## Input / output format

**Input**: RGB images from a static camera (simulation) or static + gripper camera (real-world), concatenated with goal images specifying the target state or sub-goal.

**Output**: 7-DoF robot arm actions (joint velocities/positions).

## Scoring recipe

```python
def compute_success_rate(successes, total_rollouts):
    return (successes / total_rollouts) * 100

def evaluate_chain(policy, env, goal_images, max_timesteps=180):
    completed = 0
    for goal in goal_images:
        success = False
        for t in range(max_timesteps):
            action = policy(env.observation, goal)
            env.step(action)
            if env.is_goal_reached(goal):
                success = True
                break
        if success:
            completed += 1
        else:
            break
    return completed, len(goal_images)
```

## Common pitfalls

- Goal images often exclude the robot/end-effector, forcing the model to reason about full scene configuration rather than just end-effector pose.
- Success rates for sequential tasks drop exponentially with chain length; reporting only single-task success masks long-horizon planning capability.
- The 180-timestep limit per sub-goal truncates long tasks, so success rates are highly sensitive to this hyperparameter.

## Evidence (verbatim from paper)

> We call this evaluation of performing multiple tasks on a row, long-horizon multitask with visual observations LH-MTVis. This setting is very challenging as it requires agents to be able to transition between different subgoals. ... We record the success rate of all models in Table 2. TACO-RL successfully performs long-horizon tasks that require reasoning over sequential behaviors with an final success rate of 27% which corresponds to an order of magnitude improvement upon the LMP and CQL+HER baselines.

## Citation

```bibtex
@misc{rosetebeas2022latentplans,
  title={Latent Plans for Task-Agnostic Offline Reinforcement Learning},
  author={Erick Rosete-Beas et al. (2022)},
  year={2022},
  note={arXiv:2209.08959}
}
```

- arXiv: 2209.08959