Videocraftbench Calvin Eval

---
name: videocraftbench-calvin-eval
description: Evaluates a model's ability to learn transferable, long-horizon action dynamics from real-world videos and generate coherent task execution sequences across different environments and robotic setups. Use when the user wants to benchmark on Video-CraftBench, CALVIN, or asks about evaluating this task. Reports Sequential Success Rate (%).
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2602.10102
  bibtex_key: [REDACTED]
  confidence: high
---

# videocraftbench-calvin-eval

> VideoWorld 2: Learning Transferable Knowledge from Real-world Videos — Zhongwei Ren et al. (2026) (arXiv:2602.10102, 2026)

## What this evaluates

Evaluates a model's ability to learn transferable, long-horizon action dynamics from real-world videos and generate coherent task execution sequences across different environments and robotic setups.

## Datasets

- **Video-CraftBench** — total ?; splits: test (-1)
- **CALVIN** — total ?; splits: test (-1)

## Metrics

- `Sequential Success Rate (%)` **(primary)** — range: percent
  - Percentage of trials where the model successfully completes the full sequential task (e.g., 7 steps for paper folding, 5 for block stacking), where success at each step is strictly contingent on completing all preceding steps.
- `SSIM` — range: [0, 1]
  - Structural Similarity Index Measure computed between generated and ground-truth video frames to assess visual fidelity.
- `LPIPS` — range: [0, 1]
  - Learned Perceptual Image Patch Similarity computed between generated and ground-truth video frames, where lower values indicate better perceptual quality.
- `PSNR` — range: [0, 1]
  - Peak Signal-to-Noise Ratio between generated and ground-truth video frames, measuring pixel-level reconstruction accuracy.
- `Avg. Len.` — range: other
  - Average number of steps completed successfully in sequential tasks before failure.

## Input / output format

**Input**: Initial reference image and text instructions (step-by-step or general) for video generation; for CALVIN, latent codes or ground-truth action labels for policy training.

**Output**: Autoregressively generated video clips conditioned on the initial frame and previous outputs; for CALVIN, predicted action sequences or task completion status.

## Scoring recipe

```python
def compute_metrics(predictions, gold, num_steps):
    success_count = 0
    ssim_scores, lpips_scores = [], []
    for trial in predictions:
        completed_steps = 0
        for step in range(num_steps):
            pred_action = trial[step]
            if not environment.step(pred_action):
                break
            completed_steps += 1
        if completed_steps == num_steps:
            success_count += 1
        ssim_scores.append(compute_ssim(trial, gold.video))
        lpips_scores.append(compute_lpips(trial, gold.video))
    return {
        'Sequential Success Rate (%)': (success_count / len(predictions)) * 100,
        'SSIM': sum(ssim_scores) / len(ssim_scores),
        'LPIPS': sum(lpips_scores) / len(lpips_scores)
    }
```

## Common pitfalls

- Sequential evaluation requires success at every preceding step; a single failure terminates the sequence, heavily penalizing error accumulation.
- Models often overfit to background/appearance cues rather than learning transferable action dynamics, causing failure in novel environments.
- Long-horizon autoregressive generation suffers from compounding visual degradation, making step-by-step success rates drop sharply after step 3-4.

## Evidence (verbatim from paper)

> In contrast, VideoWorld 2 (row 9) generates complete and coherent task sequences in test environments. Remarkably, without requiring large-scale pre-training for these latent codes, training solely on Video-CraftBench achieves a success rate of 68.8% on the paper folding task and up to 81.5% on the block stacking task. This demonstrates that our dLDM efficiently extracts core task-relevant action information while filtering out extraneous details like background clutter, object variations, and camera noise. Consequently, it generalizes effectively to unseen environments. Furthermore, benefiting from the appearance priors of the VDM, VideoWorld 2 not only generates accurate actions but also produces videos with higher SSIM and PSNR metrics.

## Citation

```bibtex
@misc{ren2026videoworld2,
  title={VideoWorld 2: Learning Transferable Knowledge from Real-world Videos},
  author={Zhongwei Ren et al. (2026)},
  year={2026},
  note={arXiv:2602.10102}
}
```

- arXiv: 2602.10102