Vpt Minecraft Eval

---
name: vpt-minecraft-eval
description: Evaluates an agent's ability to perform complex, multi-step sequential decision-making tasks in a 3D sandbox environment (Minecraft) using a native human-like interface. It probes zero-shot generalization, behavioral cloning fine-tuning, and reinforcement learning fine-tuning for long-horizon crafting and exploration. Use when the user wants to benchmark on webClean, contractor_house, earlygame_keyword, or asks about evaluating this task. Reports reliability.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2206.11795
  bibtex_key: baker2022vpt
  confidence: high
---

# vpt-minecraft-eval

> Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos — Bowen Baker et al. (2022) (arXiv:2206.11795, 2022)

## What this evaluates

Evaluates an agent's ability to perform complex, multi-step sequential decision-making tasks in a 3D sandbox environment (Minecraft) using a native human-like interface. It probes zero-shot generalization, behavioral cloning fine-tuning, and reinforcement learning fine-tuning for long-horizon crafting and exploration.

## Datasets

- **webClean** — total ?; splits: train (-1)
- **contractor_house** — total ?; splits: train (-1)
- **earlygame_keyword** — total ?; splits: train (-1)

## Metrics

- `keypress accuracy` — range: percent
  - Percentage of correctly predicted keypresses from video sequences.
- `mouse movement R^2` — range: [0, 1]
  - Explained variance of predicted mouse movements compared to ground truth.
- `reliability` **(primary)** — range: percent
  - Percentage of evaluation episodes in which the agent successfully obtains a target item (e.g., diamond pickaxe) within the time limit.
- `crafting/collection rate` — range: count/episode
  - Mean number of specific items collected or crafted per episode, averaged over multiple rollouts.

## Input / output format

**Input**: Per-step visual observation from the Minecraft environment (native human interface at 20Hz).

**Output**: Discrete keypresses and continuous mouse movements.

## Scoring recipe

```python
def evaluate(agent, env, num_episodes=2500, time_limit=600):
    success_counts = {item: 0 for item in target_items}
    for _ in range(num_episodes):
        obs = env.reset()
        for t in range(time_limit):
            action = agent(obs)
            obs, _, done, info = env.step(action)
            if done: break
        for item in target_items:
            if info['collected'][item]:
                success_counts[item] += 1
    return {item: count / num_episodes for item, count in success_counts.items()}
```

## Common pitfalls

- Catastrophic forgetting during RL fine-tuning can erase previously learned skills if an auxiliary KL divergence loss to the frozen pretrained policy is not applied.
- The native human interface (mouse/keyboard at 20Hz) makes exploration extremely difficult for RL agents compared to simplified action spaces, often causing training to stall without careful initialization.
- Out-of-distribution shift between contractor-labeled data and web-scraped videos can cause validation loss on contractor data to increase during foundation model training, though it does not always correlate with degraded rollout performance.

## Evidence (verbatim from paper)

> The three-phase training (pretraining, BC fine-tuning, and then RL fine-tuning) succeeds in learning extremely difficult tasks: it achieves over 80% reliability on iron pickaxes, almost 20% reliability on collecting diamonds, and 2.5% reliability on obtaining a diamond pickaxe (Fig. 7d).

## Citation

```bibtex
@misc{baker2022vpt,
  title={Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos},
  author={Bowen Baker et al. (2022)},
  year={2022},
  note={arXiv:2206.11795}
}
```

- arXiv: 2206.11795