Mcu Minecraft Eval

---
name: mcu-minecraft-eval
description: Evaluates a model's ability to follow natural language instructions to perform multi-step, spatially grounded tasks in an open-world 3D environment (Minecraft). It specifically probes capabilities in mining, combat, crafting, and smelting under human-like visibility constraints. Use when the user wants to benchmark on MCU Benchmark, or asks about evaluating this task. Reports success rate.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2503.16365
  bibtex_key: li2025jarvisvla
  confidence: high
---

# mcu-minecraft-eval

> JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mouse — Li et al. (2025) (arXiv:2503.16365, 2025)

## What this evaluates

Evaluates a model's ability to follow natural language instructions to perform multi-step, spatially grounded tasks in an open-world 3D environment (Minecraft). It specifically probes capabilities in mining, combat, crafting, and smelting under human-like visibility constraints.

## Datasets

- **MCU Benchmark** — total ?; splits: test (-1)

## Metrics

- `success rate` **(primary)** — range: [0, 1]
  - Calculated as the number of successful task completions divided by the total number of evaluation runs (at least 30 per task). Category-level performance is reported as the average success rate across all tasks within that category.

## Input / output format

**Input**: First-person visual observation (screen frame), natural language task instruction, and environment state. Agent location and inventory stats are hidden to match human player visibility.

**Output**: Discretized action token from a 51-token vocabulary representing camera movements and keyboard/mouse button presses.

## Scoring recipe

```python
def compute_success_rate(successful_runs, total_runs):
    return successful_runs / total_runs

def compute_category_avg(task_success_rates):
    return sum(task_success_rates) / len(task_success_rates)
```

## Common pitfalls

- Maximum execution steps are fixed per task to match the original MCU benchmark protocol; exceeding them counts as failure.
- Inventory and location information are intentionally hidden during evaluation to ensure fair comparison with human players and prior baselines.
- Success is binary per run; the reported metric is strictly the average success rate across at least 30 independent trials per task.

## Evidence (verbatim from paper)

> We conduct evaluations using two broad benchmarks: (i) the agent’s capacity to interact with the Minecraft environment to complete tasks; and (ii) vision-language tasks (e.g., question answering, spatial grounding) designed to assess the VLM’s understanding of Minecraft-specific knowledge. For the instruction-following tasks, we adopt the MCU Benchmark(Lin et al., [2023]), focusing on four categories—Mine Blocks, Kill Entities, Craft Items, and Smelt Items—that represent a wide range of typical game-play behaviors in Minecraft. We perform each task at least 30 times and report the success rate per task, as well as the average success rate within each category.

## Citation

```bibtex
@misc{li2025jarvisvla,
  title={JARVIS-VLA: Post-Training Large-Scale Vision Language Models to Play Visual Games with Keyboards and Mouse},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2503.16365}
}
```

- arXiv: 2503.16365