Game Eval

---
name: game-eval
description: Evaluates a vision-action model's ability to predict actions and scale ratios from video game footage, testing in-distribution and out-of-distribution generalization across 2D and 3D games. Use when the user wants to benchmark on Video Games (In-Distribution & OOD), or asks about evaluating this task. Reports Pearson correlation.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2510.05684
  bibtex_key: choi2025d2e
  confidence: high
---

# game-eval

> D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI — Suhwan Choi et al. (2025) (arXiv:2510.05684, 2025)

## What this evaluates

Evaluates a vision-action model's ability to predict actions and scale ratios from video game footage, testing in-distribution and out-of-distribution generalization across 2D and 3D games.

## Datasets

- **Video Games (In-Distribution & OOD)** — total ?; splits: train (-1), test (-1)

## Metrics

- `Pearson correlation` **(primary)** — range: other
  - Linear correlation coefficient between predicted and ground-truth action coordinates (X, Y). Measures alignment of predicted trajectories with actual movements.
- `Keypress Accuracy` — range: percent
  - Percentage of correctly predicted keypresses (keyboard or mouse) out of total steps. Calculated as (number of correct predictions / total steps) * 100.

## Input / output format

**Input**: Video frames and previous actions.

**Output**: Predicted action keys and scale ratios.

## Scoring recipe

```python
pearson_x = np.corrcoef(true_x, pred_x)[0, 1]
pearson_y = np.corrcoef(true_y, pred_y)[0, 1]
correct_keys = sum(1 for p, g in zip(pred_keys, gold_keys) if p == g)
keypress_acc = (correct_keys / len(gold_keys)) * 100
```

## Common pitfalls

- Pearson correlation measures linear alignment but ignores absolute scale, which is why Scale Ratio is also reported.
- Keypress accuracy differs significantly between keyboard and mouse inputs, so aggregating them without distinction can be misleading.

## Evidence (verbatim from paper)

> We employ an autoregressive inference pipeline to generate actions and evaluate model performance across multiple metrics... our Generalist-IDM achieves strong performance across all environments. Notably, it yields large gains in Pearson correlation (e.g., +39.5 points on Stardew Valley X) and Keyboard accuracy (e.g., +57.6 points on Brotato), demonstrating robust generalization over diverse control dynamics.

## Citation

```bibtex
@misc{choi2025d2e,
  title={D2E: Scaling Vision-Action Pretraining on Desktop Data for Transfer to Embodied AI},
  author={Suhwan Choi et al. (2025)},
  year={2025},
  note={arXiv:2510.05684}
}
```

- arXiv: 2510.05684