Gamefactory Eval

---
name: gamefactory-eval
description: This evaluation protocol assesses a video generation model's ability to follow discrete and continuous action inputs while maintaining semantic alignment with text prompts and preserving the original model's visual domain. It measures action-following accuracy, camera pose consistency, text-video semantic relevance, and overall video generation quality across in-domain and open-domain scenes. Use when the user wants to benchmark on GF-Minecraft, VPT (Find Cave), or asks about evaluating this task. Reports Flow.
metadata:
  skill_kind: dataset_eval
  source_arxiv: 2501.08325
  bibtex_key: yu2025gamefactory
  confidence: high
---

# gamefactory-eval

> GameFactory: Creating New Games with Generative Interactive Videos — Jiwen Yu et al. (2025) (arXiv:2501.08325, 2025)

## What this evaluates

This evaluation protocol assesses a video generation model's ability to follow discrete and continuous action inputs while maintaining semantic alignment with text prompts and preserving the original model's visual domain. It measures action-following accuracy, camera pose consistency, text-video semantic relevance, and overall video generation quality across in-domain and open-domain scenes.

## Datasets

- **GF-Minecraft** — total ?; splits: test (-1)
- **VPT (Find Cave)** — total ?; splits: test (-1)

## Metrics

- `Flow` **(primary)** — range: other
  - Mean squared error between the optical flow of the generated video and the reference video. Lower values indicate better action-following dynamics.
- `Cam` — range: other
  - Euclidean distance between camera poses extracted from predicted and reference videos using GLOMAP. Lower values indicate better camera pose consistency.
- `CLIP` — range: [0, 1]
  - Cosine similarity between CLIP features of the generated video and the input text prompt. Higher values indicate better semantic relevance.
- `FID` — range: other
  - Fréchet Inception Distance measuring the distribution difference between generated and reference videos. Lower values indicate better generation quality.
- `FVD` — range: other
  - Fréchet Video Distance measuring the distribution difference between generated and reference videos. Lower values indicate better temporal generation quality.

## Input / output format

**Input**: Text prompt describing the scene/game, combined with an action control signal (discrete keyboard keys or continuous mouse coordinates).

**Output**: A sequence of video frames at 360x640 resolution.

## Scoring recipe

```python
def compute_metrics(pred_video, ref_video, text_prompt):
    flow_err = mse(optical_flow(pred_video), optical_flow(ref_video))
    cam_dist = euclidean_dist(glomap_pose(pred_video), glomap_pose(ref_video))
    clip_sim = cosine_similarity(clip_encode(pred_video), clip_encode(text_prompt))
    fid = frechet_inception_distance([pred_video], [ref_video])
    fvd = frechet_video_distance([pred_video], [ref_video])
    return {'Flow': flow_err, 'Cam': cam_dist, 'CLIP': clip_sim, 'FID': fid, 'FVD': fvd}
```

## Common pitfalls

- Metrics have different optimization directions: Flow, Cam, FID, and FVD are lower-is-better, while CLIP is higher-is-better.
- Evaluating on human-biased datasets (e.g., VPT) instead of uniformly distributed action datasets leads to poor generalization on rare or counter-intuitive actions like moving backward or jumping in place.
- Confusing in-domain evaluation (training distribution) with open-domain scene generalization, which requires decoupled training to avoid style leakage.

## Evidence (verbatim from paper)

> We use these evaluation metrics: (1) Flow: calculates the optical flow of the generated video to reflect its dynamics, assessing action-following ability by measuring mean square error to the optical flow of the reference video; (2) Cam: computes the Euclidean distance between camera poses extracted from predicted videos and those extracted from reference videos, where both sets of camera poses are obtained using GLOMAP*[[32]]. (3) CLIP: computes feature similarity in the CLIP*[[37]]* space to evaluate semantic relevance to the given text prompt; (4) FID, FVD: measures distribution differences between the generated videos and the reference, providing an assessment of generation quality.

## Citation

```bibtex
@misc{yu2025gamefactory,
  title={GameFactory: Creating New Games with Generative Interactive Videos},
  author={Jiwen Yu et al. (2025)},
  year={2025},
  note={arXiv:2501.08325}
}
```

- arXiv: 2501.08325